КАФКА - как перебалансировать разделы для доступных брокеров
У нас есть 3 машины кафки и 3 сервера зоокейпера
kafka1 - 1001 ( broker ID )
kafka2 - 1002 ( broker ID )
kafka3 - 1003 ( broker ID )
у нас есть проблема, связанная с перебалансировкой разделов с доступными брокерами, мы можем видеть (вниз), что некоторые разделы недостаточно реплицированы только с двумя брокерами в isr. Вместо 3
Каков наилучший способ перебалансировать разделы темы Кафка в ISR
во-вторых, мы видим, что лидер 1002 отсутствует, каково решение для этого?
замечание - у нас 23 темы (список неполный)
[kafka@kafka01 bin]$ ./kafka-topics.sh -describe --zookeeper master:2181
Topic:lop_gt PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: lop_gt Partition: 0 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 1 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 2 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 3 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 4 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 5 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 6 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 7 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 8 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 9 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 10 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 11 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 12 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 13 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 14 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 15 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 16 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 17 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 18 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 19 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 20 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 21 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 22 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 23 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 24 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 25 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 26 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 27 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 28 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 29 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 30 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 31 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 32 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 33 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 34 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 35 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 36 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 37 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 38 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 39 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 40 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 41 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 42 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 43 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 44 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 45 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 46 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 47 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 48 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 49 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic:_schemas PartitionCount:1 ReplicationFactor:3 Configs:cleanup.policy=compact
Topic: _schemas Partition: 0 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic:ambari_kafka_service_check PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ambari_kafka_service_check Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
Topic:jr_dfse PartitionCount:10 ReplicationFactor:3 Configs:
Topic: jr_dfse Partition: 0 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: jr_dfse Partition: 1 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: jr_dfse Partition: 2 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: jr_dfse Partition: 3 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: jr_dfse Partition: 4 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: jr_dfse Partition: 5 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: jr_dfse Partition: 6 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: jr_dfse Partition: 7 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: jr_dfse Partition: 8 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: jr_dfse Partition: 9 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic:frte_nnc PartitionCount:6 ReplicationFactor:3 Configs:
Topic: frte_nnc Partition: 0 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: frte_nnc Partition: 1 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: frte_nnc Partition: 2 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: frte_nnc Partition: 3 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: frte_nnc Partition: 4 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: frte_nnc Partition: 5 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic:erw_plk PartitionCount:100 ReplicationFactor:3 Configs:
Topic: erw_plk Partition: 0 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 1 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 2 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 3 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 4 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 5 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 6 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 7 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 8 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 9 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 10 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 11 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 12 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 13 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 14 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 15 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 16 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 17 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 18 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 19 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 20 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 21 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 22 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 23 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 24 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 25 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 26 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 27 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 28 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 29 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 30 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 31 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 32 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 33 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 34 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 35 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 36 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 37 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 38 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 39 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 40 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 41 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 42 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 43 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 44 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 45 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 46 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 47 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 48 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 49 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 50 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 51 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 52 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 53 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 54 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 55 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 56 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 57 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 58 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 59 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 60 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 61 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 62 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 63 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 64 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 65 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 66 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 67 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 68 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 69 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 70 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 71 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 72 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 73 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 74 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 75 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 76 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 77 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 78 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 79 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 80 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 81 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 82 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 83 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 84 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 85 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 86 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 87 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 88 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 89 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 90 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 91 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 92 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 93 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 94 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 95 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 96 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 97 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 98 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 99 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic:loe_sd PartitionCount:6 ReplicationFactor:3 Configs:
Topic: loe_sd Partition: 0 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: loe_sd Partition: 1 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: loe_sd Partition: 2 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: loe_sd Partition: 3 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: loe_sd Partition: 4 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: loe_sd Partition: 5 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
1 ответ
У вас есть два варианта влияния на лидерство в разделе. С опцией конфигурации auto.leader.rebalance.enable установлен в true (это должно быть настройкой по умолчанию), Kafka автоматически попытается переместить лидерство для каждого раздела на вашего предпочтительного брокера. Предпочитаемый брокер является первым в списке реплик. Он выполняется как периодическая проверка, поэтому может произойти не сразу.
В качестве альтернативы - если автоматическое изменение баланса отключено - вы можете переназначить реплики вручную, используя bin/kafka-preferred-replica-election.sh инструмент. Для получения дополнительной информации см. Документы Kafka.
Однако в вашем случае кажется, что ваш брокер 1002 либо вскоре после перезапуска, либо может потребоваться больше времени для повторной синхронизации данных, поскольку брокер 1002 еще не является ISR для всех разделов. Если это постоянное состояние, и вы на 100% уверены, что у вашего брокера было достаточно времени для синхронизации всех разделов и т. Д., Возможно, есть некоторые другие проблемы с 1002. Но это трудно сказать без логов от брокера и т. Д.