Split Brain Condition не объединяется в среде kubernetes с 4.2.1, хотя узлы, похоже, согласны
Мы используем hazelcast 4.2.1 в среде kubernetes с образами openjdk:14-jdk-slim. В нашей среде разработки, где у нас есть только два узла, эти два узла иногда (вскоре после каждого 5-го развертывания) оказываются в состоянии расщепленного мозга и не сливаются, хотя они находят друг друга и договариваются о том, что делать:
Столяр первых узлов говорит, что второй узел должен присоединиться. И столяр второго не должен присоединяться к первому узлу. Но ничего не происходит. Журнал повторяется каждые пару минут, и кластеры не объединяются.
Неважно, используем ли мы политику слияния или нет. Чаще всего работает без проблем.
Лог первого узла:
2021-07-20 09:14:08.306 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.cluster.impl.MembershipManager : [10.41.31.101]:5701 [light-cluster] [4.2.1] Sending member list to the non-master nodes:
Members {size:1, ver:5} [
Member [10.41.31.101]:5701 - 7263bccd-f330-4b96-8b52-f22db7c7a90e this
]
2021-07-20 09:14:08.446 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-5] c.h.i.cluster.impl.DiscoveryJoiner : [10.41.31.101]:5701 [light-cluster] [4.2.1] Sending SplitBrainJoinMessage to [10.41.31.102]:5701
2021-07-20 09:14:08.448 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-5] c.h.i.cluster.impl.ClusterJoinManager : [10.41.31.101]:5701 [light-cluster] [4.2.1] Checking if we should merge to: SplitBrainJoinMessage{packetVersion=4, buildNumber=20210630, memberVersion=4.2.1, clusterVersion=4.2, address=[10.41.31.102]:5701, uuid='9cdd64b4-62c8-4f19-bf29-d3cef4e8e2f6', liteMember=false, memberCount=1, dataMemberCount=1, memberListVersion=1}
2021-07-20 09:14:08.449 INFO 142 --- [hz.hazelcast-instance.cached.thread-5] c.h.i.cluster.impl.ClusterJoinManager : [10.41.31.101]:5701 [light-cluster] [4.2.1] [10.41.31.102]:5701 should merge to us, both have the same data member count: 1
2021-07-20 09:14:23.277 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.p.InternalPartitionService : [10.41.31.101]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -5900145379368197006
Лог второго узла:
2021-07-20 09:14:24.149 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.p.InternalPartitionService : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299
2021-07-20 09:14:24.175 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-4] c.h.s.d.integration.DiscoveryService : [10.41.31.102]:5701 [light-cluster] [4.2.1] Using service name to discover nodes.
2021-07-20 09:14:24.176 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-6] c.h.i.cluster.impl.MembershipManager : [10.41.31.102]:5701 [light-cluster] [4.2.1] Sending member list to the non-master nodes:
Members {size:1, ver:1} [
Member [10.41.31.102]:5701 - 9cdd64b4-62c8-4f19-bf29-d3cef4e8e2f6 this
]
2021-07-20 09:14:39.149 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.p.InternalPartitionService : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299
2021-07-20 09:14:54.148 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-6] c.h.i.p.InternalPartitionService : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299
2021-07-20 09:15:08.423 DEBUG 141 --- [hz.hazelcast-instance.priority-generic-operation.thread-0] c.h.i.cluster.impl.ClusterJoinManager : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking if we should merge to: SplitBrainJoinMessage{packetVersion=4, buildNumber=20210630, memberVersion=4.2.1, clusterVersion=4.2, address=[10.41.31.101]:5701, uuid='7263bccd-f330-4b96-8b52-f22db7c7a90e', liteMember=false, memberCount=1, dataMemberCount=1, memberListVersion=5}
2021-07-20 09:15:08.423 INFO 141 --- [hz.hazelcast-instance.priority-generic-operation.thread-0] c.h.i.cluster.impl.ClusterJoinManager : [10.41.31.102]:5701 [light-cluster] [4.2.1] We should merge to [10.41.31.101]:5701, both have the same data member count: 1
2021-07-20 09:15:08.424 DEBUG 141 --- [hz.hazelcast-instance.priority-generic-operation.thread-0] c.h.i.c.i.o.SplitBrainMergeValidationOp : [10.41.31.102]:5701 [light-cluster] [4.2.1] Returning SplitBrainJoinMessage{packetVersion=4, buildNumber=20210630, memberVersion=4.2.1, clusterVersion=4.2, address=[10.41.31.102]:5701, uuid='9cdd64b4-62c8-4f19-bf29-d3cef4e8e2f6', liteMember=false, memberCount=1, dataMemberCount=1, memberListVersion=1} to [10.41.31.101]:5701
2021-07-20 09:15:09.148 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-6] c.h.i.p.InternalPartitionService : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299```