Кассандра родной транспорт отключает сама

Question

У меня есть большой скрипт Python, загружающий данные в базу данных Cassandra. В двух словах, он просматривает список строк данных и загружает каждую из них со своей отметкой времени в Cassandra.

Что я замечаю, так это то, что сценарий очень часто падает с ошибкой, подобной следующей:

Failed to write to cassandra table: ('Unable to complete the operation against any hosts',{})

В таком случае, nodetool statusbinary возвращается not running и все, что мне нужно сделать, это бежать nodetool enablebinary чтобы скрипт снова заработал - во всяком случае, до следующего сбоя.

Я не вижу ничего, что отключало бы собственный транспорт в логах - что может быть причиной того, что бинарный файл отключился сам по себе?

Номера версий: Кассандра: 3.0.9 Java: 1.8.0_162 CentOS: 6.9

Редактировать: читая журнал, я заметил следующее:

INFO [RMI TCP Connection(5893)-127.0.0.1] 2018-05-08 10:58:17,318 Server.java:160 - Starting listening for CQL clients on /XX.XXX.X.XX:9042 (unencrypted)... ERROR [CompactionExecutor:29] 2018-05-08 10:58:48,700 CassandraDaemon.java:205 - Exception in thread Thread[CompactionExecutor:29,1,main] org.apache.cassandra.io.FSReadError: java.io.IOException: Channel not open for writing - cannot extend file to required size

Более длинный бит:

WARN [CompactionExecutor:4] 2018-05-09 12:10:11,094 CLibrary.java:304 - fsync(315) failed, errno (22) {} com.sun.jna.LastErrorException: [22] h at org.apache.cassandra.utils.CLibrary.fsync(Native Method) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.CLibrary.trySync(CLibrary.java:293) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.SyncUtil.trySync(SyncUtil.java:179) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.SyncUtil.trySyncDir(SyncUtil.java:190) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:117) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:135) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:69) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:163) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:73) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.switchCompactionLocation(DefaultCompactionWriter.java:64) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.maybeSwitchWriter(CompactionAwareWriter.java:128) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:108) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) [apache-cassandra-3.0.9.jar:3.0.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_162] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162] WARN [CompactionExecutor:4] 2018-05-09 12:10:11,096 CLibrary.java:304 - fsync(316) failed, errno (22) {} com.sun.jna.LastErrorException: [22] i at org.apache.cassandra.utils.CLibrary.fsync(Native Method) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.CLibrary.trySync(CLibrary.java:293) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.SyncUtil.trySync(SyncUtil.java:179) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.SyncUtil.trySyncDir(SyncUtil.java:190) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SequentialWriter.openChannel(SequentialWriter.java:117) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:135) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:150) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.<init>(BigTableWriter.java:375) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:84) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.switchCompactionLocation(DefaultCompactionWriter.java:64) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.maybeSwitchWriter(CompactionAwareWriter.java:128) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:108) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) [apache-cassandra-3.0.9.jar:3.0.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_162] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162] ERROR [Reference-Reaper:1] 2018-05-09 12:10:14,363 Ref.java:223 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@4a962079) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@2019607399:[Memory@[0..4e0), Memory@[0..30c0)] was not released before the reference was garbage collected ERROR [CompactionExecutor:4] 2018-05-09 12:14:00,394 CassandraDaemon.java:205 - Exception in thread Thread[CompactionExecutor:4,1,main] org.apache.cassandra.io.FSReadError: java.io.IOException: Channel not open for writing - cannot extend file to required size at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:142) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.MmappedRegions.extend(MmappedRegions.java:131) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.updateRegions(MmappedSegmentedFile.java:130) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:111) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.util.SegmentedFile$Builder.buildIndex(SegmentedFile.java:198) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:244) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) ~[apache-cassandra-3.0.9.jar:3.0.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_162] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162] Caused by: java.io.IOException: Channel not open for writing - cannot extend file to required size at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:901) ~[na:1.8.0_162] at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.0.9.jar:3.0.9] ... 23 common frames omitted ERROR [CompactionExecutor:4] 2018-05-09 12:14:00,394 StorageService.java:399 - Stopping native transport INFO [CompactionExecutor:4] 2018-05-09 12:14:00,394 Server.java:180 - Stop listening for CQL clients

dmesg выводит много:

CIFS VFS: Send error in SETFSUnixInfo = -11 CIFS VFS: Send error in Close = -11 CIFS VFS: Unexpected lookup error -512

Очевидно, утечка - известная проблема Cassandra 3.0.9, которая поставляется с Netty 4.0.23 - но даже если это была проблема утечки памяти, я перезапустил Cassandra, и она все еще не работает...