Контроллер Docker UCP отключен с ошибкой: Нездоровый диспетчер UCP: невозможно связаться с менеджером: соединение: соединение отказано

Question

Я настроил тестовую среду с помощью докера UCP, через несколько дней один из контролеров случайно отключился с сообщением в UCP, что хост не работает и кластер не работает.

Журналы контейнера контроллера:

{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:03:10Z"}
{"level":"warning","msg":"Kube controller manager health check timed out","time":"2018-12-18T10:03:19Z"}
{"level":"warning","msg":"Node health error detected during _ping: Kube controller manager health check timed out","time":"2018-12-18T10:03:19Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:03:19Z"}
{"level":"warning","msg":"Kube controller manager health check timed out","time":"2018-12-18T10:03:56Z"}
{"level":"warning","msg":"Node health error detected during _ping: Kube controller manager health check timed out","time":"2018-12-18T10:03:56Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:03:56Z"}
{"level":"warning","msg":"Kube controller manager health check timed out","time":"2018-12-18T10:04:15Z"}
{"level":"warning","msg":"Node health error detected during _ping: Kube controller manager health check timed out","time":"2018-12-18T10:04:15Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:04:15Z"}
{"level":"warning","msg":"Kube controller manager health check timed out","time":"2018-12-18T10:04:32Z"}
{"level":"warning","msg":"Node health error detected during _ping: Kube controller manager health check timed out","time":"2018-12-18T10:04:32Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:04:32Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:05:07Z"}
{"level":"warning","msg":"Kube controller manager health check timed out","time":"2018-12-18T10:05:07Z"}
{"level":"warning","msg":"Node health error detected during _ping: Kube controller manager health check timed out","time":"2018-12-18T10:05:07Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:05:43Z"}
{"level":"warning","msg":"Kube controller manager health check timed out","time":"2018-12-18T10:05:43Z"}
{"level":"warning","msg":"Node health error detected during _ping: Kube controller manager health check timed out","time":"2018-12-18T10:05:43Z"}
{"level":"warning","msg":"Kube controller manager health check timed out","time":"2018-12-18T10:05:51Z"}
{"level":"warning","msg":"Node health error detected during _ping: Kube controller manager health check timed out","time":"2018-12-18T10:05:51Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context deadline exceeded","time":"2018-12-18T10:05:51Z"}
{"level":"warning","msg":"Kube controller manager health check error: unable to inspect container: context

Может быть случайная проблема с сетевым подключением? но это должно было быть восстановлено автоматически?

Answer 1

Изучив демон docker на хосте, я увидел, что система вошла в эту проблему:

https://github.com/docker/for-linux/issues/162