Проведено Gitlab show 502 из ниоткуда
Сегодня мой собственный хостинг Gitlab внезапно перестал работать. Он начал возвращать страницу с ошибкой 500. Затем я перезапустил gitlab с gitlab-ctl restart и теперь он показывает ошибку 502.
Он работал нормально, но, заглянув в журналы с gitlab-ctl tail Я вижу бесконечный цикл этого журнала:
==> /var/log/gitlab/postgresql/current <==
2020-08-07_15:09:26.73662 FATAL: could not write lock file "postmaster.pid": No space left on device
==> /var/log/gitlab/prometheus/current <==
2020-08-07_15:09:26.40303 level=info ts=2020-08-07T15:09:26.402Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
2020-08-07_15:09:26.40311 level=info ts=2020-08-07T15:09:26.403Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.13.1, branch=master, revision=)"
2020-08-07_15:09:26.40318 level=info ts=2020-08-07T15:09:26.403Z caller=main.go:333 build_context="(go=go1.12.13, user=GitLab-Omnibus, date=)"
2020-08-07_15:09:26.40323 level=info ts=2020-08-07T15:09:26.403Z caller=main.go:334 host_details="(Linux 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 genuino-gitlab (none))"
2020-08-07_15:09:26.40329 level=info ts=2020-08-07T15:09:26.403Z caller=main.go:335 fd_limits="(soft=50000, hard=50000)"
2020-08-07_15:09:26.40335 level=info ts=2020-08-07T15:09:26.403Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
2020-08-07_15:09:26.40438 unexpected fault address 0x7fb3ba4c5000
2020-08-07_15:09:26.40439 fatal error: fault
2020-08-07_15:09:26.41175 [signal SIGBUS: bus error code=0x2 addr=0x7fb3ba4c5000 pc=0x45fa98]
2020-08-07_15:09:26.41177
2020-08-07_15:09:26.41177 goroutine 1 [running]:
2020-08-07_15:09:26.41178 runtime.throw(0x24e5fcb, 0x5)
2020-08-07_15:09:26.41178 /usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc000541458 sp=0xc000541428 pc=0x42f462
2020-08-07_15:09:26.41179 runtime.sigpanic()
2020-08-07_15:09:26.41179 /usr/local/go/src/runtime/signal_unix.go:387 +0x47e fp=0xc000541488 sp=0xc000541458 pc=0x444a4e
2020-08-07_15:09:26.41180 runtime.memmove(0x7fb3ba4c5000, 0x24e3526, 0x1)
2020-08-07_15:09:26.41181 /usr/local/go/src/runtime/memmove_amd64.s:146 +0x108 fp=0xc000541490 sp=0xc000541488 pc=0x45fa98
2020-08-07_15:09:26.41182 github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffca2f98eb6, 0x1f, 0x14, 0x2aaff60, 0xc0001f68d0, 0x2aaff60)
2020-08-07_15:09:26.41183 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/promql/query_logger.go:118 +0x363 fp=0xc0005415d8 sp=0xc000541490 pc=0x1cd0c23
2020-08-07_15:09:26.41194 main.main()
2020-08-07_15:09:26.41194 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:364 +0x5477 fp=0xc000541f98 sp=0xc0005415d8 pc=0x1e1e6f7
2020-08-07_15:09:26.41195 runtime.main()
2020-08-07_15:09:26.41196 /usr/local/go/src/runtime/proc.go:200 +0x20c fp=0xc000541fe0 sp=0xc000541f98 pc=0x430dcc
2020-08-07_15:09:26.41197 runtime.goexit()
2020-08-07_15:09:26.41197 /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x45e8b1
2020-08-07_15:09:26.41198
2020-08-07_15:09:26.41198 goroutine 25 [select]:
2020-08-07_15:09:26.41200 go.opencensus.io/stats/view.(*worker).start(0xc000154aa0)
2020-08-07_15:09:26.41201 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
2020-08-07_15:09:26.41202 created by go.opencensus.io/stats/view.init.0
2020-08-07_15:09:26.41202 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57
2020-08-07_15:09:26.41202
2020-08-07_15:09:26.41203 goroutine 6 [syscall]:
2020-08-07_15:09:26.41203 os/signal.signal_recv(0x0)
2020-08-07_15:09:26.41204 /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
2020-08-07_15:09:26.41204 os/signal.loop()
2020-08-07_15:09:26.41204 /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
2020-08-07_15:09:26.41205 created by os/signal.init.0
2020-08-07_15:09:26.41205 /usr/local/go/src/os/signal/signal_unix.go:29 +0x41
2020-08-07_15:09:26.41205
2020-08-07_15:09:26.41205 goroutine 91 [GC assist marking]:
2020-08-07_15:09:26.41206 time.NewTicker(0x2540be400, 0x0)
2020-08-07_15:09:26.41207 /usr/local/go/src/time/tick.go:28 +0x48
2020-08-07_15:09:26.41207 github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc0006c4980)
2020-08-07_15:09:26.41208 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/pkg/logging/dedupe.go:70 +0x45
2020-08-07_15:09:26.41208 created by github.com/prometheus/prometheus/pkg/logging.Dedupe
2020-08-07_15:09:26.41209 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/pkg/logging/dedupe.go:60 +0xcf
2020-08-07_15:09:26.41209
2020-08-07_15:09:26.41209 goroutine 92 [semacquire]:
2020-08-07_15:09:26.41210 time.NewTicker(0x2540be400, 0x0)
2020-08-07_15:09:26.41210 /usr/local/go/src/time/tick.go:28 +0x48
2020-08-07_15:09:26.41211 github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc0004c5a70)
2020-08-07_15:09:26.41211 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/remote/write.go:78 +0x34
2020-08-07_15:09:26.41212 created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage
2020-08-07_15:09:26.41212 /var/cache/omnibus/src/prometheus/src/github.com/prometheus/prometheus/storage/remote/write.go:73 +0x102
==> /var/log/gitlab/gitlab-rails/sidekiq.log <==
could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.2/lib/pg.rb:58:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.2/lib/pg.rb:58:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.2/lib/pg.rb:58:in `connect'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:46:in `postgresql_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:889:in `new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:933:in `checkout_new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:912:in `try_to_checkout_new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:873:in `acquire_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:595:in `checkout'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:439:in `connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:1121:in `retrieve_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_handling.rb:238:in `retrieve_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_handling.rb:206:in `connection'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/database_config.rb:37:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `block in load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:667:in `block in load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:666:in `load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:624:in `block (2 levels) in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:623:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:623:in `block in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:50:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/application.rb:363:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
/opt/gitlab/embedded/bin/sidekiq:23:in `load'
/opt/gitlab/embedded/bin/sidekiq:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
==> /var/log/gitlab/gitaly/current <==
{"gitaly":2235,"level":"error","msg":"wrapper for gitaly shutting down","time":"2020-08-07T15:09:25Z","wrapper":2222}
{"level":"info","msg":"Wrapper started","time":"2020-08-07T15:09:26Z","wrapper":2276}
{"level":"info","msg":"finding gitaly","pid_file":"/var/opt/gitlab/gitaly/gitaly.pid","time":"2020-08-07T15:09:26Z","wrapper":2276}
{"level":"info","msg":"spawning a process","time":"2020-08-07T15:09:26Z","wrapper":2276}
{"gitaly":2290,"level":"info","msg":"monitoring gitaly","time":"2020-08-07T15:09:26Z","wrapper":2276}
==> /var/log/gitlab/sidekiq/current <==
could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.2/lib/pg.rb:58:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.2/lib/pg.rb:58:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.2.2/lib/pg.rb:58:in `connect'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:46:in `postgresql_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:889:in `new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:933:in `checkout_new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:912:in `try_to_checkout_new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:873:in `acquire_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:595:in `checkout'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:439:in `connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:1121:in `retrieve_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_handling.rb:238:in `retrieve_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_handling.rb:206:in `connection'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/database_config.rb:37:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `block in load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:667:in `block in load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:666:in `load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:624:in `block (2 levels) in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:623:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:623:in `block in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:50:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/application.rb:363:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/dependencies.rb:325:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
/opt/gitlab/embedded/bin/sidekiq:23:in `load'
/opt/gitlab/embedded/bin/sidekiq:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
==> /var/log/gitlab/unicorn/current <==
2020-08-07_15:09:26.91924 master failed to start, check stderr log for details
==> /var/log/gitlab/grafana/current <==
2020-08-07_15:09:27.71079 t=2020-08-07T17:09:27+0200 lvl=info msg="Starting Grafana" logger=server version=6.3.5 commit=67bad72 branch=HEAD compiled=2019-09-02T11:34:49+0200
2020-08-07_15:09:27.71309 t=2020-08-07T17:09:27+0200 lvl=info msg="Config loaded from" logger=settings file=/var/opt/gitlab/grafana/conf/defaults.ini
2020-08-07_15:09:27.71312 t=2020-08-07T17:09:27+0200 lvl=info msg="Config loaded from" logger=settings file=/var/opt/gitlab/grafana/grafana.ini
2020-08-07_15:09:27.71314 t=2020-08-07T17:09:27+0200 lvl=info msg="Path Home" logger=settings path=/var/opt/gitlab/grafana
2020-08-07_15:09:27.71316 t=2020-08-07T17:09:27+0200 lvl=info msg="Path Data" logger=settings path=/var/opt/gitlab/grafana/data
2020-08-07_15:09:27.71318 t=2020-08-07T17:09:27+0200 lvl=info msg="Path Logs" logger=settings path=/var/log/gitlab/grafana
2020-08-07_15:09:27.71320 t=2020-08-07T17:09:27+0200 lvl=info msg="Path Plugins" logger=settings path=/var/opt/gitlab/grafana/data/plugins
2020-08-07_15:09:27.71322 t=2020-08-07T17:09:27+0200 lvl=info msg="Path Provisioning" logger=settings path=/var/opt/gitlab/grafana/provisioning
2020-08-07_15:09:27.71324 t=2020-08-07T17:09:27+0200 lvl=info msg="App mode production" logger=settings
2020-08-07_15:09:27.71375 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing SqlStore" logger=server
2020-08-07_15:09:27.71380 t=2020-08-07T17:09:27+0200 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
2020-08-07_15:09:27.71443 t=2020-08-07T17:09:27+0200 lvl=info msg="Starting DB migration" logger=migrator
2020-08-07_15:09:27.73333 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing HTTPServer" logger=server
2020-08-07_15:09:27.73376 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing InternalMetricsService" logger=server
2020-08-07_15:09:27.73379 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing RemoteCache" logger=server
2020-08-07_15:09:27.73381 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing QuotaService" logger=server
2020-08-07_15:09:27.73384 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing PluginManager" logger=server
2020-08-07_15:09:27.73387 t=2020-08-07T17:09:27+0200 lvl=info msg="Starting plugin search" logger=plugins
2020-08-07_15:09:27.78100 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing RenderingService" logger=server
2020-08-07_15:09:27.78107 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing DatasourceCacheService" logger=server
2020-08-07_15:09:27.78109 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing HooksService" logger=server
2020-08-07_15:09:27.78111 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing LoginService" logger=server
2020-08-07_15:09:27.78115 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing SearchService" logger=server
2020-08-07_15:09:27.78118 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing ServerLockService" logger=server
2020-08-07_15:09:27.78120 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing TracingService" logger=server
2020-08-07_15:09:27.78123 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing UsageStatsService" logger=server
2020-08-07_15:09:27.78126 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing UserAuthTokenService" logger=server
2020-08-07_15:09:27.78128 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing CleanUpService" logger=server
2020-08-07_15:09:27.78130 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing NotificationService" logger=server
2020-08-07_15:09:27.78190 t=2020-08-07T17:09:27+0200 lvl=info msg="Initializing provisioningServiceImpl" logger=server
2020-08-07_15:09:27.78277 t=2020-08-07T17:09:27+0200 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Datasource provisioning error: database or disk is full"
А также некоторые службы gitlab не работают. Если я напечатаюgitlab-ctl status Я вижу это:
run: alertmanager: (pid 23841) 547s; run: log: (pid 957) 16556s
run: crond: (pid 23878) 547s; run: log: (pid 984) 16556s
run: gitaly: (pid 11027) 2s; run: log: (pid 969) 16556s
run: gitlab-exporter: (pid 23916) 545s; run: log: (pid 966) 16556s
run: gitlab-workhorse: (pid 23945) 545s; run: log: (pid 970) 16556s
down: grafana: 1s, normally up, want up; run: log: (pid 958) 16556s
run: logrotate: (pid 24007) 544s; run: log: (pid 989) 16556s
run: mailroom: (pid 24211) 538s; run: log: (pid 995) 16556s
run: nginx: (pid 24223) 538s; run: log: (pid 964) 16556s
run: node-exporter: (pid 24239) 537s; run: log: (pid 985) 16556s
run: postgres-exporter: (pid 24262) 537s; run: log: (pid 972) 16556s
down: postgresql: 1s, normally up, want up; run: log: (pid 961) 16556s
down: prometheus: 1s, normally up, want up; run: log: (pid 962) 16556s
run: redis: (pid 990) 16556s, got TERM; run: log: (pid 986) 16556s
run: redis-exporter: (pid 26190) 483s; run: log: (pid 988) 16556s
run: registry: (pid 26211) 483s; run: log: (pid 978) 16556s
run: sidekiq: (pid 10635) 13s; run: log: (pid 968) 16556s
run: unicorn: (pid 11069) 0s; run: log: (pid 971) 16556s
Если я сделаю df
root@genuino-gitlab:/# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 1956564 0 1956564 0% /dev
tmpfs 394316 15356 378960 4% /run
/dev/sda1 39320220 37855160 0 100% /
tmpfs 1971576 0 1971576 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 1971576 0 1971576 0% /sys/fs/cgroup
/dev/sdb 10255636 10081700 0 100% /mnt/HC_Volume_4756712
overlay 39320220 37855160 0 100% /var/lib/docker/overlay2/b2bb0a394c979a96ae9f7e9dbab740b8f903f948c2c0ee91a5f05e4f8938e1a0/merged
overlay 39320220 37855160 0 100% /var/lib/docker/overlay2/22c416ad26999a2b2f5578f60a0777eef33a344fc3b94b74b38a1ba440ff3d26/merged
tmpfs 394312 0 394312 0% /run/user/0