wandb logging PermissionError и OSError
Описание:
При проведении экспериментов с использованием
Weights and Biases
(wandb), я иногда получаюPermissionError
для Pythonlogging
библиотека иOSError
для доступа к сертификату CA TLS.У меня была следующая трассировка стека, повторяющаяся много раз с разными типами «сообщений». Я не могу различить порядок операций, но предполагаю, что к сертификату нельзя получить доступ, и это приводит к сбою скрипта, но я не знаю, почему это случается только иногда.
Если это актуально, я проводил эксперименты на сервере Ubuntu, аутентифицированном через Kerberos.
Что я пробовал:
- Я вручную проверил сертификат CA и более чем в половине случаев могу успешно проводить эксперименты. Поэтому я не думаю , что это так же , как это или это .
Трассировки стека
Message: 'handle_request: stop_status' [854/1967]Arguments: ()
--- Logging error ---
Traceback (most recent call last):
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/logging/__init__.py", line 1085, in emit
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/logging/__init__.py", line 1065, in flush
PermissionError: [Errno 13] Permission denied
Call stack:
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/threading.py", line 890, in _bootstrap
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 54, in run
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 95, in _run
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 280, in _process
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 175, in send
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 183, in send_request
Message: 'send_request: stop_status'
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 681, in check_stop_requested
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/lib/retry.py", line 102, in __call__
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 127, in execute
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/requests/api.py", line 119, in post
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/requests/api.py", line 61, in request
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/requests/adapters.py", line 416, in send
File "/home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /home/some_user/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/certifi/cacert.pem