Anisble Tower Job не работает, если размер extra-vars больше
У меня есть работа в Ansible Tower. Все входные данные для задания передаются в экстра-переменных (в формате JSON)
Когда размер дополнительных переменных мал, работа выполняется нормально. Но когда он большой, например, 800 КБ или 160000 строк экстра-переменных, задание завершается с ошибкой.
Сообщение об ошибке:
[2018-08-30 07:13:27,641: DEBUG/Worker-6835] Database error updating Job, retrying in 5 seconds (retry #1): could not receive data from server: Bad file descriptor
[2018-08-30 07:13:32,649: DEBUG/Worker-6835] Database error updating Job, retrying in 5 seconds (retry #2): could not receive data from server: Bad file descriptor
[2018-08-30 07:13:37,653: DEBUG/Worker-6835] Database error updating Job, retrying in 5 seconds (retry #3): could not receive data from server: Bad file descriptor
[2018-08-30 07:13:42,660: DEBUG/Worker-6835] Database error updating Job, retrying in 5 seconds (retry #4): could not receive data from server: Bad file descriptor
[2018-08-30 07:13:47,667: DEBUG/Worker-6835] Database error updating Job, retrying in 5 seconds (retry #5): could not receive data from server: Bad file descriptor
[2018-08-30 07:13:52,674: DEBUG/Worker-6835] Database error updating Job, retrying in 5 seconds (retry #6): could not receive data from server: Bad file descriptor
[2018-08-30 07:13:52,682: WARNING/Worker-6835] Traceback (most recent call last):
[2018-08-30 07:13:52,683: WARNING/Worker-6835] File "/usr/lib64/python2.6/logging/handlers.py", line 76, in emit
[2018-08-30 07:13:52,687: WARNING/Worker-6835] if self.shouldRollover(record):
[2018-08-30 07:13:52,687: WARNING/Worker-6835] File "/usr/lib64/python2.6/logging/handlers.py", line 150, in shouldRollover
[2018-08-30 07:13:52,687: WARNING/Worker-6835] self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
[2018-08-30 07:13:52,687: WARNING/Worker-6835] IOError: [Errno 9] Bad file descriptor
[2018-08-30 07:13:52,674: ERROR/Worker-6835] Failed to update Job after 5 retries.
[2018-08-30 07:13:52,724: WARNING/Worker-6835] /usr/lib/python2.6/site-packages/awx/lib/site-packages/celery/app/trace.py:348: RuntimeWarning: Exception raised outside body: OperationalError('could not receive data from server: Bad file descriptor\n',):
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/celery/app/trace.py", line 253, in trace_task
I, R, state, retval = on_error(task_request, exc, uuid)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/celery/app/trace.py", line 201, in on_error
R = I.handle_error_state(task, eager=eager)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/celery/app/trace.py", line 85, in handle_error_state
}[self.state](task, store_errors=store_errors)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/celery/app/trace.py", line 118, in handle_failure
req.id, exc, einfo.traceback, request=req,
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/celery/backends/base.py", line 120, in mark_as_failure
traceback=traceback, request=request)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/celery/backends/base.py", line 251, in store_result
request=request, **kwargs)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/djcelery/backends/database.py", line 29, in _store_result
traceback=traceback, children=self.current_task_children(request),
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/djcelery/managers.py", line 42, in _inner
return fun(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/djcelery/managers.py", line 181, in store_result
'meta': {'children': children}})
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/djcelery/managers.py", line 87, in update_or_create
return get_queryset(self).update_or_create(**kwargs)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/djcelery/managers.py", line 70, in update_or_create
obj, created = self.get_or_create(**kwargs)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/models/query.py", line 376, in get_or_create
return self.get(**lookup), False
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/models/query.py", line 304, in get
num = len(clone)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/models/query.py", line 77, in __len__
self._fetch_all()
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/models/query.py", line 857, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
OperationalError: could not receive data from server: Bad file descriptor
exc, exc_info.traceback)))
[2018-08-30 07:13:52,743: ERROR/Worker-6835] Pool process <Worker(Worker-6835, started daemon)> error: OSError(9, 'Bad file descriptor')
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/billiard/pool.py", line 288, in run
sys.exit(self.workloop(pid=pid))
File "/usr/lib/python2.6/site-packages/awx/lib/site-packages/billiard/pool.py", line 372, in workloop
put((READY, (job, i, (False, einfo), inqW_fd)))
OSError: [Errno 9] Bad file descriptor
Та же самая Работа с большими экстра-варами отлично работает в CLS Ansbile.
У меня нет доступа администратора к The Ansible Tower или локальной установки Tower, поэтому я не смог отладить основную причину. У нас есть какое-то понимание этого?