Как настроить тайм-аут функции init() входного сценария AzureML InferenceConfig?
Я упаковываю модель AzureML, используя Model.package
метод. Мой сценарий входа должен прочитать и обработать некоторые данные в своемinit()
, поэтому это может занять некоторое время.
MODEL_NAME = 'abv'
ws = Workspace.from_config()
model = Model(ws, name=MODEL_NAME, version=2)
myenv = Environment.from_conda_specification(name='myenv',
file_path='./env.yaml')
inference_config = InferenceConfig(entry_script='./entry_script.py',
environment=myenv)
package = Model.package(ws, [model],
inference_config)
package.wait_for_creation(show_output=True)
После запуска контейнера докеров, если появляется следующая ошибка:
...
WORKER TIMEOUT (pid:39)
worker timed out, killing gunicorn
User's init function failed
Encountered Exception Traceback (most recent call last):
File "/var/azureml-server/aml_blueprint.py", line 162, in register
main.init()
File "/var/azureml-app/main.py", line 35, in init
driver_module.init()
File "/var/azureml-app/entry_script.py", line 222, in init
model = MyModel(df, wv)
File "/var/azureml-app/entry_script.py", line 37, in __init__
df["TRANSLATED_MANAGEMENT_SUMMARY"], tf_summary)
File "/var/azureml-app/entry_script.py", line 158, in _restricted_token_embeddings
stem(token.text).lower() for token in self.nlp(document)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/spacy/language.py", line 439, in __call__
doc = proc(doc, **component_cfg.get(name, {}))
File "pipes.pyx", line 396, in spacy.pipeline.pipes.Tagger.__call__
File "pipes.pyx", line 415, in spacy.pipeline.pipes.Tagger.predict
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
return self.predict(x)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/feed_forward.py", line 40, in predict
X = layer(X)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
return self.predict(x)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/api.py", line 310, in predict
X = layer(layer.ops.flatten(seqs_in, pad=pad))
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
return self.predict(x)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/feed_forward.py", line 40, in predict
X = layer(X)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
return self.predict(x)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 131, in predict
y, _ = self.begin_update(X, drop=None)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/api.py", line 379, in uniqued_fwd
Y_uniq, bp_Y_uniq = layer.begin_update(X_uniq, drop=drop)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/feed_forward.py", line 46, in begin_update
X, inc_layer_grad = layer.begin_update(X, drop=drop)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/layernorm.py", line 65, in begin_update
N, mu, var = _get_moments(self.ops, X)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/layernorm.py", line 108, in _get_moments
var = X.var(axis=1, keepdims=True) + 1e-08
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/numpy/core/_methods.py", line 195, in _var
x = um.multiply(x, x, out=x)
File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/gunicorn/workers/base.py", line 195, in handle_abort
self.cfg.worker_abort(self)
File "/var/azureml-server/gunicorn_conf.py", line 48, in worker_abort
sys.exit(3)
SystemExit: 3
Я предполагаю, что это происходит, когда init()
функция входного скрипта занимает слишком много времени (иногда работает нормально, кстати). Как я могу изменить тайм-аут?