Произошла ошибка tenorRT ('*** Обнаружено разрушение стека ***: Python прекращен') при преобразовании замороженного графа тензорного потока

Question

Я использую TensorFlow версии 1.7, cuda8.0, Python 2.7 для оптимизации модели TensorFlow VGG-16 с помощью кода ниже:-

vgg_checkpoint = 'trained_models/models/vgg16/vgg_16.ckpt'
image_decoded = tf.placeholder(dtype=tf.float32, shape=[None, None, None, 3], name="input")
logits, _ = vgg_16(image_decoded, is_training=False)
saver = tf.train.Saver()
saver.restore(sess, vgg_checkpoint)

input_graph_def = sess.graph.as_graph_def()
output_graph_def = tf.graph_util.convert_variables_to_constants(
    sess,
    input_graph_def,
    ['vgg_16/fc8/squeezed'],
)
frozen_graph = tf.graph_util.remove_training_nodes(output_graph_def)
output_graph = "../model/tensorRT_check.pb"
with tf.gfile.GFile(output_graph, "wb") as f:
    f.write(frozen_graph.SerializeToString())
trt_graph = trt.create_inference_graph(
    input_graph_def=frozen_graph,
    outputs=['input', 'vgg_16/fc8/squeezed'],
    max_batch_size=1,
    max_workspace_size_bytes=1 << 25,
    precision_mode='FP16',
    minimum_segment_size=50
)

Обучение модели выполнено в TitanX (12 ГБ), и код запускался на том же.

2018-07-19 19:00:34.093716: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2018-07-19 19:00:34.149018: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2018-07-19 19:00:34.149335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076 pciBusID: 0000:01:00.0 totalMemory: 11.92GiB freeMemory: 11.33GiB 2018-07-19 19:00:34.149350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0 2018-07-19 19:00:34.395658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-07-19 19:00:34.395685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 2018-07-19 19:00:34.395691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N 2018-07-19 19:00:34.395901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7324 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0, compute capability: 5.2) Converted 32 variables to const ops. 2018-07-19 19:00:41.076819: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 1 *** stack smashing detected ***: python terminated Aborted (core dumped)