microsoft/onnxconverter-common

FP16 conversion yields an unusable model

eddieevt-DXC opened this issue · 2 comments

eddieevt-DXC commented

I'm working with a model in Sagemaker (Resnet50 640x640, size [1, -1, -1, 3]) converted to ONNX. When trying to get more performance out of it by converting it to FP16, the conversion succeeds but trying to run the model gives this error:

E0907 08:27:25.823138 1379 model_lifecycle.cc:626] failed to load 'sagemaker' version 1: Internal: onnx runtime error 1: 
Load model from /models/sagemaker/1/model.onnx failed:Node (StatefulPartitionedCall/map/while_loop) Op (Loop) TypeInferenceError] 
Graph attribute inferencing failed: Node (Resize__59) Op (Resize) [ShapeInferenceError] 
Either `sizes` or `scales` must be provided, but not both of them

Trying out mixed precision instead fails at shape inferencing:

Traceback (most recent call last):
  File "/workspace/fp-16-onnx-converter.py", line 15, in <module>
    model_fp16 = auto_mixed_precision.auto_convert_mixed_precision(model, input_feed, rtol=0.01, atol=0.001, keep_io_types=True)
  File "/usr/local/lib/python3.10/dist-packages/onnxconverter_common/auto_mixed_precision.py", line 80, in auto_convert_mixed_precision
    if not run_attempt(node_names):
  File "/usr/local/lib/python3.10/dist-packages/onnxconverter_common/auto_mixed_precision.py", line 72, in run_attempt
    res1 = get_tensor_values_using_ort(model, feed_dict)
  File "/usr/local/lib/python3.10/dist-packages/onnxconverter_common/auto_mixed_precision.py", line 132, in get_tensor_values_using_ort
    sess = ort.InferenceSession(model.SerializeToString(), sess_options, providers=['CUDAExecutionProvider'])
  File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 383, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 426, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_bytes, False, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Node (StatefulPartitionedCall/map/while_loop) Op (Loop) [TypeInferenceError] Graph attribute inferencing failed: Node (Resize__59) Op (Resize) [ShapeInferenceError] Either `sizes` or `scales` must be provided, but not both of them

It gives the same error with the latest shape inferencing script from Github. I am not sure where I need to post this issue as multiple parts of the ONNX stack seem involved and not working.

skottmckay commented

As the user reports the original model has no issue but the converted model has an invalid Resize node, it appears this is a converter bug.

microsoft/onnxruntime#17447

eddieevt-DXC commented

Tested EfficientDet D2 768x768 and it also exhibits the same behavior.

What I did:

  • Download TF model
  • Convert using tf2onnx
    -- At this point loading the model in an ONNX session works
  • Downcast to FP16 using the same script
    -- Loading the ONNX FP16 model is a session gives the same error