paulstarke/PhaseBetweener

Error on importing trained .onnx into Unity

dj-kefir-siorbacz opened this issue · 7 comments

dj-kefir-siorbacz commented

Hey @pauzii,

I went through the steps described here, I trained the NN using InBetweeningNetwork.py for 16 epochs and got the following .onnx results:

image

But upon drag-n-dropping the .onnx file into Unity I get the following errors:

image

ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) (at <695d1cc93cca45069c528c15c9fdd749>:0)
System.ThrowHelper.ThrowArgumentOutOfRangeException () (at <695d1cc93cca45069c528c15c9fdd749>:0)
Unity.Barracuda.Compiler.IRShapeInferenceHelper.ShapeInference.InferOutputShapeNCHW (Unity.Barracuda.Layer layer, System.Int32[] inputRanks, Unity.Barracuda.TensorShape[] inputShapes) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/ShapeInference/IRShapeInferenceHelper.cs:752)
Unity.Barracuda.Compiler.IRShapeInferenceHelper.ShapeInference.UpdateKnownTensorShapesNCHW (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] ranksByName, System.Collections.Generic.IDictionary`2[System.String,System.Nullable`1[Unity.Barracuda.TensorShape]]& shapesByName) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/ShapeInference/IRShapeInferenceHelper.cs:781)
Unity.Barracuda.Compiler.Passes.IRShapeInferenceAndConstantFusing.FuseShapesIntoConstants (Unity.Barracuda.Model& model, System.Collections.Generic.IDictionary`2[TKey,TValue] shapesByName, System.Collections.Generic.IDictionary`2[TKey,TValue] ranksByName) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/Passes/IRShapeInferenceAndConstantFusing.cs:66)
Unity.Barracuda.Compiler.Passes.IRShapeInferenceAndConstantFusing.Run (Unity.Barracuda.Model& model) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/Passes/IRShapeInferenceAndConstantFusing.cs:23)
Unity.Barracuda.Compiler.Passes.IntermediateToRunnableNHWCPass.Run (Unity.Barracuda.Model& model) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/Passes/IntermediateToRunnableNHWCPass.cs:12)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (Google.Protobuf.CodedInputStream inputStream) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:170)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (System.String filePath) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:83)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Editor/ONNXModelImporter.cs:58)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at <5ad584e208e14caaa9e6b2e6027e9204>:0)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

Asset import failed, "Assets/15.onnx" > ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) (at <695d1cc93cca45069c528c15c9fdd749>:0)
System.ThrowHelper.ThrowArgumentOutOfRangeException () (at <695d1cc93cca45069c528c15c9fdd749>:0)
Unity.Barracuda.Compiler.IRShapeInferenceHelper.ShapeInference.InferOutputShapeNCHW (Unity.Barracuda.Layer layer, System.Int32[] inputRanks, Unity.Barracuda.TensorShape[] inputShapes) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/ShapeInference/IRShapeInferenceHelper.cs:752)
Unity.Barracuda.Compiler.IRShapeInferenceHelper.ShapeInference.UpdateKnownTensorShapesNCHW (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] ranksByName, System.Collections.Generic.IDictionary`2[System.String,System.Nullable`1[Unity.Barracuda.TensorShape]]& shapesByName) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/ShapeInference/IRShapeInferenceHelper.cs:781)
Unity.Barracuda.Compiler.Passes.IRShapeInferenceAndConstantFusing.FuseShapesIntoConstants (Unity.Barracuda.Model& model, System.Collections.Generic.IDictionary`2[TKey,TValue] shapesByName, System.Collections.Generic.IDictionary`2[TKey,TValue] ranksByName) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/Passes/IRShapeInferenceAndConstantFusing.cs:66)
Unity.Barracuda.Compiler.Passes.IRShapeInferenceAndConstantFusing.Run (Unity.Barracuda.Model& model) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/Passes/IRShapeInferenceAndConstantFusing.cs:23)
Unity.Barracuda.Compiler.Passes.IntermediateToRunnableNHWCPass.Run (Unity.Barracuda.Model& model) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Compiler/Passes/IntermediateToRunnableNHWCPass.cs:12)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (Google.Protobuf.CodedInputStream inputStream) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:170)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (System.String filePath) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:83)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Editor/ONNXModelImporter.cs:58)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at <5ad584e208e14caaa9e6b2e6027e9204>:0)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

You can also see that the .onnx didn't get converted to Unity asset:
image

dj-kefir-siorbacz commented

Just to mention, my training code contains changes from here:
#5

paulstarke commented

Seems like Unity Barracuda can not read your model as there are missing information. Can you check in the inspector of the onnx file if the model parameters are stored? Changing the pytorch version might help to fix this. I'm using pytorch 1.12
Also, the warnings in the console might provide more information on what's wrong?

dj-kefir-siorbacz commented

Hi @pauzii ,

I will try your pytorch version. Could you also tell me your onnx version?

Or you could pip freeze > requirements.txt
or even

pip install pipreqs
pipreqs
dj-kefir-siorbacz commented

@pauzii
changing pytorch version didn't work:
image

dj-kefir-siorbacz commented

In this Issue on Barracuda someone says that the problem might be related to reshape operations in the NN

dj-kefir-siorbacz commented

Ok, I fixed it by modifying the source code of Barracuda:
Unity-Technologies/barracuda-release#201 (comment)

Also @pauzii, the preimported models in Unity are on PyTorch 1.9:
image

  • ❤️1
DenisKochetov commented

Ok, I fixed it by modifying the source code of Barracuda: Unity-Technologies/barracuda-release#201 (comment)

Also @pauzii, the preimported models in Unity are on PyTorch 1.9: image

Man you are the hero! It works now

  • ❤️1