ZED Mini-Unity Object Detection error: Optimization process Timeout even after optimzing the AI models outside of Unity, using the ZED Diagnostic tool
aaronlsmiles opened this issue · 3 comments
Preliminary Checks
- This issue is not a duplicate. Before opening a new issue, please search existing issues.
- This issue is not a question, feature request, or anything other than a bug report directly related to this project.
Description
Optimization process times out even after optimzing the AI models outside of Unity, using the ZED Diagnostic tool, as instructed in the literature.
I have uninstalled and reinstalled, trying different versions of Unity (2021, 2022, etc.) and CUDA (11.8, 12.1, etc.), but the issue is always the same and the ZED Diagnostic tool shows everything with a green tick, except the AI Models, which show as unoptimised.
I also can't find the optimised AI models in any folder (e.g., C:\Program Files (x86)\ZED SDK, or C:\ProgramData\Stereolabs\resources (which is empty)).
Steps to Reproduce
I followed the steps for ZED 2 here, although I am using ZED Mini the documentation says object detection works with mini too.
- Plug in ZED Mini and load Unity
- Create a new scene and delete the Main Camera
- In the Project window, go to ZED -> Prefabs and drag ZED_Rig_Mono into the Hierarchy
- Select the new ZED_Rig_Mono in the Hierarchy
- In the Motion Tracking section, make sure Estimate Initial Position is checked. This enables floor detection
- Create a new empty GameObject in the Hierarchy and rename it “3D Object Visualizer”
- Add the ZED3DObjectVisualizer component to it
- In the Projects window, go to ZED -> Examples -> Object Detection -> Prefabs
- Drag the 3D Bounding Box prefab into Bounding Box Prefab in the Inspector
Expected Result
After a short initialization period, the app should pause for 10-20 seconds as the object detection module loads. Once it does, I should be able to step into the ZED’s view and see myself enclosed in a box or have objects that the model is trained on in the scene that 3D bounding boxes appear around.
Actual Result
Error: Optimization process Timeout. Please try to optimze the AI models outside of Unity, using the ZED Diagnostic tool
UnityEngine.Debug:LogError (object)
ZEDManager/d__379:MoveNext () (at Assets/ZED/SDK/Helpers/Scripts/ZEDManager.cs:3037)
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)
ZED Camera model
ZED Mini
Environment
Edition Windows 11 Home
Version 22H2
Installed on 28/07/2023
OS build 22621.2134
Experience Windows Feature Experience Pack 1000.22659.1000.0
Processor 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz 2.30 GHz
Installed RAM 32.0 GB (31.7 GB usable)
Device ID 89E4F234-8C60-4866-B422-24FBAD207069
Product ID 00326-10000-00000-AA011
System type 64-bit operating system, x64-based processor
ZED_Diagnostic_Results:
{
"AI Models": {
"info": [
"MULTI CLASS DETECTION is not optimized",
"MULTI CLASS MEDIUM DETECTION is not optimized",
"MULTI CLASS ACCURATE DETECTION is not optimized",
"HUMAN BODY FAST DETECTION is not optimized",
"HUMAN BODY MEDIUM DETECTION is not optimized",
"HUMAN BODY ACCURATE DETECTION is not optimized",
"HUMAN BODY 38 FAST DETECTION is not optimized",
"HUMAN BODY 38 MEDIUM DETECTION is not optimized",
"HUMAN BODY 38 ACCURATE DETECTION is not optimized",
"PERSON HEAD DETECTION is not optimized",
"PERSON HEAD ACCURATE DETECTION is not optimized",
"REID ASSOCIATION is not optimized",
"NEURAL DEPTH is not optimized"
]
},
"Camera Test": {
"InternalDevicesCount": 2,
"ZEDCount": 1,
"camera": {
"Device ID": 1,
"Firmware": 1523,
"Init Output": "OK",
"Model": "ZED Mini",
"Serial Number": 16396036,
"valid": [
"<b>Camera: </b>ZED Mini",
"<b>Firmware: </b>1523"
]
},
"resolutions": {
"HD1080@15": {
"fps (input)": 15,
"fps (output)": 15,
"height (input)": 1080,
"height (output)": 1080,
"initialization": "OK",
"status": true,
"width (input)": 1920,
"width (output)": 1920
},
"HD1080@30": {
"fps (input)": 30,
"fps (output)": 30,
"height (input)": 1080,
"height (output)": 1080,
"initialization": "OK",
"status": true,
"width (input)": 1920,
"width (output)": 1920
},
"HD2K@15": {
"fps (input)": 15,
"fps (output)": 15,
"height (input)": 1242,
"height (output)": 1242,
"initialization": "OK",
"status": true,
"width (input)": 2208,
"width (output)": 2208
},
"HD720@15": {
"fps (input)": 15,
"fps (output)": 15,
"height (input)": 720,
"height (output)": 720,
"initialization": "OK",
"status": true,
"width (input)": 1280,
"width (output)": 1280
},
"HD720@30": {
"fps (input)": 30,
"fps (output)": 30,
"height (input)": 720,
"height (output)": 720,
"initialization": "OK",
"status": true,
"width (input)": 1280,
"width (output)": 1280
},
"HD720@60": {
"dropCount": 0,
"fps (input)": 60,
"fps (output)": 59,
"height (input)": 720,
"height (output)": 720,
"imageCount": 501,
"initialization": "OK",
"status": true,
"tearingCount": 0,
"width (input)": 1280,
"width (output)": 1280
},
"VGA@100": {
"fps (input)": 100,
"fps (output)": 91,
"height (input)": 376,
"height (output)": 376,
"initialization": "OK",
"status": true,
"width (input)": 672,
"width (output)": 672
},
"VGA@15": {
"fps (input)": 15,
"fps (output)": 15,
"height (input)": 376,
"height (output)": 376,
"initialization": "OK",
"status": true,
"width (input)": 672,
"width (output)": 672
},
"VGA@30": {
"fps (input)": 30,
"fps (output)": 30,
"height (input)": 376,
"height (output)": 376,
"initialization": "OK",
"status": true,
"width (input)": 672,
"width (output)": 672
},
"VGA@60": {
"fps (input)": 60,
"fps (output)": 47,
"height (input)": 376,
"height (output)": 376,
"initialization": "OK",
"status": true,
"width (input)": 672,
"width (output)": 672
}
},
"sensors": {
"Firmware": 517,
"Init Output": "OK",
"Model": "ZED Mini (MCU)",
"Serial Number": 16396036
}
},
"Devices": {
"ASMediaFound": false,
"CorruptedFirmware": false,
"MCUDetected": true,
"OC_Cam_Detected": false,
"OtherUSBDevices": [
{
"USBMode": 3,
"USB_path": "/19",
"bDescriptorType": 1,
"bDeviceProtocol": 0,
"bLength": 18,
"bMaxPacketSize0": 9,
"bNumConfigurations": 1,
"bcdDevice": "2.4",
"bcdUSB": "3.0",
"bcdUSBClass": 0,
"bcdUSBSubClass": 0,
"busNumber": 2,
"iManufacturer": 1,
"iProduct": 2,
"iSerial": 3,
"idProduct": 790,
"idVendor": 3034
}
],
"USBList": [
{
"USB_path": "/12",
"idProduct": "0x6005",
"idVendor": "0x048d"
},
{
"USB_path": "/12",
"idProduct": "0x6005",
"idVendor": "0x048d"
},
{
"USB_path": "/12",
"idProduct": "0x6005",
"idVendor": "0x048d"
},
{
"USBMode": 3,
"USB_path": "/17",
"bDescriptorType": 1,
"bDeviceProtocol": 1,
"bLength": 18,
"bMaxPacketSize0": 9,
"bNumConfigurations": 1,
"bcdDevice": "1.0",
"bcdUSB": "3.0",
"bcdUSBClass": 239,
"bcdUSBSubClass": 2,
"busNumber": 2,
"device": "ZED mini",
"iManufacturer": 1,
"iProduct": 2,
"iSerial": 0,
"idProduct": "0xf682",
"idVendor": "0x2b03"
},
{
"USBMode": 2,
"USB_path": "/1",
"bDescriptorType": 1,
"bDeviceProtocol": 0,
"bLength": 18,
"bMaxPacketSize0": 64,
"bNumConfigurations": 1,
"bcdDevice": "2.5",
"bcdUSB": "2.0",
"bcdUSBClass": 0,
"bcdUSBSubClass": 0,
"busNumber": 2,
"device": "ZED mini MCU",
"iManufacturer": 1,
"iProduct": 2,
"iSerial": 3,
"idProduct": "0xf681",
"idVendor": "0x2b03"
},
{
"USBMode": 3,
"USB_path": "/17",
"bDescriptorType": 1,
"bDeviceProtocol": 1,
"bLength": 18,
"bMaxPacketSize0": 9,
"bNumConfigurations": 1,
"bcdDevice": "1.0",
"bcdUSB": "3.0",
"bcdUSBClass": 239,
"bcdUSBSubClass": 2,
"busNumber": 2,
"device": "ZED mini",
"iManufacturer": 1,
"iProduct": 2,
"iSerial": 0,
"idProduct": "0xf682",
"idVendor": "0x2b03"
},
{
"USB_path": "/19",
"idProduct": "0x0316",
"idVendor": "0x0bda"
},
{
"USB_path": "/11",
"idProduct": "0xce00",
"idVendor": "0x048d"
},
{
"USB_path": "/4",
"idProduct": "0xc52b",
"idVendor": "0x046d"
},
{
"USB_path": "/4",
"idProduct": "0xc52b",
"idVendor": "0x046d"
},
{
"USB_path": "/14",
"idProduct": "0x0029",
"idVendor": "0x8087"
},
{
"USB_path": "/13",
"idProduct": "0xb71a",
"idVendor": "0x04f2"
},
{
"USB_path": "/13",
"idProduct": "0xb71a",
"idVendor": "0x04f2"
}
],
"USBMode": 3,
"ZED Camera Module Detected": 63106,
"ZED MCU Module Detected": 63105,
"ZEDDetected": true,
"captureDevices": [
],
"controllers": [
{
"CompatibleIDs": "PCI\\VEN_8086&DEV_43ED&REV_11",
"Description": "USB xHCI Compliant Host Controller",
"DeviceName": "Intel(R) USB 3.20 eXtensible Host Controller - 1.20 (Microsoft)",
"Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0001",
"Location": "PCI bus 0, device 20, function 0",
"Manufacturer": "Generic USB xHCI Host Controller",
"Name": "PCI",
"PhysicalName": "\\Device\\NTPNP_PCI0009"
},
{
"CompatibleIDs": "",
"Description": "USB Root Hub (USB 3.0)",
"DeviceName": "",
"Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0003",
"Location": "",
"Manufacturer": "(Standard USB HUBs)",
"Name": "USB",
"PhysicalName": "\\Device\\USBPDO-0"
},
{
"CompatibleIDs": "PCI\\VEN_8086&DEV_9A17&REV_05",
"Description": "USB xHCI Compliant Host Controller",
"DeviceName": "Intel(R) USB 3.10 eXtensible Host Controller - 1.20 (Microsoft)",
"Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0000",
"Location": "PCI bus 0, device 13, function 0",
"Manufacturer": "Generic USB xHCI Host Controller",
"Name": "PCI",
"PhysicalName": "\\Device\\NTPNP_PCI0007"
},
{
"CompatibleIDs": "",
"Description": "USB Root Hub (USB 3.0)",
"DeviceName": "",
"Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0002",
"Location": "",
"Manufacturer": "(Standard USB HUBs)",
"Name": "USB",
"PhysicalName": "\\Device\\USBPDO-1"
}
],
"valid": [
"<b>USB Bandwidth: </b> OK"
]
},
"Graphics Card": {
"deviceCount": 1,
"deviceDriverVersion": 12010,
"devices": [
{
"arch": "Ampere",
"computeCapability": "8.6",
"cores": 6144,
"name": "NVIDIA GeForce RTX 3080 Laptop GPU",
"totalMemoryBytes": "17179344896",
"totalMemoryMB": 16383.5
}
],
"initResult": 0,
"valid": [
"<b>Graphics card: </b> NVIDIA GeForce RTX 3080 Laptop GPU"
]
},
"Processor": {
"OS": "Windows 10 FallCreators",
"avxSupported": true,
"coreCount": "8",
"cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
"datetime": "2023-09-07 00:11:51",
"hyperThreading": "1",
"motherboard": "GM5TN8Y, TongFang",
"ramCapacity": "31.7471",
"ramUsage": "0",
"ramUsed": "16.697",
"threadCount": "16",
"valid": [
"<b>Processor: </b> 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
"<b>Motherboard: </b> GM5TN8Y, TongFang"
]
},
"ZED SDK": {
"CUDA Toolkit version": "V12.1.66 -->located in C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/bin/nvcc.exe",
"CUDA loading": false,
"ZED SDK Version (Diag)": "4.0.6",
"ZED SDK Version (RT)": "4.0.6",
"ai": {
"checkAI": true,
"cudaVersion": 12010,
"cudnnCudartVersion": 12000,
"cudnnVersion": 8801,
"cudnnVersionExpected": 8801,
"tensorVersion": 8600
},
"binFiles": [
"cudnn64_8.dll",
"cudnn_adv_infer64_8.dll",
"cudnn_adv_train64_8.dll",
"cudnn_cnn_infer64_8.dll",
"cudnn_cnn_train64_8.dll",
"cudnn_ops_infer64_8.dll",
"cudnn_ops_train64_8.dll",
"nvinfer.dll",
"nvinfer_builder_resource.dll",
"nvinfer_dispatch.dll",
"nvinfer_lean.dll",
"nvinfer_plugin.dll",
"nvinfer_vc_plugin.dll",
"nvonnxparser.dll",
"nvparsers.dll",
"sl_ai64.dll",
"sl_zed64.dll",
"zlibwapi.dll"
],
"driverVersion": "531.61",
"drivervDllVersion": "31.0.15.3161",
"resourcesFiles": [
],
"resourcesFilesExpected": [
"objects_performance_3.2"
],
"runtimeDllVersion": 12010,
"valid": [
"<b>ZED SDK version:</b> 4.0.6",
"<b>CUDA version:</b> V12.1.66"
]
}
}Anything else?
No response
This should be possible with ZED Mini as it states here:
Important: At the moment, only a few object classes can be detected and tracked with the 3D Object Detection API using ZED cameras (except the ZED 1 camera).
And here:
Note: Object Detection has been optimized for ZED 2/ZED 2i and uses the camera motion sensors for improved reliability. Therefore the Object Detection module requires a ZED 2/ZED 2i or ZED Mini, and sensors cannot be disabled when using the module.
But there the tutorial only addresses ZED 2.
Can anyone elaborate please, this is urgent!?
Hi @aaronlsmiles, sorry for the delay.
You should very much be able to optimize the models, at least outside of Unity.
Does optimizing the models outside of Unity work? You can find some directives here: How can I optimize the ZED SDK AI models manually?
I contacted support and there was some issue with the optimised files not downloading as the resources folder was always empty. So they sent them across and I pasted them in the folder then optimised them, which worked. They are investigating the issue.