Skeleton Tracking with Kinect Azure
Closed this issue · 21 comments
Since Keijro's code doesn't include any skeleton tracking feature, we have to add that functionality separately.
Microsoft provides the SDK for skeletal tracking completely separately.
This is the Sensor SDK
https://github.com/microsoft/Azure-Kinect-Sensor-SDK
This is the Body tracking SDK
https://docs.microsoft.com/en-us/azure/Kinect-dk/body-sdk-download
- According to the webpage version 0.9.5 of Body Tracking SDK includes C# wrapper.
- The Body tracking SDK requires Nvidia video card for probably GPU boosted skeletal detection (which both of us have)
https://docs.microsoft.com/en-us/azure/Kinect-dk/body-sdk-setup
Tutorial in C
https://docs.microsoft.com/en-us/azure/Kinect-dk/build-first-body-app
There is also example in Unity
https://github.com/curiosity-inc/azure-kinect-dk-unity
-> this example crashes with our Unity Editor setup.
The problem is The body tracking SDK is basically C++ and I don't think I can write a C# wrapper in my remaining time.
I found the following asset in the asset store.
I guess this may be a faster solution than code the C# wrapper of the body tracking SDK by ourselves.
Ok i‘ll buy it when I come home tonight, and make you a task to implement it cleanly , ok? The only issue I have with this is, I can‘t just add this code in the open source version I have to provide in the end. Correct?
Correct. You cannot include the code of the asset.
@chikashimiyama I bought the asset. But it's around 400mb. How can I push this to you?
Can we implement this as cleanly as possible, so that I can leave it out if I release it at the end?
Once you add something to the git repo it stays in principle forever. So we should not include that in git repo. We just need to register all files from the assets in .gitignore so that git system ignores all of them.
But obviously, this creates a large dependency. If somebody wants to reuse our code, they may not be able to run it without buying the asset.
To use assets is actually not a good idea and very problematic in terms of open-source publishment. But probably that's the only way to realize skeleton tracking to me.
To your question: don't push the asset. we just share with another method like we transfer.
and please re-think the above points. If the open source publishment is very important, this may be not a good move.
But obviously, this creates a large dependency. If somebody wants to reuse our code, they may not be able to run it without buying the asset.
understood. Do you see a way, of including this into our project, so that we create the smallest possible amount of dependencies? So that I can work with it, tryout things, but the project would at least start without this part of the code? and scenes, that don't utilize the code, would still build?
My idea would be, to release all the code, except this part and add info on the needs for this code in the doc. Also, with time, I might be able to find other ways of implementing skeleton tracking.
Lastly, I'll just have to tell the academy that, they should buy the asset as well, so anybody who wants to work with skeleton tracking and azure in the future, within the academy, will immediately be able to do so. And I see a lot of applications there.
got an asset. will take a look at the API of the asset. It depends on how they implement it.
The asset uses the following version of SDK
- Camera SDK 1.3.0
- Body Tracking SDK 0.9.4
We are using
- Camera SDK 1.3.0
- Body Tracking SDK 0.9.5
So they are very close
The file you sent to me is not a unity package. I need a unity package not the project that imported it.
Demo scene of the asset is not working.
- can you run Demo_Kinect4Azure on your machine?
- in terms of unity package, this might help to find unity package on your machine.
Another question is if it is possible to use this asset in parrallel with Keijiro's asset.
Both assets use same microsoft API and if so, we should use the one from the lightbuzz asset and abondon Keijiro's hack entirely because the asset is more generic than Keijiro's hack.
But I'm not sure the connection between Renderbuffer and this asset.
I'm afraid that we are opening another pandra's box. If I have three more days I can solve this problem but the remaining time is 8 hours... and we see now two major issues
- possible conflict with Keijiro's code
- Open Source problem.
This might be typical "last minute hack breaks all" situation. May be my suggestion to use asset is not the right one (actually I totally forgot about OpenSource issue when I suggested to use the asset)
My suggestion is to test asset in unity for checking the skeleton feature with Azure but postpone the integration of skeleton feature
conflicts:
I get these errors if I add the asset to our project:
Multiple plugins with the same name 'depthengine_2_0' (found at 'Assets/ThridParty/LightBuzz_Azure_Kinect/Plugins/x86_64/depthengine_2_0.dll' and 'Assets/Plugins/depthengine_2_0.dll'). That means one or more plugins are set to be compatible with Editor. Only one plugin at the time can be used by Editor.
Multiple plugins with the same name 'k4a' (found at 'Assets/Plugins/k4a.dll' and 'Assets/ThridParty/LightBuzz_Azure_Kinect/Plugins/x86_64/k4a.dll'). That means one or more plugins are set to be compatible with Editor. Only one plugin at the time can be used by Editor.
I can run the demo scene with these errors, but it won't detect any joints
I can still run Kejiro Scenes, or scenes with his code in it
In the Log-File I get a lot of these errors:
Refreshing native plugins compatible for Editor in 1.36 ms, found 10 plugins.
[2019-12-28 13:00:08.157] [error] [t=10796] [K4ABT] D:\a\1\s\src\TrackerHost\TrackerHost.cpp (484): VerifyDependencies(). Cannot locate onnxruntime.dll[2019-12-28 13:00:08.157] [error] [t=10796] [K4ABT] D:\a\1\s\src\TrackerHost\TrackerHost.cpp (131): Create(). At least one dependent library is missing!
[2019-12-28 13:00:08.157] [error] [t=10796] [K4ABT] D:\a\1\s\src\sdk\k4abt.cpp (38): tracker->Create(sensor_calibration, config) returned failure in k4abt_tracker_create()
[2019-12-28 13:00:09.055] [error] [t=10796] [K4ABT] Invalid k4abt_tracker_t 0000000000000000
My suggestion is to test asset in unity for checking the skeleton feature with Azure but postpone the integration of skeleton feature
I completely agree with you