This repository implements a realtime 3D hand posture estimation pipeline running on Jetson platform using a Azure Kinect camera.
Please refer to the following repositories before getting started here:
There are 2 stages to our pipeline
The first stage will localize the hand using a fusion of infrared and depth image.
NOTE: more detail can be found in the centernet_kinect repository
The second stage would perform 3D hand posture estimation on the region of intrest selected by the previous step.
NOTE: for training a model please refer to the Hand Posture Estimation repository
- Initially configure the pipeline/constants.py file:
- CENTERNET_MODEL_PATH please place the centernet model weights in "/checkpoint/CenterNet"
with the naming convention that was provided in the original repository- Configure the centernet portion of the file as its been described in the original repository.
if you are using the weights directly from the original repository you dont have to modify this section.
- Configure the centernet portion of the file as its been described in the original repository.
- A2J_MODEL_PATH please place the A2J model weights in "/checkpoint/A2J"
with the naming convention that was provided in the original repository- Configure the a2j portion of the file as you have set up the training pipeline for Hand Posture Estimation.
- For Faster inference we use TensorRT inference engine to optimize the models. this will take some time to compile the models and create a TRT engine
- Configure the a2j portion of the file as you have set up the training pipeline for Hand Posture Estimation.
- CENTERNET_MODEL_PATH please place the centernet model weights in "/checkpoint/CenterNet"
- Run realtime inference on a jetson platform.
cd pipeline python3 azure_kinect.py # Optional for faster inference python3 azure_kinect.py --trt True # for optimizing the models with TensorRT fp16