pyds_tracker_meta

pybind11 wrapper to access Nvidia DeepStream tracker meta info (NvDsPastFrame... classes) from Python.

The master branch of this repository is compatible with DeepStream SDK 5.0. If you are using an earlier version of the SDK (including DeepStrem SDK 5.0 Developer Preview), check out the deepstream-4.0 branch.

Notes for DeepStreamer 5.0

Starting from DeepStream SDK 5.0 the python bindings for NvDsPastFrame... classes are included in the official SDK. However some vital functions are missing:

to cast a generic user metadata to NvDsPastFrameObjBatch object
to correctly iterate through the elements of list fields of NvDsPastFrameObjBatch, NvDsPastFrameObjStream and NvDsPastFrameObjList. These fields are standard C arrays in the native SDK, so simply exposing them in python will give access in the best case only the first element of the array. The pointer arithmetic to iterate over the elements is also included in this library.

Introduction

This library provides utility functions to access to the NvDsPastFrameObjBatch type user metadata of DeepStream data streams. This metadata is generated by the object tracker of nvtracker plugin. The C API of these structures can be found in nvds_tracker_meta.h header file of DeepStream SDK. The instances of these metadata structure contain information about the object tracking results in past frames.

Some trackers do not report the state of the tracked object if there are matching detection results in the current frame. When in a later frame a detection result confirms the state of the tracked object, the past history of the tracking is reported retroactively in NvDsPastFrame... metadata. This library provides Python access to this metadata. For more information refer to the nvtracker plugin documentation.

Installation

Prerequisites

Install pybind11. The recommended way is to build it from source. Alternatively you might try simply pip3 install pybind11.
You should have gstreamer-1.0 and gstreamer-video-1.0 packages installed in your system. If you are using DeepStream, you most likely installed these packages.
You will need also the standard c++ compiler you usually find in Linux distributions. c++11 standard is used.

Compile the source

The source should be compiled on your target platform (Jetson or x86).
Set your DeepStream version and path in build.sh.
Launch build.sh
Install the compiled module with python setup.py install (use sudo or python3 if needed).

Usage

pyds_tracker_meta is meant to be used together with the standard Python bindings for DeepStream. Make sure you have pyds available.

Ensure you have set enable-past-frame property of the gst-nvtracker plugin to 1. (See nvtracker plugin documentation.)

Most likely you will use this library from the buffer probe callbacks of a gstreamer plugin pad, when the object tracking results are available. The deepstream-test2 python app shows you how to set up such a callback.

The example snippet provided bellow shows how to cast a user meta to a past frame object batch, and how to access all fields of the metadata. Add the following lines to the osd_sink_pad_buffer_probe method found int deepstream-test2.py, just after the batch_meta was acquired:

import pyds_tracker_meta

def osd_sink_pad_buffer_probe(pad, info, u_data):
    
    # ... code to acquire batch_meta ...
    
    user_meta_list = batch_meta.batch_user_meta_list
    while user_meta_list is not None:
        user_meta = pyds.NvDsUserMeta.cast(user_meta_list.data)
        
        print('user_meta:', user_meta)
        print('user_meta.user_meta_data:', user_meta.user_meta_data)
        print('user_meta.base_meta:', user_meta.base_meta)
        
        if user_meta.base_meta.meta_type != pyds.NvDsMetaType.NVDS_TRACKER_PAST_FRAME_META:
            continue
            
        pfob = pyds_tracker_meta.NvDsPastFrameObjBatch_cast(user_meta.user_meta_data)
        print('past_frame_object_batch:', pfob)
        print('  list:')
        for pfos in pyds_tracker_meta.NvDsPastFrameObjBatch_list(pfob):
            print('    past_frame_object_stream:', pfos)
            print('      streamID:', pfos.streamID)
            print('      surfaceStreamID:', pfos.surfaceStreamID)
            print('      list:')
            for pfol in pyds_tracker_meta.NvDsPastFrameObjStream_list(pfos):
                print('        past_frame_object_list:', pfol)
                print('          numObj:', pfol.numObj)
                print('          uniqueId:', pfol.uniqueId)
                print('          classId:', pfol.classId)
                print('          objLabel:', pfol.objLabel)
                print('          list:')
                for pfo in pyds_tracker_meta.NvDsPastFrameObjList_list(pfol):
                    print('            past_frame_object:', pfo)
                    print('              frameNum:', pfo.frameNum)
                    print('              tBbox.left:', pfo.tBbox.left)
                    print('              tBbox.width:', pfo.tBbox.width)
                    print('              tBbox.top:', pfo.tBbox.top)
                    print('              tBbox.right:', pfo.tBbox.height)
                    print('              confidence:', pfo.confidence)
                    print('              age:', pfo.age)
        try:
            user_meta_list = user_meta_list.next
        except StopIteration:
            break