*****************************************************************************
* Copyright (c) 2018-2020 NVIDIA Corporation. All rights reserved.
*
* NVIDIA Corporation and its licensors retain all intellectual property
* and proprietary rights in and to this software, related documentation
* and any modifications thereto. Any use, reproduction, disclosure or
* distribution of this software and related documentation without an express
* license agreement from NVIDIA Corporation is strictly prohibited.
*****************************************************************************
===============================================================================
1. Prerequisites:
===============================================================================
Please follow instructions in the deepstream_sdk_release_package/README on how
to install the prerequisites for Deepstream SDK, the DeepStream SDK itself and
the apps.
Additionally you must have the following development packages installed
GStreamer-1.0
GStreamer-1.0 Base Plugins
GStreamer-1.0 gstrtspserver
X11 client-side library
Glib json library - json-glib-1.0
To install these packages, execute the following command:
$ sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev \
libgstrtspserver-1.0-dev libx11-dev libjson-glib-dev
Also refer ../deepstream-test4/README for detailed Dependencies for the various
supported IoT protocols
===============================================================================
2. To compile:
===============================================================================
$ cd apps/sample_apps/deepstream-test5
$ make
===============================================================================
3. Config files provides with this package
===============================================================================
1) test5_config_file_src_infer.txt - A 4 X File(mp4) stream analytics pipeline
config
2) test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt - A 2 X RTSP
Camera analytics pipeline config
3) test5_config_file_src_infer_tracker_sgie.txt - A 4 X File(mp4) stream
analytics pipeline config with SGIE usage
Please provide message broker connection string in [sink1] section
===============================================================================
3.1 IoT Protocols supported and cloud configuration
===============================================================================
Details on the IoT Protocols (like KAFKA, Azure, AMQP, etc)
supported by nvmsgbroker plugin is listed in the DeepStream Plugin Manual.
DeepStream Public documentation may be referred to setup IoT hubs/servers/brokers
specific to the protocol in use.
[sink] group keys associated with type=6 for nvmsgconv and nvmsgbroker
configuration are discussed in the DeepStream_Development_Guide ->
Reference Application Configuration -> Configuration Groups.
===============================================================================
4. To run:
===============================================================================
$ ./deepstream-test5-app -c configs/test5_config_file_src_infer.txt -p 1
$ ./deepstream-test5-app -c configs/test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt -p 0
===============================================================================
5. Option -p [Default=1]
===============================================================================
5.1 -p 1 (PLAYBACK=1)
(Base UTC timestamp from filename of format - *__M_DD_YYYY_H_MIN_SEC_MSEC_AM/PM_UTC<Offset>.mp4)
Means - the first video frame timestamp is embedded in the URI.
Example: VideoFileSavedFromCamera__4_11_2018_4_59_59_433_AM_UTC-07_00.mp4
$ ./deepstream-test5-app -c test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt -p 1
=============================================================================
NOTE: If URI does not have original timestamp embedded, a WARNING will be
thrown to user and current system time will be used as base time for all metadata
==============================================================================
5.2 -p 0 (PLAYBACK=0)
(Base UTC timestamp for meta-data from RTCP SR for RTSP Sources
only; any sources other than type=4 will default to PLAYBACK=1)
$ ./deepstream-test5-app -c test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt -p 0
==============================================================================
NOTE 1: For RTSP Streams, Config file shall use 'type=4'(NV_DS_SOURCE_RTSP)
in [source] groups
==============================================================================
NOTE 2: If the RTSP Camera/Server does not send RTCP Sender Report - or until
the first Sender Report arrive, the buffer/metadata timestamps sent out by
nvmsgbroker will be based on the initial timestamp as zero.
==============================================================================
===============================================================================
6. Option -m [Default=0]
PGIE Model used; {0 - Unknown [DEFAULT]}, {1: Resnet 4-class [Car, Bicycle, Person, Roadsign]}
===============================================================================
This option is to demonstrate the code which can read
PGIE, Tracker, and SGIE Metadata from NvDsBatchMeta and fill
NvDsEventMsgMeta in bbox_generated_probe_after_analytics()
for the nvmsgconv schema and JSON-metadata-generation.
For more details run the below command on terminal post DS installation:
gst-inspect-1.0 nvmsgconv
==============================================================================
NOTE: Customers using `-m 0` with custom PGIE, SGIE models,
please read the documentation under "IMPORTANT Note 1" tag
in deepstream_test5_app_main.c for changes in code required
to relay SGIE Classifier-Metadata to nvmsgconv
6.1 -m 0 (Model(s) used are unknown)
$ ./deepstream-test5-app -c test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt -m 0
6.2 -m 1 (Analytics Model(s) used are:
resnet(4-class model) -> Tracker (Optional) -> SGIE0 (Vehicle-type)
-> SGIE1 (Vehicle-Color) -> SGIE2 (Vehicle-Make))
$ ./deepstream-test5-app -c test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt -m 1
===============================================================================
7. Option -o
GIE Model update configuration override file; {provide path of override config file}
This option is used to demonstrate the on-the-fly model update feature.
deepstream-test5-app to be launched with -o <ota_override_file> option to test
on-the-fly OTA functionality.
Steps to test the OTA functionality
1) Run deepstream-test5-app with -o <ota_override_file> option
2) While DS application is running, update the <ota_override_file> with new model details
and save it
3) File content changes gets detected by deepstream-test5-app and then it starts
model-update process
Currently only model-update feature is supported as a part of OTA functionality.
Assumption for On-The-Fly model updates are as below:
1) New model must have same network parameter configuration as of previous model
(e.g. network resolution, network architecture, number of classes)
2) Engine file or cache file of new model to be provided by developer
3) Other primary gie configuration parameters like group-threshold, bbox color, gpu-id,
nvbuf-memory-type etc updated parameters if provided in the override file,
will not have any effect after model switch.
4) Secondary gie model-update is not validated, primary model-update is validated
5) No frame drop / frames without inference should be observed during on-the-fly
model update process
6) In case of model update failure, error message will be printed on the console and
pipeline should continue to run with older model configuration
7) config-file parameter is needed to suppress the config file parsing error prints,
values from this config file are not used during model switch process
===============================================================================
8. Multiple broker sinks
===============================================================================
Multiple broker sinks might be required for sending a message to multipe
backends simultaneously or to send specific message to particular backend.
By default sink of type = 6 adds message converter and message broker
components in the pipeline.
In case of multiple brokers [message-converter] group can be used to add
single message converter in the pipeline with multiple sinks of type = 6
having disable-msgconv set to 1.
If multiple message converters are also required along with multiple brokers
then "msg-conv-comp-id" and "msg-broker-comp-id" should be set properly to
avoid duplicate messages. These fields force converter / broker components to
process only those messages having same value for componentId field and ignore
other messages. User should modify the application to fill componentId field of
NvDsEventMsgMeta structure.
===============================================================================
9. Smart Record - Event based recording
===============================================================================
Event based recording can be enabled by setting "smart-record" under [sourceX]
group. Currently only source type = 4 (RTSP) is supported.
There are two ways in which smart record events can be triggerd.
1) Through cloud messages.
set smart-record=1 and configure [message-consumerX] group accordingly.
Refer deepstream_c2d_msg_utils.c for format of cloud message to trigger
the smart record start / stop events.
2) Through local events.
set smart-record=2, this will enable smart record through cloud messages
as well as local events.
To demonstrate the event based recording through local events, application
by default triggers start / stop events every ten seconds.
This interval and other parameters are configurable.
Please refer plugin manual and gst-nvdssr.h header file for more details about
smart record.
===============================================================================
10. Message consumer
===============================================================================
deepstream-test5-app can be configured to work as message consumer for
cloud messages. After parsing the received message and based on the content
of the message specific action(s) can be triggered.
e.g. NvDsSrcParentBin*, which holds the smart record context, is passed as an
argument in start_cloud_to_device_messaging() that is then used to trigger
start/stop of smart record.
By default, event based recording has been implmented to demonstrate the
usage of message consumer.
User need to implement the custom logic to work on other types of received
messages.
Following minimum json message is expected to trigger the start / stop
of smart record.
{
command: string // <start-recording / stop-recording>
start: string // "2020-05-18T20:02:00.051Z"
end: string // "2020-05-18T20:02:02.851Z",
sensor: {
id: string
}
}
To subscribe to cloud messages configure the [message-consumer] group(s)
accordingly.
===============================================================================
===============================================================================
General Introduction
===============================================================================
This document shall describe about the sample deepstream-test5 application.
This sample builds on top of the deepstream-app sample to demonstrate how to:
* Use "nvmsgconv" and "nvmsgbroker" plugins in the pipeline.
* Create NVDS_EVENT_MSG_META type of meta and attach to buffer.
* Use NVDS_EVENT_MSG_META for different types of objects e.g. vehicle, person etc.
* Provide copy / free functions if meta data is extended through "extMsg" field.
"nvmsgconv" plugin uses NVDS_EVENT_MSG_META type of metadata from the buffer
and generates the "DeepStream Schema" payload in Json format. Static properties
of schema are read from configuration file in the form of key-value pair.
Check dstest5_msgconv_sample_config.txt for reference. Generated payload is attached
as NVDS_META_PAYLOAD type metadata to the buffer.
"nvmsgbroker" plugin extracts NVDS_META_PAYLOAD type of metadata from the buffer
and sends that payload to the server using protocol adaptor APIs.
Generating custom metadata for different type of objects:
In addition to common fields provided in NvDsEventMsgMeta structure, user can
also create custom objects and attach to buffer as NVDS_EVENT_MSG_META metadata.
To do that NvDsEventMsgMeta provides "extMsg" and "extMsgSize" fields. User can
create custom structure, fill that structure and assign the pointer of that
structure as "extMsg" and set the "extMsgSize" accordingly.
If custom object contains fields that can't be simply mem copied then user should
also provide function to copy and free those objects.
Refer generate_event_msg_meta() to know how to use "extMsg" and "extMsgSize"
fields for custom objects and how to provide copy/free function and attach that
object to buffer as metadata.
NOTE:
-----
#Rebuilding the Deepstream based Docker image:
----------------------------------------------
In order to extend DeepStream docker image with your modified config file settings,
follow instructions listed in the "Docker Containers" section of the DeepStream Plugin
Manual, for creating your own docker image based on DeepStream.
The associated steps are shown below specifically for the test5 usecase for convenience:
Step 1: Modify test5 config file as required; assume new config file is named test5_config_new.txt
Step 2: Create Dockerfile for building new DeepStream docker image for jetson which includes modified config file
$ cat <<EOF > Dockerfile
#==============================================================================
FROM nvcr.io/nvidia/deepstream-l4t:<tag>
ADD test5_config_new.txt /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test5/configs/
CMD ["/bin/bash"]
WORKDIR /opt/nvidia/deepstream/deepstream/
#==============================================================================
EOF
Step 3: Rebuild new docker image
$ docker build -t ds_test5:1 .
#Run deepstream-test5-app as Azure iotedge module:
--------------------------------------------------
# Refer to sections within DS_PACKAGE_DIR/sources/libs/azure_protocol_adaptor/module_client/README to:
1. Create azure IotHub
2. Register a new Azure IoT Edge device from the Azure portal
3. Setup and install Azure Iot Edge on your machine
4. Install dependencies
5. Specify connection string
6. Use cfg file (optional)
7. Install & setup nvidia-docker
8. Deploy a iotedge module
9. Start the edge runtime
# On Azure iothub deployment page, use the information below as an example to use deepstream-test5-app using
* sample config file "test5_config_file_src_infer_azure_iotedge.txt"
* to send messages with minimal schema
* with display disabled
* Message topic = mytopic
(Note: Message topic cant be empty)
Deployment of a Module:
-----------------------
On the Azure portal, Click on the Iot edge device you have created and click Set Modules:
Container Registry Settings:
Name: NGC
Address: nvcr.io
User Name: $oauthtoken
Password: <password (or API key) from your NGC account>
Deployment modules:
Add new module:
- Name: ds_test5
- Image URI:
For x86: nvcr.io/nvidia/deepstream:<tag>
For jetson: nvcr.io/nvidia/deepstream-l4t:<tag>
- container Create options:
{
"HostConfig": {
"Runtime": "nvidia"
},
"WorkingDir": "/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test5/configs/",
"ENTRYPOINT": [
"/usr/bin/deepstream-test5-app",
"-c",
"test5_config_file_src_infer_azure_iotedge.txt",
"-p",
"1",
"-m",
"1"
]
}
- specify route options for the module:
option 1) Default route where every message from every module is sent upstream
{
"routes": {
"route": "FROM /messages/* INTO $upstream"
}
}
option 2) you can mention specific routes where messages sent upstream based on topic name
ex: in the sample test programs, topic name "mytopic" is used for the module name ds_test5
{
"routes": {
"route": "FROM /messages/modules/ds_test5/outputs/mytopic INTO $upstream",
}
}
Start the edge runtime and verify if modules are running:
---------------------------------------------------------
#Restart the iotedge on your system
sudo systemctl restart iotedge
#Give it a few seconds
#check edge runtime status
systemctl status iotedge
#List the modules running
sudo iotedge list
#check output from the modules
sudo iotedge logs ds<ds-version>_test5
===============================================================================
FAQ
===============================================================================
1) I see the following WARNING:
WARNING; playback mode used with URI [file:///.../streams/sample_720p.mp4] not conforming
to timestamp format; check README; using system-time
===============================================================================
Answer:
The file URI does not have the base timestamp (first video frame timestamp in
the format discussed above in Section 5.2
2) I see the following WARNING:
nvmsgbroker queue overrun; Older Message Buffer Dropped; Network bandwidth might be insufficient
===============================================================================
Answer:
The GstQueue just before nvmsgbroker is set to hold a max number of buffers (say 20).
This means - if the network bandwidth is not enough to scale and send the
metadata over nvmsgbroker overflowing this queue, the queue will leak
(older buffers will be dropped with this WARNING).
You could change this number at ../../apps-common/src/deepstream_sink_bin.c
In function - create_msg_conv_broker_bin() -
g_object_set(G_OBJECT(bin->queue), "max-size-buffers", 20, NULL);
3) What do I need to change in the test5_config_file_src_infer.txt or
test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt config?
===============================================================================
Answer:
i) [sinkN] group config with type=6
needs to be reconfigured.
Main change required are:
a) msg-broker-proto-lib - KAFKA/Azure
b) msg-broker-conn-str - The broker-connection-string
c) topic
For more details on these configs check documentation with:
`gst-inspect-1.0 nvmsgbroker`
Example:
[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=dstest5_msgconv_sample_config.txt
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so
msg-broker-conn-str=host;port;example-topic
topic=example-topic
ii) [sourceN] for RTSP cameras/streams
The streams provided in config are placeholders;
User needs to change these for IP cameras or streams in use
4) I see a WARNING log similar to the one below:
%3|1560387814.967|FAIL|rdkafka#producer-1| [thrd:kafka.broker.host:9092/1021]: kafka.broker.host:9092/1021: Receive failed: Disconnected
%3|1560387814.967|ERROR|rdkafka#producer-1| [thrd:kafka.broker.host:9092/1021]: kafka.broker.host:9092/1021: Receive failed: Disconnected
===============================================================================
Answer:
Just shows that TCP connection got disconnected after a period of inactivity.
But it will get reestablished when there are messages to be sent.
5) I see a lot of artifacts/video-glitches on the decoded stream
(when tiler + display) is enabled.
===============================================================================
Answer:
If using RTSP Cameras, network latency above a configurable threshold causes
RTP buffer drops.
Please fine tune your camera's [source] group configuration in the
DeepStream config file using the "latency" config key.
The rtp-jitterbuffer shall drop buffers when its buffering
more than the configured latency.
A sample usage with details on the latency key provided below.
[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://foo.com/sample1.mp4
num-sources=1
gpu-id=0
nvbuf-memory-type=0
#max Latency of rtp buffers to
#buffer at the source in ms (milliseconds)
#Default=100ms
latency=100
Additional info: `$gst-inspect-1.0 rtspsrc`
6) How does the data in nvmsgconv config file (say - dstest5_msgconv_sample_config.txt)
specified in [sink1] msg-conv-config key
map with the streams specified in DeepStream config file (say -
test5_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt) ?
===============================================================================
Answer:
[config-group-index] shall be the same among these config files
for a particular source.
Example:
[source0] in DeepStream config maps to:
[sensor0], [place0], [analytics0] groups in the nvmsgconv config file
[source1] in DeepStream config maps to:
[sensor1], [place1], [analytics1] groups in the nvmsgconv config file