Object tracking with Tello

What's this

This is test code of object tracking/detection with Tello.

You can do followings:

Video streaming from Tello on your browser.
Object / Color Tracking.
Change tracking mode (streamonly, test, tracking).
Object Detection(MobileNet-SSD)
Face Detection and Face analytics (Age/Gender, Emotion, Head Pose, Facial Landmarks)

Note: Object Detection and Face Detection/analytics require OpenVINO toolkit

Face detection demo (YouTube Link)

Color tracking (YouTube Link)

Reference

Tello SDK Documentation (v1.3) Commands this app used are based on sdk v1.3

https://dl-cdn.ryzerobotics.com/downloads/Tello/20180910/Tello%20SDK%20Documentation%20EN_1.3.pdf

Flask Video streaming

http://blog.miguelgrinberg.com/post/video-streaming-with-flask

https://github.com/ECI-Robotics/opencv_remote_streaming_processing/

Environment

Windows 10
Python 3.6.x
Flask 0.12.2
opencv-python 3.4.3.18
Chrome / Firefox
OpenVINO Toolkit 2019 R4+ (Required only when object/face detection is enanbled )

Note1: If you use Ubuntu, issues#4 maybe help you.

Note2: "In OpenVINO toolkit 2020.1 build+, CPU extensions (including AVX) have been included into CPU MKL-DNN plugin." libcpu_extension_avx2.so No such file or directory Error

Required Packages

Install required packages with pip

pip install -r requirements.txt

pip install opencv-python
pip install flask

How to use

Command Option

$ python app.py -h
usage: app.py [-h] [-a {camshift,meanshift}] [-t] [-tr]
              [-c {blue,skyblue,red,yellow,green}] [--enable_detection]
              [-m_ss MODEL_SSD] [-m_fc MODEL_FACE] [-m_ag MODEL_AGE_GENDER]
              [-m_em MODEL_EMOTIONS] [-m_hp MODEL_HEAD_POSE]
              [-m_lm MODEL_FACIAL_LANDMARKS] [-l CPU_EXTENSION]
              [-d {CPU,GPU,FPGA,MYRIAD}] [-d_ag {CPU,GPU,FPGA,MYRIAD}]
              [-d_em {CPU,GPU,FPGA,MYRIAD}] [-d_hp {CPU,GPU,FPGA,MYRIAD}]
              [-d_lm {CPU,GPU,FPGA,MYRIAD}] [-pp PLUGIN_DIR] [--labels LABELS]
              [-pt PROB_THRESHOLD] [-ptf PROB_THRESHOLD_FACE]

optional arguments:
  -h, --help            show this help message and exit
  -a {camshift,meanshift}, --algorithm {camshift,meanshift}
                        selct object tracking algorithm
  -t, --test            test mode (disable tracking motion)
  -tr, --tracking       tracking mode (enable tracking motion)
  -c {blue,skyblue,red,yellow,green}, --color {blue,skyblue,red,yellow,green}
                        select tracking color in color.ini
  --enable_detection    enable object detection using MobileNet-SSD
  -m_ss MODEL_SSD, --model_ssd MODEL_SSD
                        Required. Path to an .xml file with a trained
                        MobileNet-SSD model.
  -m_fc MODEL_FACE, --model_face MODEL_FACE
                        Optional. Path to an .xml file with a trained
                        Age/Gender Recognition model.
  -m_ag MODEL_AGE_GENDER, --model_age_gender MODEL_AGE_GENDER
                        Optional. Path to an .xml file with a trained
                        Age/Gender Recognition model.
  -m_em MODEL_EMOTIONS, --model_emotions MODEL_EMOTIONS
                        Optional. Path to an .xml file with a trained Emotions
                        Recognition model.
  -m_hp MODEL_HEAD_POSE, --model_head_pose MODEL_HEAD_POSE
                        Optional. Path to an .xml file with a trained Head
                        Pose Estimation model.
  -m_lm MODEL_FACIAL_LANDMARKS, --model_facial_landmarks MODEL_FACIAL_LANDMARKS
                        Optional. Path to an .xml file with a trained Facial
                        Landmarks Estimation model.
  -l CPU_EXTENSION, --cpu_extension CPU_EXTENSION
                        MKLDNN (CPU)-targeted custom layers.Absolute path to a
                        shared library with the kernels impl.
  -d {CPU,GPU,FPGA,MYRIAD}, --device {CPU,GPU,FPGA,MYRIAD}
                        Specify the target device for MobileNet-SSSD / Face
                        Detection to infer on; CPU, GPU, FPGA or MYRIAD is
                        acceptable.
  -d_ag {CPU,GPU,FPGA,MYRIAD}, --device_age_gender {CPU,GPU,FPGA,MYRIAD}
                        Specify the target device for Age/Gender Recognition
                        to infer on; CPU, GPU, FPGA or MYRIAD is acceptable.
  -d_em {CPU,GPU,FPGA,MYRIAD}, --device_emotions {CPU,GPU,FPGA,MYRIAD}
                        Specify the target device for for Emotions Recognition
                        to infer on; CPU, GPU, FPGA or MYRIAD is acceptable.
  -d_hp {CPU,GPU,FPGA,MYRIAD}, --device_head_pose {CPU,GPU,FPGA,MYRIAD}
                        Specify the target device for Head Pose Estimation to
                        infer on; CPU, GPU, FPGA or MYRIAD is acceptable.
  -d_lm {CPU,GPU,FPGA,MYRIAD}, --device_facial_landmarks {CPU,GPU,FPGA,MYRIAD}
                        Specify the target device for Facial Landmarks
                        Estimation to infer on; CPU, GPU, FPGA or MYRIAD is
                        acceptable.
  -pp PLUGIN_DIR, --plugin_dir PLUGIN_DIR
                        Path to a plugin folder
  --labels LABELS       Labels mapping file
  -pt PROB_THRESHOLD, --prob_threshold PROB_THRESHOLD
                        Probability threshold for object detections filtering
  -ptf PROB_THRESHOLD_FACE, --prob_threshold_face PROB_THRESHOLD_FACE
                        Probability threshold for face detections filtering

Object tracking settings (parameters in Tello.cfg)

frame property

[camera]
# deifne resize property of frame.
# (480 * 360  is recommended)
# Note: This is resize parameter of frames. OpenCV VideoCapture with udp streaming can not set cv2.CAP_PROP_XX.
resize_prop = (480, 360)
.
.

Tello setting

[tello]
# tello address and port is not needed to change.
tello_addr = ('192.168.10.1', 8889)
# define move cm by operation (MIN, MAX, Tello sdk MAX)
right = (20, 30, 500)
left = (20, 30, 500)
.
.

tracking setting

[tracking]
.
.
# !! important !! set the motion limit range of Tello for your safety 
#  (x, y, z (cm), rotate (dgree))
position_limit = (300, 300, 300, 180)

Note:

Verify the motion limit range of Tello for your safety. position_limit is not the actual position of Tello but the position calcurated by this applicaiton per sending command to Tello.

Run app

Power on the Tello.
Connect to Tello's wifi network (TELLO-XXXXXX)
Run app (ex. Color tracking with test mode.)

$ python app.py --test --color yellow

Access to the streaming url on your browser

http://127.0.0.1:5000/

connet to Tello (click the connect button)

Note:

if you can not connect to Tello (no response:ok ) , make sure to connect Tello's network and try to restart app (Stop(Ctrl + C) , then run app again).
Red buttons (arrow icons) send flip motion command to Tello. Be careful around!
"test mode" means that 'object tracking' is enabled but ** motion of Tello (sending command to Tello) is disabled **.

colors are defined in color.ini

# define hsv (hue, staturation, lightness)
[yellow]
lower = 20, 100, 100
upper = 40, 255, 255
.
.

Object detection mode

You can test Object deteciton using MobileNet-SSD (detection button). Detection mode requires OpenVINO Toolkit R4/R5 installed on your PC and CPU extension dll ,IR files bellow.

Requirements

Install Intel® Distribution of OpenVINO™ toolkit for Windows* 10 https://software.intel.com/en-us/articles/OpenVINO-Install-Windows
URL: https://drive.google.com/open?id=1YKbwy9W0MZObls9dy_0n90MQoRq0RdOB
- File extension.zip
- Size: 32,084,333 byte
- MD5 hash : 31d7c77ade31fd1cb9cca6c9a92128f3
Extract extension.zip and store extension folder under the "tellooo"

extension/cpu_extension.dll
extension/IR/FP16/MobileNetSSD_deploy.bin
extension/IR/FP16/MobileNetSSD_deploy.mapping
extension/IR/FP16/MobileNetSSD_deploy.xml
extension/IR/FP32/MobileNetSSD_deploy.bin
extension/IR/FP32/MobileNetSSD_deploy.mapping
extension/IR/FP32/MobileNetSSD_deploy.xml

Download Face detection models IR files

cd extension/IR/
models="face-detection-retail-0004 age-gender-recognition-retail-0013 emotions-recognition-retail-0003 head-pose-estimation-adas-0001 landmarks-regression-retail-0009"
for model in $models
do
wget --no-check-certificate -P FP16 https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/${model}/FP16/${model}.xml
wget --no-check-certificate -P FP32 https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/${model}/FP16/${model}.bin
done

Run App

Run app (ex. Color tracking with test mode AND enable object detection.)

$ python app.py --test --color yellow --enable_detection

If you use MYRIAD plugin (NCS2（Neural Compute Stick2))

$ python app.py --test --color yellow --enable_detection --device MYRIAD

Access to the streaming url on your browser

http://127.0.0.1:5000/

Misc

Test on your PC

Modify 'Tello_addr' of '[Tello]' section in tello.cfg. You can change this when you test udp video streaming on your pc. (see test\server.py)

[tello]
tello_addr = ('127.0.0.1', 50007)

To test video streaming , use ffmpeg.

$ ffmpeg -f dshow -i video="Integrated Camera" -preset ultrafast -vcodec libx264 -tune zerolatency -b 900k -f mpegts udp://127.0.0.1:11111

Note: If you can't work streaming with ffmpeg and have an error with OpenCV(4.2.0-openvino) like MFX: Unsupported extension:, you should donwload opencv_videoio_ffmpeg420_64.dll in opencv\bin.

See this forums in details. ValueError: cannot reshape array

Video capture does not display

Make sure firewall settings on your PC. Tello sends video stream to your PC via udp port 11111, which is needed to be open.

Store bootstrap and jQuery on your pc

bootstrap, jQuery and font awesome are stored locally since you have no internet connection while connecting Tello. (see static/js , static/css)

Tested with these version:

jquery-3.3.1
bootstrap-4.1.3
font awesome-free-5.5.0
popper.js-1.14.5 (umd)

kodamap/tellooo