The Darkflow is the implementation of YOLO (originally written in C) in Python/TensorFlow. For a detailed overview of Darkflow the reader is encouraged to check the Darkflow github repository where the installtion process along with other useful information can be found. If the reader is interested to train Darkflow from scratch on a different dataset it is recommended to download Darkflow from the above github repository and follow the installation procedure there. However, if the reader is interested to replicate this work please download the content of this repository and follow the installation procedure in 1.1 (below) and skip to section 2.
According to Darkflow page there are three different ways to install Darkflow. In this tutorial we will follow the first method. First, you need to navigate to the Darkflow page. Next, click on the "clone or download" to download the entire repository as a ".ZIP" file or use the url to clone the repository on your computer (you need to have git insalled on your computer). Next, unzip the contents of the file. After unziping the contents navigate to the "root" directory where the setup.py along with other files and directories(e.g. cfg directory) is located. On the command line (inside the root directory where the setup.py file is located) enter the following command (make sure you have python3 installed on your system):
python3 setup.py build_ext --inplace
Depending on whether you have the necessary python packages on your system you may receive error messages! Cython and opencv-python are the two most likely packages that you may get error messages about. Make sure you have them installed and python has access to them. Once the setup.py has run successfully, you can test the Darkflow model by exceuting the following command in the root directory.
flow --imgdir sample_img/ --model cfg/tiny-yolo.cfg --load bin/tiny-yolo.weights --gpu 1.0
Flow is an executive command. Flow should be set in your computer's path (if it is not set by default). By executing the above line Darkflow looks inside the sample_image directory for test images. Next, it uses tiny-yolo.cfg configuration file located inside cfg directory. Then, it takes the tiny-yolo.weights located inside bin directory. If your system has GPU installed and enabled it will run on GPU. If the model ran successfully you would be able to see the results inside out directory located in sample_image.
You can have access to several different configuration files inside cfg directory. There are several cfg files in the cfg directory each for a different YOLO flavor (including tiny-YOLO, YOLO-VOC, and YOLO; read this page for a description of each flavor). Each cfg file contains the structure of the model along with the required parameters for traianing that model (e.g. learning rate, decay rate, momentum). For training a model the cfg file and the path to the cfg file's location has to be given along with the --model flag (similar to the code above).
Depending on the model you are planning to use the appropriate weight file needs to be downloaded3 from this google drive. (The same link can be accessed by navigating to the intro section of this page.) Once the right weight file has downloaded put it inside the bin directory so the flow can have access to it (e.g. bin/tiny-yolo.weights).
To create your annotation file you need to follow the YOLO annotation format which requires a .xml format. For the purpose of the car detection model labelImg was used. LabelImg will allow you to store annotations and class names in both .txt and .xml formats.
In order to train and run Darkflow on the police car dataset, make sure you have the following files and directories after downloading and unziping the current github repository.
- annotations directory
- bin directory
- built_graph directory
- cfg directory
- ckpt directory
- images directory
- labels.txt file
The following shows the structure of the Darkflow police car detection model:
root
|
|___ /annotations
| |___ annotation.xml (your .xml files)
|
|___ /bin
| |___ weight files (the original and renamed weight files)
|
|___ /built_graph
| |___ .meta and .pb files
|
|___ /cfg
| |___ .cfg files (the original and renamed one)
|
|___ /ckpt
| |___ check point files will be stroed here
|
|___ /images
| |___ your .png or .jpeg (place your training images here)
|
|___ labels.txt (here only police car label/class)
Where you keep the annotaions files. In case of this model there are as many .xml files as there are images of police cars. Each .xml file gives the coordinates of the ground truth box. For instance, example_01.xml file contains the coordiantes of the ground truth box around a police car in the image 01.
Where you keep your weight files. Notice that the Darkflow requires you to provide it with two identical weight files with different file names. For examples if you are using tiny-yolo model, you have to keep the original name of the weight file that is yolov2-tiny-voc.weights while changing the name of the second weight file to something different (e.g. yolov2-tiny-c2.weights). Please read the 5th part of the Training on your own dataset section here for more details.
This is the checkpoint directory. During the traingin session Darkflow will store a group of four different files inside this directory. This process happens repeatedly after a given number of epochs. An example is given below;
1- yolov2-tiny-voc_c2-250.data-00000-of-00001
2- yolov2-tiny-voc_c2-250.index
3- yolov2-tiny-voc_c2-250.meta
4- yolov2-tiny-voc_c2-250.profile
For the rest of this tutorial we need two of these files. First, the .data file which contains the updated weights after 250 epochs. And the .meta file. See the Training new model section here for further details on how to test your model from a given checkpoint.
Once the desired checkpoint has achived the following command has to be applied to convert the .data file to a protobuf file (.pb). The .pb file is the weight file that will be used in Jetson nano developer kit.
flow --model cfg/yolo-new.cfg --load -1 --savepb
flow --model cfg/yolo.cfg --load bin/yolo.weights --savepb
Remember that the .weight file in the command line above has to be replaced by desired .data file from the checkpoint (ckpt) directory. Please read the Save the built graph to a protobuf file (.pb) here for more details.
Put your configuration file inside this directory (e.g. yolov2-tiny-voc.cfg). There are two lines that need to be changed when it comes to training on custom datasets; number of classes and number of filters. Before making any changes to the any files remember that, similar to the weight files, a copy of the desired configuration file with a different name needs to be made and placed in the cfg directory. For instance, one may decide to use the original yolov2-tiny-voc.cfg file. Then, one has to make another copy of the same file with a different name, e.g. yolov2-tiny-voc_c2.cfg, and place it along with the original file in the cfg directory. Next, comes the filter and class changes. These changes need to be done to the renamed configuration file. At the end of the renamed file you can find both classes and filters parameters. Place the number of classes you are going to train your model on (in case of police car the original value is replaced with 1). However, for the numner of filters you have to follow the following equation;
num * (classes + 5)
Where one has to replace num with 5 and classes with the number classes you are training on. For example, in case of Police car detection model the number of classes is 1 and num is 5 which according to the equation the number of filters becomes 5*6 = 30. Thus, the original value of the filter in the renamed cfg file will be replaced with 30 (from 125 to 30). The user has to use the modified and renamed configuration file when training their model! Please read Training on your own dataset section here for more details.
This directory contains your training images (i.e. police car images).
This file contains the label/class of the objects to be detected (i.e. police car)
Once you have built the required file system and place the necessary files inside the correct directory you can run the following command to start traingin your model.
flow --model cfg/yolo-new.cfg --load bin/tiny-yolo.weights --train --gpu 1.0
The above line uses yolo-new configuration file along with tiny-yolo weights and runs on a GPU. For a complete list of manditory and optional flags type in the following command on the command line;
flow --h
You can find a good introduction on seting up your Jetson nano develper kit here, however, below you can find a very short summary of the steps you need to take to prepare your Jetson device.
1- You need to have a microSD card
2- Download Jetson Nano Developer Kit SD Card Image here
3- Write the image to your microSD card using Etcher (pick the one that suits your operating system)
4- After writing the image to your microSD card insert the card into your Jetson device
5- Power on and you are good to go!
The operating system on the Jetson is a linux operting system so to install the necessary python packages you can simply use the pip command. Once you have the python packages installed you need to install Darkflow on Jetson following one of the three methods provided here (also described in section 1.1 ). However, this time you do need to create the same file system structure. This time you only need to have the built_graph, and sample_img directories where you need to transfer the created .meta and .pb files from your local machine onto the Jetson and inside the built_graph directory. Your test images have to be stored inside the sample_img directory on the Jetson machine. Once you have install the Darkflow on Jetson and transfered the files and test images you can use the following command to make prediction on your test images.
flow --pbLoad built_graph/yolo.pb --metaLoad built_graph/yolo.meta --imgdir sample_img/
You can find and read more about the command above here.
In order to do real-time car detection you need to connect a usb camera to Jetson. It would be more convenient to install a camera software to enable Jetson to detect the camera. For the purpse of this work guvcview was used (you can use any softwares!). You can install it by simply typing;
sudo apt-get install guvcview
Once you have this software installed and after connecting your usb camera, the camera should be detected by Jetson and a window should pop up on the screen showing the live camera (if the camera was not detected open the guvcview and try to find the camera from inside the gucview). Finally, to do real-time detection you need to type in the following command from inside the root directory of the Darkflow (where the built_graph and sample_img directories are located);
flow --pbLoad built_graph/yolo.pb --metaLoad built_graph/yolo.meta --demo camera
Please read this section for more details.
In this section a general overview of the expansion header of the Jetson will be given. The expansion header will be used for sending signal to an LED.
In order to send signal from Jetson to the LED one needs to knwo which pins on the expansion header to use. Generally, the pins on the expansion header fall into several different categories (e.g. ground, 5v constant power). For the purpose of this work we will be dealing with three different pins; GND (i.e. ground), 5V (constant power), and GPIO (i.e. general purpose input/output) pins. To have a better understanding of the location and the functionality of each pin read this and this. Note that the pin layout of the Jetson follows that of the Raspbery Pi. (The reader is encouraged to do more research on the similarity of the Raspberry pi's 3 and Jetson's expansion headers!) In order to interact with the pins (i.e. send/receive signals to/from the pins) one has to install the Jetson.GPIO library. To do so, run the following command;
sudo pip install Jetson.GPIO
It is very likely that after importing the library you will get a permission error. In order to mitigate the error navigate to the /sys/class/gpio and change the permission of the export and unexport files from only write to both read and write for all users (use sudo chmod 666 export and the same for unexport). To test whether things work correctly type in the following command on the command line in a terminal and hit enter;
GPIO.getmode()
It should print out either 10 or 11 or other values. Here, 10 means GPIO.BOARD and 11 means GPIO.BCM. Check this for further details on what each value means. After installing the Jetson.GPIO library follow the instructions here under the Setting User Permissions section. You need to create a new group, add your username to the group and copy a .rules file to the given path in the /etc/udev/rules.d/ (see the instructions). Finally, you need to restart the Jetson so the permissions take effect. If you do not follow the instructions one has to change all the file permissions manually. You will save yourself time by following the instructions on permissions!
Note that there are four types of mode to be set but the two modes that are relavent to this work are BOARD and CBM. Depending on the mode you are in, pins can be accessed by their original numbers (written next to each pin on the header) or non-origianl numbers. Setting mode to BOARD you should use pin numbering on the board (e.g. 12 for pin 12) while setting mode to CBM requires gpio numbering (e.g. 79 for pin 12). Read [this](read this for more details) for further details.
The image below shows the layout of the bread board.
Initially, the gpio 12 was used (on Jetson's expansion header) but it did not work (I am not sure why)! Instead, pin 16 was used as the gpio pin (try different pins and see which ones work)!
In order to make the LED work you need to add the right resistor to the circut. A resistor with high value will not let enough current pass through the circut and the LED will remain off while a resistor with too little resistance will damage the LED as the current that will reach the LED will be more than what it can take. For the purpose of this work a resistor with 135 kilo ohm resistance was used. Note that in the image above you can see the color bands on the resistor which indicate the resistance of the resistor. Also, the legs of the LED are of different lengh with longer one beign positive (not always the case)! So one must place the LED in the bread board correctly. You can find more details on LEDs andpicking the right resistor here and here.
As the end goal is to send a signal to the LED once a police car is detected one has to make some changes to the Darkflow's predict.py file. This file is responisble for creating a bounding box around the detected object (here the police car). For the purpose of this work we add a few lines of code to this file so everytime the code tries to draw a bounding around the police car it, also, send a signal to the ondicated pin. The path to this file is given below;
/usr/local/lib/python3.6/dist-packages/darkflow/net/yolov2/predict.py)
https://developer.nvidia.com/embedded/learn/get-started-jetson-nano-devkit