lingerun/deepgaze

Computer Vision library for human-computer interaction. It implements Head Pose and Gaze Direction Estimation Using Convolutional Neural Networks, Skin Detection through Backprojection, Motion Detection and Tracking, Saliency Map.

Updates

Update 10/06/2017 The PDF of the article "Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods" is available for free download in the next 50 days using this special link

Update 04/06/2017 Article "Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods" have been accepted for publication in Pattern Recogntion (Elsevier). The Deepgaze CNN head pose estimator module is based on this work.

Update 31/05/2017 Implementation of the new package saliency_map.py. The package contains an implementation of the FASA algorithm for saliency detection [example] [wiki]

Update 22/03/2017 Fixed a bug in mask_analysis.py and almost completed a more robust version of the CNN head pose estimator.

Update 21/11/2016 New package color_classification.py. The package contains an implementation of the histogram intersection algorithm for colour classification [example]. Read more on my blog post.

Update 04/11/2016 New package motion_tracking.py. The package contains an implementation of Particle Filter, which can be used to follow a target in presence of noisy measurements [example] [video]

Update 01/11/2016 Comparison of three different motion detection algorithms [example] [video]

Update 28/10/2016 New package motion_detection.py. Using the classes in this package it is possible to track moving objects through background subtraction. Possible applications of this algorithm are people detection, vehicle detection and tracking [example]

Update 21/10/2016: New package color_detection.py added. Using the classes inside this package it is possible to detect colors [example], skin [example] and faces [example]

Update 19/10/2016: Working example on how to use CNNs for the pitch estimation [code] Working example on how to use CNNs for both yaw and pitch estimation [code]

Update 05/10/2016: Working example on how to use CNNs for head pose estimation (for the moment only yaw angle) [code]

Update 20/09/2016: Work in progress. The code provided at the moment does not still implement gaze detection. There is a beta version of the class which implements the CNN head pose estimator of the yaw angle [code]. You can use it loading the variables stored in this [file].

What is Deepgaze?

Deepgaze is a library for human-computer interaction, people detection and tracking which uses Convolutional Neural Networks (CNNs) for face detection, head pose estimation and classification. The focus of attention of a person can be approximately estimated finding the head orientation. This is particularly useful when the eyes are covered, or when the user is too far from the camera to grab the eye region with a good resolution. When the eye region is visible it is possible to estimate the gaze direction, which is much more informative and can give a good indication of the FOA. Deepgaze contains useful packages for:

• Head pose estimation (Perspective-n-Point, Convolutional Neural Networks)
• Face detection (Haar Cascade)
• Skin and color detection (Range detection, Backprojection)
• Histogram-based classification (Histogram Intersection)
• Motion detection (Frame differencing, MOG, MOG2)
• Motion tracking (Particle filter)
• Saliency map (FASA)

Deepgaze is based on OpenCV and Tensorflow, some of the best libraries in computer vision and machine learning. Deepgaze is an open source project and any contribution is appreciated, feel free to fork the repository and propose integrations.

This library is the result of a recent work, if you use the library in academic work please cite the following paper:

Patacchiola, M., & Cangelosi, A. (2017). Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods. Pattern Recognition, http://dx.doi.org/10.1016/j.patcog.2017.06.009.

Why should I use Deepgaze?

Because Deepgaze makes your life easier! The implementation of many algorithms such as face detectors, pose estimators and object classificators can be painful. Deepgaze has been designed to implement those algorithms in a few lines of code. Deepgaze is helpful for both beginners and advanced users who want to save time. All the code contained in Deepgaze is optimised and it is based on state-of-the-art algorithms.

What is a Convolutional Neural Network?

A convolutional neural network (CNN, or ConvNet) is a type of feed-forward artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field. Convolutional networks were inspired by biological processes and are variations of multilayer perceptrons designed to use minimal amounts of preprocessing. They have wide applications in image and video recognition, recommender systems and natural language processing [wiki]

Main contributors

This is an updated list of the main contributors of the project. We are looking for contributors! If you want to contribute adding a new module or improving an existing one, send an email to our team!

• Massimiliano Patacchiola: project leader and main contributor
• Joel Gooch: head pose estimation
• Ishit Mehta: CNN-cascade face detection
• Luca Surace: Haar-cascade multi-face detection

Prerequisites

The current version of Deepgaze is based on Python 2.7, a porting for Python 3.0 has been scheduled for the next year.

To use the libray you have to install:

• Numpy [link]

sudo pip install numpy

• OpenCV 2.x (not compatible with OpenCV >= 3.x) [link]

sudo apt-get install libopencv-dev python-opencv

• Tensorflow [link]

sudo pip install tensorflow

Some examples may require additional libraries:

• dlib [link]

Installation

Download the repository from [here] or clone it using git:

git clone https://github.com/mpatacchiola/deepgaze.git

To install the package you have to enter in the Deepgaze folder and run the setup.py script (it may require root privileges):

cd deepgaze
sudo python setup.py install

If you want to track all the installed files you can record the installation process in a text file using the --record flag:

sudo python setup.py install --record record.txt

Done! Now give a look to the examples below.

Examples

• Head Pose Estimation using the Perspective-n-Point algorithm in OpenCV [code] [video]

• Head Pose Estimation in-the-wild using Perspective-n-Point and dlib face detector [code] [video]

• Head Pose Estimation in images using Convolutional Neural Networks [code]

• Color detection using the Histogram Backprojection algorithm [blog] [code]

• Skin detection using the HSV range color detector [code]

• Face detection using the HSV range color detector [code]

• Motion detection and tracking using frame differencing on a video streaming [code]

• Motion detection and tracking comparison of three algorithms on a video streaming [code] [video]

• Motion tracking with unstable measurements using Particle Filter [code] [video]

• Motion tracking with multiple backprojection for playing chrome's dinosaur game [blog] [code] [video]

• Classify object using their colour fingerprint (histogram intersection) [blog] [code]

• Implementation of the FASA (Fast, Accurate, and Size-Aware Salient Object Detection) algorithm [code] [wiki] [link]

Acknowledgements

• The example "head pose estimation using Perspective-n-Point" is partially based on the C++ version you can find here, and on the workshop "Developing an attention system for a social robot" which was part of the 2nd International Summer School on Social Human-Robot Interaction.

• To implement the Bayes and Particle Filters I followed the great repository of rlabbe which you can find here