/MTF

Modular Tracking Framework

Primary LanguageC++BSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

Welcome to the home of Modular Tracking Framework (MTF) - a highly efficient and extensible library for registration based tracking that utilizes a modular decomposition of trackers in this domain. Each tracker within this framework comprises the following 3 modules:

  1. Search Method (SM): ESM, IC, IA, FC, FA, LM, NN, PF, RANSAC or LMS
  2. Appearance Model (AM): SSD, SAD, ZNCC, SCV, LSCV, NCC, MI, CCRE, KLD, SSIM, SPSS, RIU, NGF, PCA or DFM
    • Illumination Model (ILM): GB, PGB and RBF
  3. State Space Model (SSM): Spline (50+ DOF), TPS (50+ DOF), Homography (8 DOF), Affine (6 DOF), Similitude (4 DOF), Isometry (3 DOF), Transcaling (3 DOF - translation + isotropic scaling) or Translation (2 DOF)

Please refer this paper for more details on the system design and this one for some preliminary results. There is also a dedicated website where Doxygen documentation will soon be available along with detailed tutorials and examples. It also provides several datasets formatted to work with MTF.

The library is implemented entirely in C++ though a Python interface called pyMTF also exists and works seamlessly with our Python Tracking Framework. A Matlab interface similar to Mexvision is currently under development too. We also provide a simple interface for ROS called mtf_bridge for seamless integration with robotics applications. A ROS package that uses it to exemplify integration of MTF with ROS is present in the ROS sub folder.

Currently, MTF only works under Unix though support for Windows is under active development and will be added soon. Though it has been tested comprehensively only under Linux, specifically Ubuntu 14.04, it should work on Macintosh systems too (see Compile/Runtime Notes section below for resolving possible issues).

MTF is provided under BSD license and so is free for research and commercial applications. We do request, however, that this paper be cited by any publications resulting from projects that use MTF so more people can get to know about and benefit from it. Finally, if any issues are encountered while installing or running the library, please create an entry in the issues section and we will do our best to resolve it as soon as possible.

Installation:

  • Prerequisites:
    • MTF uses some C++11 features so a supporting compiler is needed (GCC 4.7 or newer)
    • Eigen should be installed and added to the C/C++ include paths. This can be done, for instance, by running echo "export C_INCLUDE_PATH=${C_INCLUDE_PATH}:/usr/include/eigen3" >> ~/.bashrc and echo "export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/include/eigen3" >> ~/.bashrc assuming that Eigen is installed in /usr/include/eigen3
    • OpenCV should be installed.
      • OpenCV 2.4.x is recommended since possible compatibility issues with OpenCV 3.x may prevent successful compilation (see Compile/Runtime Notes section below)
    • FLANN library and its dependency HDF5 should be installed for the NN search method
      • NN can be disabled at compile time using nn=0 switch if these are not available (see compile time switches)
    • Boost Library should be installed
    • Intel TBB / OpenMP should be installed if parallelization is to be enabled.
    • ViSP library should be installed if its template tracker module or input pipeline is enabled during compilation (see compile time switches).
      • Note that version 3.0.0+ is required. The Ubuntu apt package is 2.8 and is therefore incompatible.
    • Caffe is needed for some optional modules including FMaps, Regnet and GOTURN if these are enabled during compilation
    • Xvision should be installed if it is enabled during compilation (see compile time switches).
      • Not recommended as Xvision is very difficult to install and configure on modern systems
  • Download the source code as zip file or clone using git clone https://github.com/abhineet123/MTF.git.
  • MTF comes with both a make and a cmake build system where the former is recommended for developers/contributors as it offers finer level of control while the latter is for users of the library who only want to install it once (or when the former does not work). For cmake, first use the standard method (i.e. mkdir build && cd build && cmake ..) to create the makefile and then use one of the make commands as specified below.
  • Use one of the following make commands to compile and install the library and the demo application:
    • make or make mtf : compiles the shared library (libmtf.so) to the build directory (Build/Release)
    • make install : compiles the shared library if needed and copies it to /usr/local/lib; also copies the headers to /usr/local/include/mtf; (use make install_lib or make install_header for only one of the two); if third party trackers are enabled, their respective library files will be installed too;
      • this needs administrative (sudo) privilege; if not available, the variables MTF_LIB_INSTALL_DIR and MTF_HEADER_INSTALL_DIR in the makefile can be modified to install elsewhere. This can be done either by editing the file itself or providing these with the make command as: make install MTF_LIB_INSTALL_DIR=<library_installation_dir> MTF_HEADER_INSTALL_DIR=<header_installation_dir>
        • these folders should be present in LD_LIBRARY_PATH and C_INCLUDE_PATH/CPLUS_INCLUDE_PATH environment variables respectively so any application using MTF can find the library and headers.
    • make exe : compiles the example file Examples/src/runMTF.cc to create an executable called runMTF that uses this library to track objects. This too is placed in the build directory.
      • library should be installed before running this, otherwise linking will not succeed
    • make install_exe: creates runMTF if needed and copies it to /usr/local/bin; this needs administrative privilege too - change MTF_EXEC_INSTALL_DIR in Examples/Examples.mak (or during compilation as above) if this is not available
    • make mtfi : all of the above - recommended command that compiles and installs the library and the executable
    • make py/make install_py : compile/install the Python interface to MTF - this creates a Python module called pyMTF.so that serves as a front end for running these trackers from Python.
      • usage of this module is fully demonstrated in the mtfTracker.py file in our Python Tracking Framework
      • installation location can be specified through MTF_PY_INSTALL_DIR (defaults to /usr/local/lib/python2.7/dist-packages/)
      • currently only supports Python 2.7 so will give compilation errors if Python 3 is also installed and set as default
    • make uav/make install_uav : compile/install an application called trackUAVTrajectory that tracks the trajectory of a UAV in a satellite image of the area over which it flew while capturing images from above
    • make mos/make install_mos : compile/install an application called createMosaic that constructs a live mosaic from a video of the region to be stitched
    • make all/make install_all : compile/install all example applications that come with MTF
    • make app app_name=<APPLICATION_NAME>: build a custom application that uses MTF with its source code in <APPLICATION_NAME>.cc; the compiled executable goes in the build directory;
      • location of the source file (<APPLICATION_NAME>.cc) can be specified through MTF_APP_SRC_DIR (defaults to Examples/src)
      • make mtfa will install it too - installation location can be specified through MTF_APP_INSTALL_DIR (defaults to the current folder)
    • Compile time switches for all of the above commands (only applicable to the make build system - for cmake there are corresponding options for some of the switches that can be configured before building its makefile):
      • only_nt=1/nt=1 will enable only the Non Templated (NT) implementations of SMs and disable their templated versions that are extremely time consuming to compile though being faster at runtime (disabled by default)
        • can be very useful for rapid debugging of AMs and SSMs where headers need to be modified;
        • the NT implementation of NN only works with GNN since FLANN library needs its object to be templated on the AM;
      • nn=0 will disable the templated implementation of NN search method (enabled by default).
        • should be specified if FLANN is not available
        • only matters if the previous option is not specified
      • feat=1 will enable the Feature based grid tracker (disabled by default).
      • lt=0 will disable the third party open source learning based trackers - DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT - that are also bundled with this library (in ThirdParty subfolder) (enabled by default except MIL, DFT and GOTURN).
      • gtrn=1 will enable GOTURN deep learning based tracker (disabled by default)
        • requires Caffe to be installed and configured
          • if not installed in ~/caffe/build/install, specify CAFFE_INSTALL_DIR either at compile time or by editing ThirdParty/GOTURN/GOTURN.mak
        • optional data for use with it is available here - this should be extracted inside the MTF root folder before running GOTURN
      • dft=1 will enable the Descriptor Fields Tracker (disabled by default)
        • this is known to have issues with OpenCV 3.x so should not be enabled if that is being used
      • mil=1 will enable the Multiple Instance Learning tracker (disabled by default)
        • this too is known to have issues with OpenCV 3.x so should not be enabled if that is being used
      • vp=1 will enable ViSP template trackers and input pipeline (disabled by default).
        • if a runtime linking error is encountered, add ViSP library install path to LD_LIBRARY_PATH variable by running, for instance, echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/x86_64-linux-gnu/" >> ~/.bashrc
      • xv=1 will enable Xvision trackers and input pipeline (disabled by default and not recommended).
      • o=0 will compile the library in debug mode (libmtf_debug.so) and disable optimization - the corresponding executables will be suffixed with 'd' ( for example runMTFd)
      • ver=<version_postfix> will postfix all compiled library and executable files with the provided postfix so that multiple versions may coexist on the same system without conflict; e.g. ver=2 will create libmtf_v2.so and runMTF_v2 and same for all libraries created during MTF compilation (including those for third party trackers);
      • header_only=1 can be used with make exe or make mtfe to build a stand alone version of runMTF called runMTFh that does not need to link with libmtf.so
        • this disables third party trackers automatically as several of those have their own libraries built from within MTF
        • it will still need to link against external libraries that it uses (like OpenCV, FLANN and boost)
        • it may take some time (20-30 minutes) to compile as the entire library has to be consolidated into a single executable
        • this version should be faster in theory as the compiler has access to all code at once and can thus optimize better but tests so far have not found any significant difference in speed from the standard version
    • Clean up commands:
      • make clean : removes all the .o files and the .so file created during compilation from the Build folder.
      • make mtfc : also removes the executable.

Compile/Runtime Notes:

  • if an error like cannot find -l<name of library> (for instance cannot find -lhdf5 or cannot find -lmtf) occurs at compile time, location of the corresponding library should be added to LIBRARY_PATH and LD_LIBRARY_PATH environment variables in the .bashrc file.
    • For instance HDF5 installs in /usr/local/hdf5/lib by default so echo "export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/hdf5/lib" >> ~/.bashrc and echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/hdf5/lib" >> ~/.bashrc can be run to add its path there. Similarly, MTF installs to /usr/local/lib by default so use echo "export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc and echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc for it.
    • The bashrc file should be reloaded by either restarting the terminal or running . ~/.bashrc and compilation should be tried again.
  • similar compile time errors pertaining to missing header files may be encountered in which case the location of the include folder of the corresponding library should be added to C_INCLUDE_PATH and CPLUS_INCLUDE_PATH variables as for Eigen
  • if a runtime error like cannot find libmtf or something similar is encountered while running runMTF, /usr/local/lib should be added to LIBRARY_PATH and LD_LIBRARY_PATH environment variables as above (assuming the default MTF installation directory has not been changed - otherwise the new path should be used instead).
  • if MTF had earlier been compiled with only_nt=0 (or nn=1) and is recompiled with only_nt=1 (or nn=0) such that the executable does not get recompiled, a runtime error of the type: symbol lookup error: runMTF: undefined symbol: may occur on running the executable; to resolve this, remove runMTF (or whatever executable is causing the error) from the build folder (Build/Release or Build/Debug) by running, for instance rm Build/Release/runMTF and recompile; this happens because using only_nt=1 or nn=0 turns off part of the code in the library but the executable will continue to expect these to be present till it is recompiled;
  • Using MTF on Macintosh Systems
    • if make build system is used, some thirdparty modules might not compile successfully if cmake uses clang instead of gcc as the default compiler; in such cases compiling with lt=0 to disable all thirdparty modules is the best option.
    • if cmake build system is used, then export CC=/usr/bin/gcc and export CXX=/use/bin/g++ must be run so that cmake uses gcc during compilation.
    • if FLANN based NN is enabled with cmake, environment variables may need to be set for hdf5 to be found using, for instance, export CMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}:/Applications/HDF_Group/HDF5/1.10.0-patch1/
  • Using MTF with OpenCV 3.x
    • MTF has been tested and found to work with OpenCV 3.1 installed using this method (option 2) on Ubuntu 14.04 but general compatibility with other system configurations is not guaranteed since comprehensive testing has only been doe with 2.4.x.
    • third party trackers in particular are likely to have issues with OpenCV 3.x as many use legacy versions so these should be disabled using lt=0 if compilation or linking errors pertaining to these are found.
    • if a linking error of type /usr/bin/ld: cannot find -lippicv occurs, remove -ippicv from opencv pkg-config configuration file which is usually located at /usr/local/lib/pkgconfig/opencv.pc or follow the procedures suggested here

[](* if MTF had earlier been compiled with only_nt=0 (or nn=1) and is recompiled with only_nt=1 (or nn=0) or vice versa, some linking error with missing symbols related to mtf::NNParams or mtf::FLANNParams may occur when compiling the executable; to fix this, remove NNParams.o from the build folder (Build/Release or Build/Debug) by running, for instance rm Build/Release/NNParams.o so this recompiles too; some of the code in NNParams is conditional on whether standard version of NN (that uses FLANN) is enabled or not so having a wrong version compiled causes this error and, unfortunately, there is no way to automatically recompile this file when FLANN based NN is enabled or disabled through compile time switches;)

[](* switching from only_nt=1 to only_nt=0 can sometimes also lead to linking errors of the type: undefined reference to mtf::gnn::GNN<[some AM]>::[some function]; to resolve this, remove GNN.oandFGNN.ofrom the build folder by running, for instancerm Build/Release/GNN.o Build/Release/FGNN.o` and recompile.)

Setting Parameters:

MTF parameters can be specified either in the cfg files present in the Config sub folder or from the command line. Please refer the ReadMe in the Config sub folder or this configuration page for detailed instructions.

Some preconfigured cfg files are provided here to get the system up and running quickly. Config files involving dataset sequences assume that all datasets are located in the Datasets sub folder under the active directory. The zip file should be extracted in the Config folder to replace the existing cfg files there (if any):

Running the demo application:

Use either make run or runMTF to start the tracking application using the settings specified as above.

Building the ROS package:

A simple ROS package called mtf_bridge that demonstrates the integration of MTF with ROS applications is included in the ROS sub directory. To build it, first create a catkin workspace by running catkin_init_workspace from ROS/src and then build this package by running catkin_make from ROS. Also refer this page for more details on building and running ROS packages.

For Developers

Adding a new Appearance Model (AM) or State Space Model (SSM):

make

  1. Modify the .mak file in the respective sub directory to:
    • add the name of the AM/SSM to the variable APPEARANCE_MODELS or STATE_SPACE_MODELS respectively
    • add rule to compile the .o of the new AM/SSM that is dependent on its source and header files - simplest method would be to copy an existing command and change the names of the model.
  2. Modify makefile to add any extra dependencies that the new files need to include under variable FLAGS64 and extra libraries to link against under LIB_MTF_LIBS

cmake

  1. Modify the .cmake files in the respective sub directories:

    • Modify the AM/AM.cmake file to add the name of the AM to set(APPEARANCE_MODELS …)
    • Modify the SSM/SSM.cmake file to add the name of the SSM to set(STATE_SPACE_MODELS …)

  2. Modify "mtf.h" to add a new AM/SSM option in the overloaded function getTrackerObj (starting lines 498 and 533 respectively) that can be used with your config file to pick the AM/SSM, and create an object with this selected model.

  3. Modify "Macros/register.h" to add the new AM/SSM class name under _REGISTER_TRACKERS_AM/_REGISTER_HTRACKERS_AM or _REGISTER_TRACKERS/_REGISTER_TRACKERS_SSM/_REGISTER_HTRACKERS/_REGISTER_HTRACKERS_SSM respectively

All steps are identical for adding a new Search Method (SM) too except the last one which is not needed. Instead this header needs to be included and the appropriate macro needs to be called from its source file to register the SM with all existing AMs and SSMs. Refer to the last 2 lines of the .cc file of any existing SM to see how to do this.

Example: Implementing a minimalistic AM that can be used with Nearest Neighbour search method:

You need to create a new derived class from AppearanceModel. Implement the following functions:

  1. initializePixVals/updatePixVals : takes the current location of the object as a 2xN matrix where N = no. of sampled points and each column contains the x,y coordinates of one point. By default, it is simply supposed to extract the pixel values at these locations from the current image but there is no restriction by design and you are free to compute anything from these pixel values.
  2. initializeDistFeat/updateDistFeat : these compute a distance feature transform from the current patch.
  3. getDistFeat: returns a pointer to an array containing the distance feature vector computed by the above function. There is also an overloaded variant of updateDistFeat that takes a pointer as input and directly writes this feature vector to the pre-allocated array pointed to by this pointer.
  4. distance functor (operator()): computes a scalar that measures the dissimilarity or distance between two feature vectors (obtained using the previous two functions).

Example: Implementing a minimalistic AM that can be used with Particle Filter search method:

You need to create a new derived class from AppearanceModel. Implement the following functions:

  1. initializePixVals/updatePixVals : Same as with NN. No need to overwrite if you just need to extract the pixel values
  2. getLikelihood: get the likelihood/weights of the particle
  3. initializeSimilarity: initialize some class variables like the initial features.
  4. updateSimilarity: update the similarity between current patch and the template