SynthDet: An end-to-end object detection pipeline using synthetic data

Overview

SynthDet is an open source project that demonstrates an end-to-end object detection pipeline using synthetic image data. The project includes all the code and assets for generating a synthetic dataset in Unity. Based on recent research, SynthDet utilizes Unity's Perception package to generate highly randomized images of 63 common grocery products (example: cereal boxes and candy) and export them along with appropriate labels and annotations (2D bounding boxes). The synthetic dataset generated can then be used to train a deep learning based object detection model. This project is geared towards ML practitioners and enthusiasts who are actively exploring synthetic data or just looking to get started.

Getting started with SynthDet

Components

SynthDet Unity Project - Sample computer vision data generation project, demonstrating proper integration and usage of the Perception package for environment randomization and ground-truth generation.
3D Assets - High quality models of 63 commonly found grocery products
Unity's Perception package.

Unity Project overview

This project utilizes the Unity Perception package for randomizing the environment and capturing ground-truth on each frame. Randomization includes elements such as lighting, camera post processing, object placement, and background.

Visit the Unity project documentation page for a brief overview on how ground truth generation and domain randomization are achieved in SynthDet.

Tutorials

In addition to the above, in order to learn how to create a project like SynthDet from scratch using the Perception package, we recommend you follow the Perception Tutorial.

Additional documentation

Inspiration

SynthDet was inspired by the following research paper from Google Cloud AI:

Hinterstoisser, S., Pauly, O., Heibel, H., Marek, M., & Bokeloh, M. (2019). An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detection.

Unity project development

The original version of the SynthDet Unity project was developed in tandem with the early versions of Unity's Perception package. This project closely followed the synthetic data generation method introduced by the above mentioned Google Cloud AI paper. However, the original project did not use the randomization toolset that was introduced in later versions of the Perception package. To access this original project, and for more details on how it was implemented to replicate the research paper, please visit the SynthDet_Original branch of this repository. The results reported in our related blog posts were based on this original project. That said, early experiments with datasets generated using the current version of the project have shown very similar model-training performance to that of the original one.

Support

For general questions or concerns please contact the Unity Computer Vision team at computer-vision@unity3d.com.

For feedback, bugs, or other issues please file a GitHub issue and the Unity Computer Vision team will investigate the issue as soon as possible.

Citation

If you find this project useful, consider citing it using:

@misc{synthdet2020,
    title={Training a performant object detection {ML} model on synthetic data using {U}nity {P}erception tools},
    author={You-Cyuan Jhang and Adam Palmar and Bowen Li and Saurav Dhakad and Sanjay Kumar Vishwakarma and Jonathan Hogins and Adam Crespi and Chris Kerr and Sharmila Chockalingam and Cesar Romero and Alex Thaman and Sujoy Ganguly},
    howpublished = {\url{https://blogs.unity3d.com/2020/09/17/training-a-performant-object-detection-ml-model-on-synthetic-data-using-unity-computer-vision-tools/}},
    journal={Unity Technologies Blog},
    publisher={Unity Technologies},
    year={2020},
    month={Sep}
}