vswarm

The vswarm package enables decentralized vision-based control of drone swarms without relying on inter-agent communication or visual fiducial markers.

This repository contains the source code accompanying our article:

F. Schilling, F. Schiano, and D. Floreano, "Vision-based Drone Flocking in Outdoor Environments," IEEE Robotics and Automation Letters (RA-L), vol. 6, no. 2, pp. 2954-2961, 2021. [arXiv] [IEEE Xplore] [Citation]

The package provides its main functionality in the following modules:

Visual detection of neighboring drones from omnidirectional images using a convolutional neural network
Relative localization based on camera parameters and the known physical size of the drones
Multi-target tracking to estimate the position and velocity of the drones
Multi-agent control using a Reynolds-inspired flocking algorithm

The following video provides a high-level overview of the method and a demonstration of vswarm in action:

Requirements

This package requires Ubuntu 18.04 (Bionic) and ROS Melodic. The simulation environment is based on Gazebo. The majority of the code is written in Python 3.

Installation

The installation instructions for our simulation environment can be found in a dedicated installation.md. The code can also run onboard a physical drone with a suitable companion computer such as the NVIDIA Jetson TX2.

Usage

To launch three drones in Gazebo:

roslaunch vswarm vswarm_gazebo.launch n:=3

Use the vswarm command to let the drones take off and switch to offboard mode (i.e. to start the vision-based flocking algorithm):

vswarm takeoff    # Drones take off and hover at 2.5m altitude
vswarm offboard   # Drones start flocking

By default, the drones are migrating in a square according to a set of predefined waypoints.

The parameters of the most important nodes (object detection, relative localization, multi-target tracking, and flocking) can be adjusted using the dynamic reconfigure GUI:

rqt_reconfigure

Note: The reason for the relatively low confidence score of the detections despite the visually simple environment is the fact that the detection model has never been trained on simulated data (but on the dataset below).

Dataset

The vswarm drone detection dataset (~2GB) can be downloaded here: [Google Drive] [SWITCHdrive].

For more information about the dataset, check out the dataset.md

Citation

If you use this work in an academic context, please cite the following article:

@article{schilling_vswarm_2021,
    title   = {Vision-based Drone Flocking in Outdoor Environments},
    author  = {Schilling, Fabian and Schiano, Fabrizio and Floreano, Dario},
    journal = {IEEE Robotics and Automation Letters},
    year    = {2021},
    volume  = {6},
    number  = {2},
    pages   = {2954--2961},
    doi     = {10.1109/LRA.2021.3062298},
    issn    = {2377-3766}
}

Acknowledgments

We used a variety of third party tools and software libraries which we would like to acknowledge:

ethz-asl/kalibr for camera calibration
ultralytics/yolov3 for training the detector
facontidavide/plotjuggler for visualizing PX4 ulog and ROS bag files
rfs_tracking_toolbox for heavy inspiration of the filter implementation

License

This project is released under the MIT License. Please refer to the LICENSE for more details.

lithiumhydride2/vswarm_cp