safe-control-gym

Physics-based CartPole and Quadrotor Gym environments (using PyBullet) with symbolic a priori dynamics (using CasADi) for learning-based control, and model-free and model-based reinforcement learning (RL).

These environments include (and evaluate) symbolic safety constraints and implement input, parameter, and dynamics disturbances to test the robustness and generalizability of control approaches. [PDF]

@article{brunke2021safe,
         title={Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning},
         author={Lukas Brunke and Melissa Greeff and Adam W. Hall and Zhaocong Yuan and Siqi Zhou and Jacopo Panerati and Angela P. Schoellig},
         journal = {Annual Review of Control, Robotics, and Autonomous Systems},
         year={2021},
         url = {https://arxiv.org/abs/2108.06266}}

To reproduce the results in the article, see branch ar.

@misc{yuan2021safecontrolgym,
      title={safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning},
      author={Zhaocong Yuan and Adam W. Hall and Siqi Zhou and Lukas Brunke and Melissa Greeff and Jacopo Panerati and Angela P. Schoellig},
      year={2021},
      eprint={2109.06325},
      archivePrefix={arXiv},
      primaryClass={cs.RO}}

To reproduce the results in the article, see branch submission.

Install on Ubuntu/macOS

Clone repo

git clone https://github.com/utiasDSL/safe-control-gym.git
cd safe-control-gym

Option A (recommended): using `conda`

Create and access a Python 3.10 environment using conda

conda create -n safe python=3.10
conda activate safe

Install the safe-control-gym repository

pip install --upgrade pip
pip install -e .

Option B: using venv and poetry

Create and access a Python 3.10 virtual environment using pyenv and venv

pyenv install 3.10
pyenv local 3.10
python3 -m venv safe
source safe/bin/activate
pip install --upgrade pip
pip install poetry
poetry install

Note

You may need to separately install gmp, a dependency of pycddlib:

conda install -c anaconda gmp

sudo apt-get install libgmp-dev

Architecture

Overview of safe-control-gym's API:

Configuration

Performance

We compare the sample efficiency of safe-control-gym with the original OpenAI Cartpole and PyBullet Gym's Inverted Pendulum, as well as gym-pybullet-drones. We choose the default physic simulation integration step of each project. We report performance results for open-loop, random action inputs. Note that the Bullet engine frequency reported for safe-control-gym is typically much finer grained for improved fidelity. safe-control-gym quadrotor environment is not as light-weight as gym-pybullet-drones but provides the same order of magnitude speed-up and several more safety features/symbolic models.

Environment	GUI	Control Freq.	PyBullet Freq.	Constraints & Disturbances^	Speed-Up^^
Gym cartpole	True	50Hz	N/A	No	1.16x
InvPenPyBulletEnv	False	60Hz	60Hz	No	158.29x
cartpole	True	50Hz	50Hz	No	0.85x
cartpole	False	50Hz	1000Hz	No	24.73x
cartpole	False	50Hz	1000Hz	Yes	22.39x

gym-pyb-drones	True	48Hz	240Hz	No	2.43x
gym-pyb-drones	False	50Hz	1000Hz	No	21.50x
quadrotor	True	60Hz	240Hz	No	0.74x
quadrotor	False	50Hz	1000Hz	No	9.28x
quadrotor	False	50Hz	1000Hz	Yes	7.62x

^ Whether the environment includes a default set of constraints and disturbances

^^ Speed-up = Elapsed Simulation Time / Elapsed Wall Clock Time; on a 2.30GHz Quad-Core i7-1068NG7 with 32GB 3733MHz LPDDR4X; no GPU

Getting Started

Familiarize with APIs and environments with the scripts in examples/

cd ./examples/                                                                    # Navigate to the examples folder
python3 pid/pid_experiment.py --algo pid --task quadrotor --overrides ./pid/config_overrides/quadrotor_3D/quadrotor_3D_tracking.yaml       # PID trajectory tracking with the 3D quadcopter
python3 no_controller/verbose_api.py --task cartpole --overrides no_controller/verbose_api.yaml               #  Printout of the extended safe-control-gym APIs

Systems Variables and 2D Quadrotor Lemniscate Trajectory Tracking

Verbose API Example

List of Implemented Controllers

List of Implemented Safety Filters

References

Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., & Schoellig, A. P. (2022). Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5, 411-444.
Yuan, Z., Hall, A. W., Zhou, S., Brunke, L., Greeff, M., Panerati, J., & Schoellig, A. P. (2022). safe-control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning in robotics. IEEE Robotics and Automation Letters, 7(4), 11142-11149.

Related Open-source Projects

gym-pybullet-drones: single and multi-quadrotor environments
stable-baselines3: PyTorch reinforcement learning algorithms
bullet3: multi-physics simulation engine
gym: OpenAI reinforcement learning toolkit
casadi: symbolic framework for numeric optimization
safety-gym: environments for safe exploration in RL
realworldrl_suite: real-world RL challenge framework
casadi: symbolic framework for numeric optimization

TODOs (August 2022)

Publish to PyPI
Create resource list with papers, projects, blog posts (Cat's, etc.) using safe-control-gym

University of Toronto's Dynamic Systems Lab / Vector Institute for Artificial Intelligence

swuuu/safe-control-gym