gradSim: Differentiable simulation for system identification and visuomotor control
gradSim is a unified differentiable rendering and multiphysics framework that allows solving a range of control and parameter estimation tasks (rigid bodies, deformable solids, and cloth) directly from images/video. Our unified computation graph — spanning from the dynamics and through the rendering process — enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
We recommend creating a conda
environment (or alternatively, a virtualenv
environment) before proceeding. Recommended python versions are 3.6 or higher. Tested versions include python 3.6 and 3.7.
conda create -n gradsim python=3.7
conda activate gradsim
In a conda
or virtualenv
environment, first install your favourite version of PyTorch
(versions >=1.3.0
recommended; untested with lower) by following instructions from pytorch.org
Our code relies on NVIDIA kaolin
functionality. Please install NVIDIA Kaolin by following instructions from the official docs.
After activating the environment (and installing PyTorch
and kaolin
), run
python setup.py build develop
This will install the gradsim
package, which you can now use! To smoketest and verify that the package has been installed, fire up your python interpreter and run the following quick commands.
>>> import gradsim
>>> gradsim.__version__
gradsim
heavily relies on the Usd
package from pxr
, which is installed by default when you install kaolin
. For most of the examples to run, you will need to configure your PYTHONPATH
environment variable so that the pxr
module (and its Usd
submodule) are available to python.
First, navigate to the directory where you cloned the kaolin
repository.
cd path/to/kaolin/root/directory
In this directory, execute the following commands:
export KAOLIN_HOME=$PWD
export PYTHONPATH=${KAOLIN_HOME}/'build/target-deps/nv_usd/release/lib/python'
export LD_LIBRARY_PATH=${KAOLIN_HOME}/'build/target-deps/nv_usd/release/lib/'
Only do this if you have NOT followed step 4a. If you have already followed step 4a, skip this step
Instead of step 4a, you may use the provided installer to install the pxr
module (and its Usd
submodule).
./buildusd.sh
IMPORTANT: After the above step, run the following command from the base directory of this repo (i.e., directory containing this README
file)
source setenv.sh
This script will setup your PYTHONPATH
and LD_LIBRARY_PATH
environment variables.
The Ninja build system must be installed prior to running many of our demos.
conda install -c conda-forge ninja
On Mac systems, if Py3ODE is not installed, the build of that package might fail, in which case, please clone their github repo into a separate folder:
cd .. # assuming you are currently in the directory of this project
git clone https://github.com/filipeabperes/Py3ODE.git
cd Py3ODE
And compile and install their Python module,...
CC=/usr/bin/clang CFLAGS="-O -stdlib=libc++ \
-I/Library/Developer/CommandLineTools/usr/include/c++/v1/" \
python setup.py build
python setup.py install
Our code release includes a number of examples. Here is a brief description of each example, and a sample launch command.
NOTE: Each script takes in a number of commandline arguments. So please read through the code for a complete idea.
NOTE: All of the following scripts are contained in the
examples
directory
This example demonstrates a cube falling under the influence of gravity. Run this using the following command. Best run from inside the examples
directory.
cd examples
python hellogradsim.py
This will save a rendered gif
at examples/cache/hellogradsim.gif
. The quality of this gif might be a tad poor if using the cube.obj
model (from examples/sampledata/cube.obj
) as it has just 12 triangles.
This examples demonstrates how to apply forces at arbitrary chosen points on a rigid body, as opposed to only the center of mass (such forces generate rotations, etc. which are of interest).
cd examples
python demoforces.py
This example demonstrates the usage of gradsim
to recover the (unknown) mass of an object (represented as a triangle mesh), acted upon by an arbitrarily chosen force.
cd examples
python demo_mass_known_shape.py
For a list of available commandline arguments, run
python demo_mass_known_shape.py --help
This examples optimizes for the length of a simple pendulum from a video sequence.
python demo_pendulum.py
For a list of available commandline arguments, run
python demo_pendulum.py --help
This examples optimizes for the lengths of a double pendulum from a video sequence.
python demo_double_pendulum.py
For a list of available commandline arguments, run
python demo_double_pendulum.py --help
A 2D bouncing ball demo. Assumes the ball is bounced with an initial horizontal velocity, and that gravity is the only external force acting on it.
python demo_bounce2d.py
For a list of available commandline arguments, run
python demo_bounce2d.py --help
A simple dflex
smoketest. If dflex
is being used for the first time, all of the kernels will be compiled.
python hellodflex.py
Optimizes the parameters of an FEM model (uses the NeoHookean hyperelasticity model). In particular, the demo shows how to optimize for the mass of each particle in the deformable mesh to math a rendered video sequence.
python demo_fem.py
A cloth parameter optimization demo. Estimate the velocity of each particle in a cloth mesh, to match a target rendering (video sequence).
python demo_cloth.py
A visuomotor control demo that trains a neural network to actuate the tetrahedrons of a gear
mesh, in order for it to achieve a specified target pose.
python control_fem_gear.py
A visuomotor control demo that optimizes the initial velocity of a cloth, in order for it to achieve a specified target pose.
python control_cloth.py
A visuomotor control demo that optimizes the actuation phases of a walker, to make it walk towards a target pose (specified as an image).
python control_walker.py
Files used for running experiment 1 of the gradSim ICLR submission
Scripts that plot loss landscapes found in the paper / appendix.
A subset of scripts used to generate visualizations for the paper.
In the plots
folder, we release all data and scripts used in the plots for the ICLR version of our paper.
For attribution in academic contexts, please cite this work as
Citation
Jatavallabhula and Macklin et al., "gradSim: Differentiable simulation for system identification and visuomotor control", ICLR 2021.
BibTeX citation
@article{gradsim,
title = {gradSim: Differentiable simulation for system identification and visuomotor control},
author = {Krishna Murthy Jatavallabhula and Miles Macklin and Florian Golemo and Vikram Voleti and Linda Petrini and Martin Weiss and Breandan Considine and Jerome Parent-Levesque and Kevin Xie and Kenny Erleben and Liam Paull and Florian Shkurti and Derek Nowrouzezahrai and Sanja Fidler},
journal = {International Conference on Learning Representations (ICLR)},
year = {2021},
url = {https://openreview.net/forum?id=c_E8kFWfhp0},
pdf = {https://openreview.net/pdf?id=c_E8kFWfhp0},
}
If you find this code useful, please also consider citing Kaolin and DFlex.
@misc{kaolin,
author = {Clement Fuji-Tsang and Masha Shugrina and Jean-Francois Lafleche and Charles Loop and Towaki Takikawa and Jiehan Wang and Wenzheng Chen and Sanja Fidler and Jason Gorski and Rev Lebaredian and Jianing Li and Michael Li and Krishna Murthy Jatavallabhula and Artem Rozantsev and Frank Shen and Edward Smith and Gavriel State and Tommy Xiang},
title = {Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research},
howpublished = {\url{https://github.com/NVIDIAGameWorks/kaolin}},
year = {2019}
}