leonardtang/tdw

ThreeDWorld simulation environment

ThreeDWorld (TDW)

A Platform for Interactive Multi-Modal Physical Simulation

ThreeDWorld (TDW) is a platform for interactive multi-modal physical simulation. With TDW, users can simulate high-fidelity sensory data and physical interactions between mobile agents and objects in a wide variety of rich 3D environments.

Getting Started (Read this first!)

Code of Conduct

Changelog

Website

License

API

Commands and Output Data

Document	Description
Command API	API for every command a controller can send to the build.
Command API Guide	Overview of how to send commands to the build.
Output Data	API for all output data a controller can receive from the build.

Python tdw module

Frontend

Document	Description
tdw module	Overview of the Python tdw module.
Controller	Base class for all controllers.
TDWUtils	Utility class.
AssetBundleCreator	Covert 3D models into TDW-compatible asset bundles.
PyImpact	Generate impact sounds at runtime.
DebugController	Child class of Controller that has useful debug features.
Librarian	"Librarians" hold asset bundle metadata records.
BinaryManager	Manage multiple instances of TDW builds on a remote server.
FluidTypes	Access different NVIDIA Flex fluid types.

Backend

Document	Description
Build	Helper functions for downloading the build.
PyPi	Helper functions for checking the version of the tdw module on PyPi.

Audio and Video

Document	Description
Impact Sounds	Generate impact sounds at runtime using PyImpact.
PyImpact	PyImpact API.
Audio/Video Recording	Record audio, video, or audio+video.
Remote rendering	How to render using xpra.

Avatars (Agents)

Document	Description
Observation Data	Different means of determining what an agent is observing.
The Sticky Mitten Avatar	Comprehensive documentation for a physics-driven agent with "sticky mittens".
Avatar Movement	Different techniques for moving agents.

Benchmarks and Speed

Document	Description
Benchmarks	Performance benchmarks.
Performance Optimizations	Increase simulation speed.

Examples

Document	Description
Example Controllers	Index of all example controllers in the repo.
Use Cases (see below)	The "use cases" section showcases "advanced" usage of TDW.

Misc.

Document	Description
VR	VR in TDW.
Humanoids	Add "humanoids" and play animations.
Releases	Release versioning in TDW.

Physics (PhysX and Flex)

Document	Description
Physics	Common physics problems and solutions.
NVIDIA Flex	Add soft bodies, cloth, and fluids to TDW.
FluidTypes	Access different NVIDIA Flex fluid types.

Remote Server

Document	Description
Docker	Create a Docker container for TDW.
Remote rendering	How to render using xpra.
BinaryManager	Manage multiple instances of TDW builds on a remote server.
bash scripts	Useful bash scripts for Linux.

Rendering and Photorealism

Document	Description
Asset Bundle Librarians	Overview of what asset bundles are, how to add objects, scenes, materials, HDRI skyboxes, and humanoids, and how to access each asset bundle's metadata.
Model Screenshotter	Generate images of every model in TDW.
Material Screenshotter	Generate images of every material in TDW.
Materials, textures, and colors	Defines materials, textures, and colors.
Depth of Field	Prevent blurry images and increase realism.
Remote rendering	How to render using xpra.
Observation Data	Different means of determining what an agent is observing.

Scene Setup

Document	Description
Scene Setup	Overview of how to set up a scene.
Asset Bundle Librarians	Overview of what asset bundles are, how to add objects, scenes, materials, HDRI skyboxes, and humanoids, and how to access each asset bundle's metadata.
Model Librarian	Overview of how to add objects and access metadata.
Rotation	Different means of rotating objects and agents in a scene.
Scene Reset	How to reset a scene.

TDW and 3D Objects

Document	Description
Model Librarian	Overview of how to add objects and access metadata.
Model Screenshotter	Generate images of every model in TDW.
Non-free models	Access the TDW "full model library".
Local 3D models	Add your own objects to TDW.
ShapeNet models	Convert ShapeNET models into TDW objects.
Composite Objects Creating Composite Objects	Use and create "composite objects".
AssetBundleCreator	API for the AssetBundleCreator class (used to convert 3D models into TDW-compatible asset bundles).
Rotation	Different means of rotating objects and agents in a scene.

Troubleshooting TDW

Document	Description
Debug TDW	Several strategies for debugging errors in your code.
DebugController	API for the DebugController class.
Depth of Field	Prevent blurry images and increase realism.
Performance Optimizations	Increase simulation speed.
OS X	Common OS X problems and solutions.
Physics	Common physics problems and solutions.

Use Cases

Use Case	Description
Image dataset	Generate 1.3M photorealistic images.
IntPhys (demo)	Demo of how to simulate IntPhys in TDW.
Humanoid videos	Generate a dataset of humanoid animations.
tdw_sound20k	Generate an audio dataset.
tdw_physics	Generate a physics dataset.
Rube Goldberg (demo)	Demo of complex physical interactions between objects, with PyImpact generation of impact sounds, set in a photorealistic scene.