Interactive Lego Machine Learning Environment

LTRON is an environment for interactive machine learning assembly problems using Lego bricks. It is based on LDraw the Open Model Repository, and uses additional data from LDCAD.

If you use LTRON in acadmic work, please cite our ECCV 2022 Paper:

@inproceedings{walsman2022break,
  author = {Aaron Walsman, Muru Zhang, Klemen Kotar, Karthik Desingh, Ali Farhadi, Dieter Fox},
  title = {Break and Make: Interactive Structural Understanding Using LEGO Bricks},
  booktitle = {European Conference on Computer Vision},
  year={2022}
}

Versions

The v1.0.0 branch is the code associated with the ECCV '22 paper above. The master branch contains active development towards version 2, which adds new features and streamlines some interface design issues, but is currently quite unstable and contains dependencies to non-public code. Stay tuned for future releases.

Examples

8661-1 - Carbon Star.mpd

31025 - Mountain Hut - Mountain Hut.mpd

10030-1 - Imperial Star Destroyer - UCS.mpd

Installation:

To install LTRON run:

pip install ltron
ltron_asset_installer

The first line will download LTRON from pypi and install it in your python path. The second line will download and install a set of models and part files that are necessary for LTRON. By default, these assets are installed to ~/.cache/ltron and ~/.cache/splendor. This will take around 3GB of space.

Release Versions:

Embodied AI Workshop, CVPR 2021: (pypi 0.0.X) (branch TODO).
ECCV 2022: (pypi 1.0.X) (branch v1.0.0)
Ongoing Work (not in pypi) (branch master)

Testing Things Out:

Once installed you should be able to run ltron_viewer to interactively inspect a lego model. For example from the top directory you can run:

ltron_viewer "~/.cache/ltron/collections/omr/ldraw/8661-1 - Carbon Star.mpd"

Or:

ltron_viewer "~/.cache/ltron/collections/omr/ldraw/75050-1 - B-Wing.mpd"

Or:

ltron_viewer "~/.cache/ltron/collections/omr/ldraw/75060 - Slave I.mpd"

You can interact with the model by clicking on it and dragging the mouse around. LMB - Orbit. RMB - Pan. Scroll - Zoom. There are a few keys you can press h to hide the brick you are hovering over, v to show all hidden bricks and m to switch back and forth between mask mode and regular rendering. See other options in ltron/visualization/ltron_viewer.py.

Requirements:

gym
numpy
pyquaternion
gdown
tqdm
splendor-render

The splendor-render package only works on Ubuntu at the moment, and requires OpenGL 4.6. As long as you have a modern GPU with recent drivers you should be fine.

You may need to install freeglut:

sudo apt-get install freeglut3-dev

File Formats

LTRON uses the LDraw file formats (.dat, .ldr, .mpd) to describe bricks and models. We convert all bricks to obj files for use in splendor-render. These are installed to ~/.cache/splendor/ltron_assets_low or ~/.cache/splendor/ltron_assets_high.

The Open Model Repository files are installed to ~/.cache/ltron/collections/omr/ldraw.

Data Layout:

This section is current as of v1.0.X, but may be streamlined in future releases:

As noted above, the LTRON assets are placed by default in ~/.cache/ltron. This location can be changed by setting the $LTRON_HOME environment variable. Additionally, some rendering assets are placed in ~/.cache/splendor. This location can be changed by setting the $SPLENDOR_HOME` environment variable.

All LTRON datasets are registered in $LTRON_HOME/settings.cfg. In LTRON each dataset contains a json file that describes where to find the files associated with that dataset, and various metadata. These are listed in the [datasets] header in the settings.cfg file. These json files usually refer to a particular "collection" of ldraw files. You can think of a collection as a root file path where several ldraw files and/or episode zip files live. These locations are specified in the [collections] header in the settings.cfg file.

Looking the $LTRON_HOME/collections/random_construction_6_6/rc_6_6.json file we can see what kind of data it contains:

splits: a set of names corresponding to a block of files used for training or testing. rc_6_6.json contains nine splits: a train_N, test_N and train_episodes_N for N=2,4,8. The path to each of these splits starts with {random_construction_6_6}/... which tells the system to look for these files inside the random_construction_6_6 collection.
max_instances_per_scene: an upper bound on number of instances that exist in any of the ldraw files used in the dataset.
max_edges_per_scene: an upper bound on the number of connections between bricks that exist in any of the ldraw files used in this dataset.
shape_ids: a class label for each brick shape used in this dataset
class_ids: a separate class label for each brick color used in this dataset

Using this structure provides access to the following commands:

ltron.dataset.get_dataset_info('dataset_name') returns the contents of the json dataset file
ltron.dataset.get_dataset_paths('dataset_name', 'split_name', subset=None, rank=0, size=1) returns a list of paths belonging to a particular split.
ltron.dataset.get_zip_paths('dataset_name', 'split_name', subset=None, rank=0, size=1) returns a zipfile object and a list of its contents for a particular split.

LtronEnv Structure

We have tried to make the LtronEnv structure modular so that it can be customized for a variety of observation types, action spaces and tasks. Therefore the LtronEnv gym environment is a container for multiple components, each of which specify their own action/observation spaces and state variables. When LtronEnv's reset/step/etc. method is called, it executes each of its component's reset/step/etc. methods in order and accumulating the result into a dictionary based on the component's name. Currently only the Break and Make (ltron/gym/envs/break_and_make_env.py in v1.0.0) is stable.

Further Notes:

All 3D units are in LDraw LDUs. One LDU is approximately 0.4 mm, so the physical extents of these scenes can be quite large.

Disclaimer:

LTRON is based entirely on fan-made content with open licensing, and is not associated with the LEGO group in any way.

aaronwalsman/ltron