FudanVI/compositional-scene-representation-toolbox

A toolbox of compositional scene representation learning methods and benchmark datasets.

Compositional Scene Representation Toolbox

This is an accompanied toolbox for the survey article: Compositional Scene Representation Learning via Reconstruction: A Survey [Yuan et al., IEEE TPAMI 2023]. The toolbox contains code for synthesizing multiple datasets that could be used for benchmarking compositional scene representation learning methods, and collects the implementations of the following papers:

• Spatial Mixture Models with Learnable Deep Priors for Perceptual Grouping [Yuan et al., AAAI-19]
• Generative Modeling of Infinite Occluded Objects for Compositional Scene Representation [Yuan et al., ICML-19]
• Knowledge-Guided Object Discovery with Acquired Deep Impressions [Yuan et al., AAAI-21]
• Unsupervised Learning of Compositional Scene Representations from Multiple Unspecified Viewpoints [Yuan et al., AAAI-22]
• Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction. [Gao & Li, UAI-23]
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models [Eslami et al., NeurIPS-16]
• Neural Expectation Maximization [Greff et al., NeurIPS-17]
• Multi-Object Representation Learning with Iterative Variational Inference [Greff et al., ICML-19]
• MONet: Unsupervised Scene Decomposition and Representation [Burgess et al.]
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations [Engelcke et al., ICLR-20]
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition [Lin et al., ICLR-20]
• Object-Centric Learning with Slot Attention [Locatello et al., NeurIPS-20]
• Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views [Li et al., NeurIPS-20]
• Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations [Emami et al., ICML-21]
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement [Engelcke et al., NeurIPS-21]

The README.md file in each folder contains the instructions on how to run the code.

Submodules

Initialize submodules using the following command.

git submodule update --init --recursive

Create Benchmark Datasets

Change the current working directory to compositional-scene-representation-datasets and follow the instructions described in README.md to create benchmark datasets.

Evaluate Performance on Benchmark Datasets

AIR

Change the current working directory to air-unofficial/experiments_benchmark and run run.sh and run_nc.sh.

cd air-unofficial/experiments_benchmark
./run.sh
./run_nc.sh
cd ../..

Run air-unofficial/experiments_benchmark/evaluate.ipynb to evaluate the trained models.

N-EM

Change the current working directory to nem-unofficial/experiments_benchmark and run run.sh and run_nc.sh.

cd nem-unofficial/experiments_benchmark
./run.sh
./run_nc.sh
cd ../..

Run nem-unofficial/experiments_benchmark/evaluate.ipynb to evaluate the trained models.

IODINE

Change the current working directory to iodine-unofficial/experiments_benchmark and run run.sh and run_nc.sh.

cd iodine-unofficial/experiments_benchmark
./run.sh
./run_nc.sh
cd ../..

Run iodine-unofficial/experiments_benchmark/evaluate.ipynb to evaluate the trained models.

GMIOO

Change the current working directory to infinite-occluded-objects/experiments_benchmark and run run.sh and run_nc.sh.

cd infinite-occluded-objects/experiments_benchmark
./run.sh
./run_nc.sh
cd ../..

Run infinite-occluded-objects/experiments_benchmark/evaluate.ipynb to evaluate the trained models.

MONet

Change the current working directory to monet-unofficial/experiments_benchmark and run run.sh.

cd monet-unofficial/experiments_benchmark
./run.sh
cd ../..

Change the current working directory to monet-unofficial_nc/experiments_benchmark and run run.sh.

cd monet-unofficial_nc/experiments_benchmark
./run.sh
cd ../..

Run monet-unofficial/experiments_benchmark/evaluate.ipynb to evaluate the trained models.

SPACE

Change the current working directory to SPACE/src and run run.sh.

cd SPACE/src
./run.sh
cd ../..

Change the current working directory to SPACE_nc/src and run run.sh.

cd SPACE_nc/src
./run.sh
cd ../..

Run SPACE/evaluate.ipynb to evaluate the trained models.

Slot Attention

Change the current working directory to slot-attention-unofficial/experiments_benchmark and run run.sh and run_nc.sh.

cd slot-attention-unofficial/experiments_benchmark
./run.sh
./run_nc.sh
cd ../..

Run slot-attention-unofficial/experiments_benchmark/evaluate.ipynb to evaluate the trained models.

EfficientMORL

Change the current working directory to EfficientMORL and run run.sh and run_nc.sh.

cd EfficientMORL
./run.sh
./run_nc.sh
cd ..

Run EfficientMORL/evaluate.ipynb to evaluate the trained models.

GENESIS and GENESIS-V2

Change the current working directory to genesis and run run.sh.

cd genesis
./run.sh
cd ..

Change the current working directory to genesis_nc and run run.sh.

cd genesis_nc
./run.sh
cd ..

Run genesis/evaluate.ipynb to evaluate the trained models.