Tested on python3.9, ubuntu 18.04, cuda 11.3.
Tested LIV and DINOv2. DINOv2 doesn't need additional dependencies. For LIV, after installation, run the following code to fix torch
# had to downgrade torch for my system that has cuda 11.3
pip uninstall torch torchaudio torchvision
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
# install chardet if needed
pip install chardet
I have updated requirements.txt
to avoid reinstalling torch after installing LIV.
I have added scripts/decompose_video.py
.
- Follow the instruction for installing
mujuco-py
and install the following apt packages if using Ubuntu:
sudo apt install -y libosmesa6-dev libgl1-mesa-glx libglfw3 patchelf
- create conda env with Python==3.9
conda create -n uvd python==3.9 -y && conda activate uvd
- Install any/all standalone visual foundation models from their repos separately before setup UVD, in case dependency conflicts, e.g.:
VIP
git clone https://github.com/facebookresearch/vip.git
cd vip && pip install -e .
python -c "from vip import load_vip; vip = load_vip()"
R3M
git clone https://github.com/facebookresearch/r3m.git
cd r3m && pip install -e .
python -c "from r3m import load_r3m; r3m = load_r3m('resnet50')"
LIV (& CLIP)
git clone https://github.com/penn-pal-lab/LIV.git
cd LIV && pip install -e . && cd liv/models/clip && pip install -e .
python -c "from liv import load_liv; liv = load_liv()"
DINOv2 and ResNet pretrained with ImageNet-1k are directly loaded via torch hub and torchvision.
- Under this UVD repo directory, install other dependencies.
pip install -e .
We provide a simple API for decompose RGB videos:
import torch
import uvd
# (N sub-goals, *video frame shape)
subgoals = uvd.get_uvd_subgoals(
"/PATH/TO/VIDEO.*", # video filename or (L, *video frame shape) video numpy array
preprocessor_name="vip", # Literal["vip", "r3m", "liv", "clip", "vc1", "dinov2"]
device="cuda" if torch.cuda.is_available() else "cpu", # device for loading frozen preprocessor
return_indices=False, # True if only want the list of subgoal timesteps
)
or run
python demo.py
to host a Gradio demo locally with different choices of visual representations.
We post-processed the data released from original Relay-Policy-Learning that keeps the successful trajectories only and adapt the control and observations used in our paper by:
python datasets/data_gen.py raw_data_path=/PATH/TO/RAW_DATA
Also consider to force set Builder = LinuxCPUExtensionBuilder
to Builder = LinuxGPUExtensionBuilder
in PATH/TO/CONDA/envs/uvd/lib/python3.9/site-packages/mujoco_py/builder.py
to enable (multi-)GPU acceleration.
Since UVD's goal is to be an off-the-shelf method applying to any existing policy learning frameworks and models, across BC and RL, we provide minimal scripts for benchmarking the runtime showing negligible runtime under ./scripts
directory:
python scripts/benchmark_decomp.py /PATH/TO/VIDEO
and passing --preprocessor_name
with other preprocessors (default vip
) and --n
for the number of repeated iterations (default 100
).
For inference or rollouts, we benchmark the runtime by
python scripts/benchmark_inference.py
and passing --policy
for using MLP or causal GPT policy; --preprocessor_name
with other preprocessors (default vip
); --use_uvd
as boolean arg for whether using UVD or no decomposition (i.e. final goal conditioned); and --n
for the number of repeated iterations (default 100
). The default episode horizon is set to 300. We found that running in the terminal would be almost 2s slower every episode than directly running with python IDE (e.g. PyCharm, under the script directory and run as script instead of module), but the general trend that including UVD introduces negligible extra runtime still holds true.
If you find this project useful in your research, please consider citing:
@misc{zhang2023universal,
title = {Universal Visual Decomposer: Long-Horizon Manipulation Made Easy},
author = {Zichen Zhang and Yunshuang Li and Osbert Bastani and Abhishek Gupta and Dinesh Jayaraman and Yecheng Jason Ma and Luca Weihs},
title = {Universal Visual Decomposer: Long-Horizon Manipulation Made Easy},
year = {2023},
eprint = {arXiv:2310.08581},
}