MineCLIP is a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Guided by MineCLIP reward, our trained MineAgents are able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. MineCLIP is trained using MineDojo video dataset.
Specifically, MineCLIP model is a contrastive video-language model that learns to correlate video snippets and natural language descriptions. It is multi-task by design because it is trained on open-vocabulary and diverse English transcripts. With the trained MineCLIP model in the hand, we train language-conditioned MineAgents that take as input raw pixels and predict discrete control.
In this repo, we provide model code for MineCLIP, MineAgent, and env wrappers for dense reward shaping and sample envs used in our paper. This codebase is under MIT License.
MineCLIP requires Python ≥ 3.9. We have tested on Ubuntu 20.04. Installing MineCLIP codebase is as simple as:
pip install git+https://github.com/MineDojo/MineCLIPWe provide implementation for two MineCLIP variants, mineclip_attn and mineclip_avg. Here are demo scripts you can run both variants.
python3 main/mineclip/run.py variant=attn|avgChoose one variant from attn and avg. If everything goes well, you should see Inference successful printed out.
Pretrained weights are provided for attn and avg. You can run demos below to load pretrained weights.
python3 main/mineclip/load_ckpt.py variant=attn|avg ckpt.path=PATH/TO/DOWNLOADED/CKPTChoose one variant from attn and avg and specify path for downloaded weights. If everything goes well, you should see Successfully loaded ckpt printed out.
We provide implementation for MineAgent model. To run a MineAgent that takes a single-step observation and outputs a single-step action, run
python3 main/mineagent/run_single.pyIf everything goes well, you should see Inference successful printed out.
To run a MineAgent demo with an environment in the loop, execute
python3 main/mineagent/run_env_in_the_loop.pyIf everything goes well, you should see a Minecraft client pops up and an agent fight against a spider (though currently it is a random policy).
We provide env wrappers for dense reward shaping used in our paper. Specifically, we provide dense reward shaping for tasks in groups of animal-zoo and mob-combat. Corresponding wrappers can be found here and here.
You can also find two sample env implementations HuntCowDenseRewardEnv and CombatSpiderDenseRewardEnv. They correspond to tasks "Hunt Cow" and "Combat Spider" in the paper.
Our paper is posted on Arxiv. If you find our work useful, please consider citing us!
@article{fan2022minedojo,
title = {MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge},
author = {Linxi Fan and Guanzhi Wang and Yunfan Jiang and Ajay Mandlekar and Yuncong Yang and Haoyi Zhu and Andrew Tang and De-An Huang and Yuke Zhu and Anima Anandkumar},
year = {2022},
journal = {arXiv preprint arXiv: Arxiv-2206.08853}
}