choiszt/OctoGibson

Octopus: Embodied Vision-Language Programmer from Environmental Feedback
Jingkang Yang^*,1  Yuhao Dong^*,2,5  Shuai Liu^*,3,5  Bo Li^*,1 
Ziyue Wang^†,1  Chencheng Jiang^†,4  Haoran Tan^†,3  Jiamu Kang^†,2 
Yuanhan Zhang¹  Kaiyang Zhou¹  Ziwei Liu^1,5,✉

¹S-Lab, Nanyang Technological University  ²Tsinghua University 
³Beijing University of Posts and Telecommunications  
⁴Xi'an Jiaotong University  ⁵Shanghai AI Laboratory 
^* Equal Contribution  ^† Equal Engineering Contribution  ^✉ Corresponding Author

---

Project Page | Octopus Paper

🐙 Introducing Octopus

Octopus is a novel VLM designed to proficiently decipher an agent’s vision and textual task objectives and to formulate intricate action sequences and generate executable code.

This repository provides:

• Training data collection pipeline in octogibson environment,
• Evaluation pipeline in octogibson environment,
• Evaluation pipeline in octogta environment,
• Training pipeline of the octopus model.

Contact: Leave issue or contact jingkang001@e.ntu.edu.sg and dongyh20@mails.tsinghua.edu.cn. We are on call to respond.

🦾 Updates

[2023-10]

1. 🤗 Introducing Project Octopus' homepage: https://choiszt.github.io/Octopus.
2. 🤗 Check our paper introducing Octopus in details.

🏁 Get Started

1. Training Data Collection: For data collection from octogibson environment, you need to set up two conda environments: omnigibson and gpt4. The omnigibson environment has an agent to act following the instruction from gpt4 environment. Please checkout here for detailed information.
2. Evaluation in OctoGibson: We provide the pipeline that the simulator sends messages to the Octopus server and gets responses to control the agent.
3. Evaluation in OctoGTA: We provide instructions, code, and MOD so that the Octopus can complete tasks in the GTA environment. Please checkout here for detailed information.
4. Octopus Training: We provide code for training Octopus. Please checkout here for detailed information.

📑 Citation

If you found this repository useful, please consider citing:

@article{yang2023octopus,
    author = {Jingkang Yang and Yuhao Dong and Shuai Liu and Bo Li and Ziyue Wang and Chencheng Jiang and Haoran Tan and Jiamu Kang and Yuanhan Zhang and Kaiyang Zhou and Ziwei Liu},
    title = {Octopus: Embodied Vision-Language Programmer from Environmental Feedback},
    year = {2023},
    license = {\url{http://arxiv.org/licenses/nonexclusive-distrib/1.0/}}
}

👨‍🏫 Acknowledgements

We thank the OmniGibson team for their help and great contribution to the open-source community.