This repository contains code used in LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action by Dhruv Shah, Błażej Osiński, Brian Ichter, and Sergey Levine.
In order to dive into the code, we suggest starting from the following:
jupyter_experiment.ipynb- notebook to run the LM-Nav pipeline for text queries on two different graphs. This includes running GPT3, CLIP, and our custom graph search algorithm.colab_experiment.ipynb- colab version of the above notebook. You can easily run it in your browser!ablation_text_to_landmark.ipynb- notebook with ablation experiments for the language processing part: comparing GPT3 to open-source alternatives and a simple NLP baseline.
The code was tested with python 3.7.13. It assumes access to GPU and CUDA 10.2 is installed.
To run locally, install the package:
pip install .
Then simply open jupyter_experiments.ipynb or ablation_text_to_landmark.ipynb in jupyter notebook.
For the LM-Nav pipeline notebooks, we added a cached version of the OpenAI API calls for the sample queries. If you also want to run the GPT-3 part of the pipeline, you need to provide OpenAI API key (see the docs out the API docs here), and pass it the OpenAI API, e.g.:
OPENAI_API_KEY=sk-[] jupyter notebook
Likewise, in order to re-run the ablation experiments with the open-source models you need to specify GooseAI API key.
If you find this work useful, please consider citing:
@misc{shah2022lmnav,
title={LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action},
author={Dhruv Shah and Blazej Osinski and Brian Ichter and Sergey Levine},
year={2022},
eprint={2207.04429},
archivePrefix={arXiv},
primaryClass={cs.RO}
}