tested on Ubuntu 18.04 and 20.04
Environment setup
conda create --name py38 --file spec-file.txt python=3.8
conda activate py38
Install CLIP + opencv
pip install ftfy regex tqdm dqrobotics rospkg similaritymeasures
pip install git+https://github.com/openai/CLIP.git
pip install opencv-python
Download models
pip install gdown
gdown --folder https://drive.google.com/drive/folders/1HQNwHlQUOPMnbPE-3wKpIb6GMBz5eqDg?usp=sharing -O models/.
Download synthetic dataset
gdown --folder https://drive.google.com/drive/folders/1_bhWWa9upUWwUs7ln8jaWG_bYxtxuOCt?usp=sharing -O data/.
Download image dataset(optional)
gdown --folder https://drive.google.com/drive/folders/1Pok_sU_cK3RXZEpMfJb6SQIcCUfBjJhh?usp=sharing -O image_data/.
cd src
python interactive.py
How to use:
- press 'o' to load the original trajectory
- press 'm' to modify the trajectory using our model for the given input on top.
- press 't' to set a different interaction text.
- press 'u' to update the trajctory setting the modified traj as the original one
intructions for additional keyboard commands are shown in script output.
IMPORTANT: make sure that conda isn't initialized in your .bashrc file, otherwise, you might face conflicts between the python versions
For realtime object detection:
git clone https://github.com/arthurfenderbucker/realsense_3d_detector.git
terminal 1
roscore
terminal 2
roscd latte/src
python interactive.py --ros true
install coppelia simulator
https://www.coppeliarobotics.com/helpFiles/en/ros1Tutorial.htm
add export COPPELIASIM_ROOT_DIR=~/path/to/coppeliaSim/folde
to your ~/.bashrc
cd <ros_workspace>/src
git clone https://github.com/CoppeliaRobotics/ros_bubble_rob
git clone --recursive https://github.com/CoppeliaRobotics/simExtROS.git sim_ros_interface
cd <ros_workspace>
catkin config -DPYTHON_EXECUTABLE=$CONDA_PREFIX/bin/python -DPYTHON_INCLUDE_DIR=$CONDA_PREFIX/include/python3.8 -DPYTHON_LIBRARY=$CONDA_PREFIX/lib/libpython3.8.so -DSETUPTOOLS_DEB_LAYOUT=OFF
catkin config --install
catkin build
overview of the project model_overview.ipynb
model variations and ablasion studies Results.ipynb
user study interface user_study.py
generate syntetic dataset src/data_generator_script.py