VisualTactile: A Python repository from NathanWu7

Dependencies

Some dependencies can be installed by

pip install -r ./requirements.txt

Isaac Gym

Our framework is implemented on Isaac Gym simulator, the version we used is Preview Release 4. You may encounter errors in installing packages, most solutions can be found in the official docs.

Pointnet2

Install pointnet++ manually.

cd {the dir for packages}
git clone --recursive https://github.com/erikwijmans/Pointnet2_PyTorch
cd Pointnet2_PyTorch
# [IMPORTANT] comment these two lines of code:
#   https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/pointnet2_ops_lib/pointnet2_ops/_ext-src/src/sampling_gpu.cu#L100-L101
pip install -r requirements.txt
pip install -e .

Finally, run the following to install other packages.

# make sure you are at the repository root directory
pip install -r requirements.txt

Assets

Google Drive Link: https://drive.google.com/file/d/16qATmRousv1vgVg3yQnahmRkPLoQ8lzG/view?usp=drive_link

Training

Tensorboard logdir :/run

1. RL

For env modify this file : cfg/task/xxxx.yaml
numEnvs : 512+

obs_type: ["oracle"]

For algo modify this file: cfg/train/sac/sac_xxxx.yaml
load_iter: when rl model saved

python3 train.py --task xxxx --algo sac --headless

For testing:

python3 train.py --task xxxx --algo sac --headless --test

2. VTA

For env modify this file : cfg/task/xxxx.yaml
numEnvs : 16+
obs_type: ["oracle","pointcloud","tactile"]

For algo modify this file: cfg/train/vta/vta_xxx.yaml

rl_algo: "sac" --> Choose algorithm
rl_iter: 10000 --> When RL model saved (iter)
max_iterations: 10000

python3 train.py --task xxxx --algo vta --headless

VTA test cannot run under headless model

3. VTP

For algo modify this file: cfg/train/vtp/vtp_xxx.yaml
sample_batch_size: 32 + (sample from replay_size * numEnvs)
replay_size: 300 + (total data: replay_size * numEnvs, each step update -> numEnvs)
lr: 0.001 --> learning rate
Other config files are the same as VTA

python3 train.py --task xxxx --algo vtp --headless