Your head is there to move you around

This repo contains information necessary to run the code in "Your head is there to move you around". Preprint here: https://www.biorxiv.org/content/10.1101/2021.07.09.451701v2.abstract

Prelims

Python dependencies

Python 3.8. In a new conda environment, pip install -r requirements.txt. You may use the Dockerfile if you prefer.

Electrophysiology datasets

Data licenses do not allow us to redistribute derived datasets, hence the end-user will need to download datasets and preprocess them manually as shown below. For review purposes, however, you may download preprocessed datasets from Google Cloud as follows:

pip install gsutil
mkdir zips
gsutil rsync -r gs://vpl-bucket/data_derived zips
unzip 'zips/*.zip' -d data_derived

Download and preprocess

You can download the datasets crcns-mt1, crcns-mt2, crcns-pvc1 and crcns-pvc4 from crcns.org. You will need to register to do so. Drop them in a raw_data folder in folders that have the corresponding names (e.g. crcns-mt1, crncs-mt2, etc.). In paths.py, edit RAW_DATA and DERIVED_DATA to point to raw_data and derived_data folders. To preprocess the datasets:

Run scripts/generate_crncs_mt1_derived.m in Matlab. You will need the auxillary .m files that come with the dataset. Edit the path in the m file. Then run scripts/generate_crcns_mt1_derived.py.
Copy the contents of crcns-mt2 verbatim into the data_derived folder
Run scripts/derive_dataset.py --dataset pvc1
Run scripts/derive_dataset.py --dataset pvc4

The preprocessing scripts for the packlab-mst dataset (Mineault et al. 2011) are organized similar to the crcns-mt1 data; the scripts are generate_packlab_mst_derived.m and generate_packlab_mst_derived.py.

Airsim dataset

You can download the exact dataset used in the paper on gdrive, and drop it in data_derived. The code necessary to generate this dataset is in the airsim repo. It takes several days to run and requires some manual work.

Checkpoints

Download pretrained models from Google Drive and drop them in a checkpoints folder. We include CPC and DorsalNet checkpoints.

For the rest of the checkpoints, please download the following from SlowFast:

Kinetics/c2/SLOWFAST_8x8_R50
Kinetics/c2/I3D_8x8_R50

Edit paths.py CHECKPOINTS accordingly.

Train DorsalNet

You can train DorsalNet on Airsim data using:

python train_sim.py 
  --exp_name trainit 
  --submodel dorsalnet_untied 
  --dataset airsim_batch2 
  --batch_size 64 
  --learning_rate 3e-3 
  --softmax 
  --decoder center 
  --num_epochs 100

This takes about 6 hours.

Use pretrained DorsalNet

notebooks/Use DorsalNet.ipynb shows how to initialize and use DorsalNet with the checkpoint you downloaded.

Estimate prefered stimuli

Use notebooks/Show first layer filters.ipynb to show first layer filters. Use notebooks/Calculate standardized stimuli.ipynb to calculate responses to gratings, plaids, etc. Use scripts/dot_reverse_correlation.py to calculate prefered motion vector fields. Run notebooks/Show optimal stimuli.ipynb to assemble quiver plots and calculate prefered image sequences using gradient descent. See also here to visualize these optimal stimuli.

Align deep neural nets to brain data

Align deep neural nets to brain data using run_many.sh. This should take a few weeks to run. You will need a wandb.ai account to save the results remotely.

Compile brain data results

Use notebooks/Compare results_physiology.ipynb to assemble the brain data results into tables and plots.

Heading decoding

Use run_heading.sh to run the heading decoding model. Use notebooks/Compare results_heading.ipynb to assemble results into tables and plots.

kencan7749/your-head-is-there-to-move-you-around