A Unifying Principle for the Functional Organization of Visual Cortex

Code accompanying our paper on topographic deep neural networks.

🔗 Read the paper!

Authors:

Eshed Margalit 📧
Hyodong Lee
Dawn Finzi
James J. DiCarlo
Kalanit Grill-Spector*
Daniel L. K. Yamins*

*co-senior authors

Overview of the repository

This repository has the following components:

configs/:
- analysis_configs/: YAML files that establish the link between checkpoint paths and model names, used during analyses and figure generation
- config/: Training configuration files for use with the VISSL training framework
notebooks/: Jupyter notebooks (saved as Markdown files) that reproduce all figures and supplementary figures in the paper
scripts/: standalone scripts to be run from the command line. Contains a dedicated README explaining each script's purpose.
spacetorch/: installable Python package providing classes and methods for model training, evaluation, plotting, and figure creation
train.py: core model training script, modeled after this example in the VISSL repository.

Data and model weights

Using or training models

You can download model weights here, under tdann_data/tdann/checkpoints.

Take a look at the instructions for help using the existing models or training your own.

Datasets

There are a number of datasets used in this work:

Dataset	Description	Where to get it
ImageNet	Our models were trained on the initial release of ImageNet, i.e., before faces were detected and blurred. We haven't tested our models on the more recent releases.	https://www.image-net.org/
fLoc	The functional localizer (fLoc) images consist of 144 images each of 10 categories, plus a scrambled image category that we ignore in this work.	http://vpnl.stanford.edu/fLoc/
Sine Gratings	A set of sine gratings images at 8 orientations, 8 spatial frequencies, 5 spatial phases, and two types of colors: black/white and red/cyan.	OSF Download under `tdann_data/datasets/sine_grating_images_20190507`
Ecoset	A natural image dataset introduced in Mehrer et al., 2021	https://www.kietzmannlab.org/ecoset/

Other

The easiest way to get the code to run is to clone the folder structure on the machine the code was tested on. Download the tdann_data folder and place it in a location of your filesystem with plenty of free storage.

Next, set the environment variable ST_BASE_FS to point to that directory, either temporarily:

export ST_BASE_FS=/path/to/downloaded/dir

or permanently, by adding those lines to your shell config file (e.g., .bashrc/.zshrc).

The spacetorch package will look in that folder first to find checkpoints, unit positions, etc. If you have large datasets (like ImageNet) elsewhere on your filesystem, either edit paths.py as needed, or pass different arguments to the Dataset constructors.

Installation

To install dependencies, follow the instructions in INSTALL.md.

Citation

Please use the following citation for the paper or the code in this repository:

Margalit, E., Lee, H., Finzi, D., DiCarlo, J. J., Grill-Spector, K., & Yamins, D. L. K. (2024). A unifying framework for functional organization in early and higher ventral visual cortex. Neuron.

neuroailab/TDANN