Designed for deep net feature extraction, dimensionality reduction, and benchmarking, this repo contains a number of convenience functions for loading and instrumentalizing a variety of (PyTorch) models. Models available include those from:
- the PyTorch-Image-Models (Timm) library
- the Torchvision model zoo
- the Taskonomy (visual_priors) project
- the VISSL (self-supervised) model zoo
- the Detectron2 model zoo
- ISL's MiDas models, FaceBook's DINO models...
Check out these repos that benchmark these models on human fMRI and mouse optical physiology data.
A tutorial that demonstrates the main functionality of this pipeline in both behavior and brains may be found here.
This repository is a work in progress; please feel free to file any issues you find.
If you find this repository useful, please consider citing the work that fueled its most recent development:
@article{conwell2023pressures,
title={What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines},
author={Conwell, Colin and Prince, Jacob S and Kay, Kendrick N and Alvarez, George A and Konkle, Talia},
journal={bioRxiv},
year={2023}
}(Also remember to cite any of the specific models you use by referring to their original sources linked in the model_typology.csv file).
- Squeezing your deep nets for science!
Recently, our team has been working on a new, highly-accelerated version of this codebase called Deepjuice -- effectively, a bottom-up reimplementation of all DeepDive functionalities that allows for end-to-end benchmarking (feature extraction, SRP, PCA, CKA, RSA, and regression) without ever removing data from the GPU.
DeepJuice is currently in private beta, but if you're interested in trying out, please feel free to contact me (Colin Conwell) by email: conwell[at]g[dot]harvard[dot]edu