Candidate Models for Brain-Score: Which Artificial Neural Network is most Brain-Like?

Candidate models to evaluate on brain measurements, i.e. neural and behavioral recordings. Brain recordings are packaged in Brain-Score.

Quick setup

pip install "candidate_models @ git+https://github.com/brain-score/candidate_models"

The above command will not install ML frameworks such as Pytorch, TensorFlow or Keras.

You can install them yourself or using the following commands (in a conda environment):

Pytorch: conda install pytorch torchvision -c pytorch
Keras: conda install keras
TensorFlow: conda install tensorflow. To use the predefined TensorFlow models, you will have to install the TF-slim library. See here for quick instructions.

Usage

PYTHONPATH=. python candidate_models --model alexnet

During first-time use, ImageNet validation images (9.8 GB) will be downloaded, so give it a couple of minutes.

See the examples for more elaborate examples.

Environment variables

Environment variables are prefixed with CM_ for this framework. Environment variables from brain-score and model-tools might also be useful.

Variable	Description
CM_HOME	path to framework root
CM_TSLIM_WEIGHTS_DIR	path to stored weights for TensorFlow/research/slim models
MT_IMAGENET_PATH	path to ImageNet file containing the validation image set
RESULTCACHING_HOME	directory to cache results (benchmark ceilings) in, `~/.result_caching` by default (see https://github.com/mschrimpf/result_caching)

Installing the TF-slim image models library

TensorFlow-slim does unfortunately not provide an actual pip-installable library here, instead we have to download the code and make it available.

git clone https://github.com/qbilius/models/ tf-models
export PYTHONPATH="$PYTHONPATH:$(pwd)/tf-models/research/slim"
# verify
python -c "from nets import cifarnet; mynet = cifarnet.cifarnet"

Alternatively, you can also move/symlink these packages to your site-packages.

Troubleshooting

Could not find a version that satisfies the requirement brain-score

pip has trouble when dependency links are private repositories (as is the case now for brain-score). To circumvent, install brain-score by hand before installing candidate_models: pip install --process-dependency-links git+https://github.com/dicarlolab/brain-score.

Could not find a version that satisfies the requirement tensorflow

TensorFlow doesn't always catch up with newer Python versions. For instance, if you have Python 3.7 (check with python -V), TensorFlow might only work up to Python 3.6. If you're using conda, it usually installs the very newest version of Python. To fix, downgrade python: conda install python=3.6.

Failed to build pytorch

Some versions of PyTorch cannot be installed via pip (e.g. CPU). Instead, you need to build pytorch from their provided wheel. Check the website for installation instructions, right now they are (e.g. for Linux, Python 3.6, no CUDA): pip install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-linux_x86_64.whl && pip install torchvision. Or just use conda, e.g., for CPU: conda install pytorch-cpu torchvision-cpu -c pytorch

No module named `nets` / `preprocessing`

You probably haven't installed TensorFlow/research/slim. Follow the instructions [here](https://github.com/tensorflow/models/tree/master/research/slim#Install).

ImportError: cannot import name '_obtain_input_shape'

keras_squeezenet unfortunately does not run with keras > 2.2.0. To fix, pip install keras==2.2.0.

tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable

If this happened when running a keras model, your tensorflow and keras versions are probably incompatible. See the setup.py for which versions are supported.

Restoring from checkpoint failed. (...) Assign requires shapes of both tensors to match.

Most likely your passed image_size does not match up with the image size the model expects (e.g. inception_v{3,4} expect 299 insead of 224). Either let the framework infer what image_size the model needs (run without --image_size) or set the correct image_size yourself.

MobileNet weight loading failed.

Error message e.g. Assign requires shapes of both tensors to match. lhs shape= [1,1,240,960] rhs shape= [1,1,240,1280].

There is an error in the MobileNet implementation which causes the multiplier to not be applied properly: the number of channels sometimes go beyond what they ought to be (e.g. for the last layer). The line in question needs to be prefixed with a conditional:

if i != len(conv_defs['spec']) - 1 or multiplier >= 1:
    opdef.multiplier_func(params, multiplier)