Exploring artificial neural networks with psychophysical experiments.
The osculari
package provides an easy interface for different techniques to explore and interpret
the internal presentation of deep neural networks.
- Supporting the following pretrained models:
- All classification and segmentation networks from PyTorch's official website.
- All OpenAI CLIP language-vision models.
- All Taskonomy networks.
- Managing convolution and transformer architectures.
- Allowing to readout the network at any given depth.
- Training a linear classifier on top of the extract features from any network/layer.
- Experimenting with 2AFC and 4AFC paradigms.
At a granular level, Kornia is a library that consists of the following components:
Module | Description |
---|---|
osculari | Open source library to explore and interpret pretrained deep neural networks. |
osculari.datasets | A module to create datasets and dataloaders to train and test linear probes. |
osculari.models | A module to readout pretrained networks and add linear layers on top of them. |
osculari.paradigms | A module to implement psychophysical paradigms to experiment with deep networks. |
pip install osculari
Alternative installation options
pip install -e .
pip install git+https://github.com/ArashAkbarinia/osculari
Please check the example page of our documentation with many notebooks that can also be executed on Google Colab.
Quick start
Let's create a linear classifier on top of the extracted features from a pretrained network to
perform a binary classification task (i.e., 2AFC – two-alternative-force-choice). This is easily
achieved by calling the paradigm_2afc_merge_concatenate
from the osculari.models
module.
architecture = 'resnet50' # network's architecture
weights = 'resnet50' # the pretrained weights
img_size = 224 # network's input size
layer = 'block0' # the readout layer
readout_kwargs = {
'architecture': architecture,
'weights': weights,
'layers': layer,
'img_size': img_size,
}
net_2afc = osculari.models.paradigm_2afc_merge_concatenate(**readout_kwargs)
The osculari.datasets
module provides datasets that are generated randomly on the fly with
flexible properties that can be dynamically changed based on the experiment of interest.
For instance, by passing a appearance_fun
to the ShapeAppearanceDataset
class, we can
dynamically merge foreground masks with background images to generate stimuli of interest.
def appearance_fun(foregrounds, backgrounds):
# implementing the required appearance (colour, texture, etc.) on foreground and merging
# to background.
return merged_imgs, ground_truth
num_samples = 1000 # the number of random samples generated in the dataset
num_imgs = net_2afc.input_nodes # the number of images in each sample
background = 128 # the background type
dataset = osculari.datasets.geometrical_shapes.ShapeAppearanceDataset(
num_samples, num_imgs, img_size, background, appearance_fun,
unique_bg=True, transform=net_2afc.preprocess_transform()
)
The osculari.paradigms
module implements a set of psychophysical paradigms. The train_linear_probe
function trains the network on a dataset following the paradigm passed to the function.
# experiment-dependent function to train on an epoch of data
epoch_fun = osculari.paradigms.forced_choice.epoch_loop
# calling the generic train_linear_probe function
training_log = osculari.paradigms.paradigm_utils.train_linear_probe(
net_2afc, dataset, epoch_fun, './osculari_test/'
)
The osculari.paradigms
module also implements a set of psychophysical experiments similar to the
experiments conducted with human participants. In this example, we use the staircase
function
to gradually measure the network's sensitivity.
# experiment-dependent function to test an epoch of data
test_epoch_fun = osculari.paradigms.forced_choice.test_dataset
# the test dataset implementing desired stimuli.
class TestDataset(TorchDataset):
def __getitem__(self, idx):
return stimuli
test_log = osculari.paradigms.staircase(
net_2afc, test_epoch_fun, TestDataset(), low_val=0, high_val=1
)
We welcome all contributions to the project that extend or improve code and/or documentation! Please read the CONTRIBUTING page and follow the Code of Conduct.