i-machine-think/diagNNose

diagNNose is a Python library that facilitates a broad set of tools for analysing hidden activations of neural models.

diagNNose ·

Paper: https://arxiv.org/abs/2011.06819

Demo:

Documentation: https://diagnnose.readthedocs.io

This library contains a set of modules that can be used to analyse the activations of neural networks, with a focus on NLP architectures such as LSTMs and Transformers. In particular, it contains functionality for :

• Extracting activations from different types of (language) models and providing quick access to these stored activations.
• Training diagnostic classifiers (Hupkes et al., 2018) on extracted activations.
• Training control tasks (Hewitt & Liang, 2019) parallel to these diagnostic classifiers.
• Performing model-agnostic feature attributions (Murdoch et al., 2018) on a model.
• Running a broad linguistic suite of targeted syntactic evaluations on a language model.

🎉 diagNNose has been presented at BlackboxNLP 2020! The paper can be found here.

Documentation can be found at diagnnose.readthedocs.io.

Our library is officially registered with pip and can be installed by running pip install diagnnose. The preferred version of Python is ≥3.6. The required packages are stated in requirements.txt.

Quick Tour

The workflow of diagNNose is divided into several building blocks, that can be combined for various experiments.

We provide a few examples that demonstrate the library. An interactive and more extensive interface for these scripts is also provided in the form of a notebook:

Activation Extraction

The activations of a model can be extracted using an Extractor that takes care of batching and selecting activations of interest.

Fine-grained activation selection is possible by defining a selection_func, that selects an activation based on the current sentence index and corpus item.

from torchtext.data import Example

from diagnnose.config import create_config_dict
from diagnnose.corpus import Corpus
from diagnnose.extract import Extractor
from diagnnose.models import LanguageModel, import_model
from diagnnose.tokenizer.create import create_tokenizer

if __name__ == "__main__":
    config_dict = create_config_dict()

    tokenizer = create_tokenizer(**config_dict["tokenizer"])
    corpus: Corpus = Corpus.create(tokenizer=tokenizer, **config_dict["corpus"])
    model: LanguageModel = import_model(**config_dict["model"])

    def selection_func(w_idx: int, item: Example) -> bool:
        return w_idx == item.extraction_idx

    extractor = Extractor(
        model, corpus, selection_func=selection_func, **config_dict["extract"]
    )
    activation_reader = extractor.extract()

Research using diagNNose

• Jumelet, Zuidema & Hupkes (2019): Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment
• Weber, Jumelet, Bruni & Hupkes (2021): Language Modeling as a Multi-Task Problem

Citing

If you intend on using diagNNose for your research, please cite us as follows. Feel free to reach out, we'd love to help!

@inproceedings{jumelet-2020-diagnnose,
    title = "diag{NN}ose: A Library for Neural Activation Analysis",
    author = "Jumelet, Jaap",
    booktitle = "Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.blackboxnlp-1.32",
    pages = "342--350",
}