This repository hosts the probing framework used in our paper A Matter of Framing: The Impact of Linguistic Formalism on Probing Results (EMNLP 2020). We provide scripts to generate multi-formalism role-semantic data for English and German, as well as the full probing kit for further experimentation. If you use it, please cite:
@inproceedings{kuznetsov-gurevych-2020-matter,
title = "A matter of framing: {T}he impact of linguistic formalism on probing results",
author = "Kuznetsov, Ilia and
Gurevych, Iryna",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-main.13",
pages = "171--182",
}
Abstract: Deep pre-trained contextualized encoders like BERT demonstrate remarkable performance on a range of downstream tasks. A recent line of research in probing investigates the linguistic knowledge implicitly learned by these models during pre-training. While most work in probing operates on the task level, linguistic tasks are rarely uniform and can be represented in a variety of formalisms. Any linguistics-based probing study thereby inevitably commits to the formalism used to annotate the underlying data. Can the choice of formalism affect probing results? To investigate, we conduct an in-depth cross-formalism layer probing study in role semantics. We find linguistically meaningful differences in the encoding of semantic role- and proto-role information by BERT depending on the formalism and demonstrate that layer probing can detect subtle differences between the implementations of the same linguistic formalism. Our results suggest that linguistic formalism is an important dimension in probing studies, along with the commonly used cross-task and cross-lingual experimental settings.
Don't hesitate to send us an e-mail or report an issue, if something is broken or if you have further questions!
Contact person: Ilia Kuznetsov, kuznetsov@ukp.informatik.tu-darmstadt.de
https://www.ukp.tu-darmstadt.de/
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.
This repository provides the following:
- Scripts to generate cross-formalism data for PropBank, VerbNet and FrameNet role labels for English and German (data_preprocessing)
- A probing kit that can
- Generate edge-style probing tasks
- Run layer probing
- Analyze the layer probing results
- Auxiliary mapping files to merge SemLink with CoNLL-2009 ST data
- We use semi-automatically aligned cross-formalism data in our experiments. However, the kit can be used for any kind of edge probing!
- See source_data/demo.json for the edge probing task format. It's very simple and similar to the one used in jiant. We pre-tokenize the tasks into wordpieces to save time during training/testing.
- The probing.ExperimentManager class takes care of the neat project structure.
- Before you run your experiments, you specify a workspace. The script will generate the following folder structure:
- configs - AllenNLP config files for experiments are stored here
- datasets - pickled pre-processed datasets are stored here
- tasks - generated probing tasks are stored here
- out - models etc. are stored here
- After you run the experiments, you can analyze the results using the probing/Analytics.ipynb jupyter notebook
We use python 3.6. Create a fresh conda environment and install the dependencies:
- allennlp==0.9.0
- pandas
- seaborn
- anytree
- jupyter
- torch 1.2.0 (conda install pytorch==1.2.0 torchvision==0.4.0 -c pytorch)
If you want to re-run our experiments, you must prepare the data, see data_preprocessing. Demo.py contains the minimal running example, and emnlp2020.py is the full pipeline for the paper.