Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction

This repository contains the code of our paper:
Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction
Christoph Alt, Marc Hübner, Leonhard Hennig

Our code depends on huggingface's PyTorch reimplementation of the OpenAI GPT, and AllenNLP - so thanks to them.

The code is tested with:

Python 3.6.6
PyTorch 1.0.1
AllenNLP 0.7.1

Installation

First, clone the repository to your machine and install the requirements with the following command:

pip install -r requirements.txt

Second, download the OpenAI GPT archive (containing all model related files):

wget --content-disposition https://cloud.dfki.de/owncloud/index.php/s/kKdpoaGikWnL4tn/download

Prepare the data

We evaluate our model on the NYT dataset and use the version provided by OpenNRE.

Follow the OpenNRE instructions for creating the NYT dataset in JSON format:

download the nyt.tar file.
extract the archive with: tar -xvf nyt.tar
create the protobuf files: protoc --proto_path=. --python_out=. Document.proto
convert the protobuf files to json: python protobuf2json.py .
move train.json and test.json to data/open_nre_nyt/

Training

E.g. for training on the NYT dataset, run the following command:

CUDA_VISIBLE_DEVICES=0 allennlp train \
    experiments/configs/model_paper.json \
    -s <MODEL AND METRICS DIR> \
    --include-package tre

Evaluation

CUDA_VISIBLE_DEVICES=0 python ./experiments/utils/pr_curve_and_predictions.py \
    <MODEL AND METRICS DIR> \
    ./data/open_nre_nyt/test.json \
    --output-dir <RESULTS DIR> \
    --archive-filename <MODEL ARCHIVE FILENAME>

Trained Models

The model(s) we trained on NYT to produce our paper results can be found here:

Dataset	Masking Mode	AUC	Download
NYT	None	0.422	Link

Download and extract model files

Download the archive corresponding to the model you want to evaluate (links in the table above).

wget --content-disposition <DOWNLOAD URL>

Run evaluation

For example, to evaluate the NYT model used in the paper, run the following command:

CUDA_VISIBLE_DEVICES=0 python ./experiments/utils/pr_curve_and_predictions.py \
    <DIR CONTAINING THE MODEL ARCHIVE> \
    ./data/open_nre_nyt/test.json \
    --output-dir ./results/ \
    --archive-filename model_lm05_wu2_do2_bs16_att.tar.gz

Citations

If you use our code in your research or find our repository useful, please consider citing our work.

@inproceedings{alt-etal-2019-fine,
    title = "Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction",
    author = {Alt, Christoph  and
      H{\"u}bner, Marc  and
      Hennig, Leonhard},
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/P19-1134",
    pages = "1388--1398",
}

License

DISTRE is released under the Apache 2.0 license. See LICENSE for additional details.

charosen/DISTRE