A PyTorch implementation using BERT from Google to solve the task of NLI. This repo is the code used to obtain a Masters in Computer Science at NTNU.
To achieve this, the PyTorch implementation of BERT has been used. This repo is not officially maintaned by Google, but the official Tensorflow repo links to this one as a good PyTorch implementation which achieves the same results as the original implementation.
Install virtualenv and create a virtual environment by running
pip3 install virtualenv
mkdir venv
virtualenv --python=/usr/bin/python3.6 venv/This is not necessary on the remote server, as a python environment is set up for each slurm job.
As the system requires quite heavy computational power, the experiments have been run on IDUN. To login on IDUN, run
ssh -l stianste idun-login3.hpc.ntnu.no
where stianste is the username, and then
sbatch run_job.slurm
to run the job contained in run_job.slurm. To watch the job run
watch -n 1 squeue -u stianste
In order to run the TOEFL11 example on the remote host, you need to transfer the exisiting
directory over ssh. Assuming that the data is avaliable in the data folder, and
depending on where on the server you want to put it:
scp -r data/NLI-shared-task-2017 stianste@idun-login3.hpc.ntnu.no:/lustre1/work/stianste/BERT-NLI/ dataAfter that, one can run BERT on the TOEFL11-data set by running sbatch slurm_jobs/toefl.slurm.
The output of the job will be located in outputs/toefl.out. In order to watch
the output while the job is running, run tail -f outputs/toefl.out.
The same goes for slurm_jobs/reddit.slurm, which runs BERT using cross-validation
over the Reddit-L2 data set.
Most of the code neccessary to run BERT is located in the run_BERT_NLI.py file.
To see an example of what arguments this script requires and how to run it,
see the toefl.slurm script. Furthermore, the data_processors.py file
is used to read in test and training data. As long as the custom DataProcessor
class implements the basic methods required, it can be loaded in run_BERT_NLI.py
and used for training and testing BERT.
The different ensembles can be found in simple_ensemble.py – which trains the ensemble
on the TOEFL11 data set – and the reddit_ensemble.py script which trains an ensemble
on the Reddit-L2 data. If BERT is to be used in combination with the ensemble,
the test and training outputs of BERT must be located in the same folder
as the predictions of the base classifiers.