CrossLingualELMo

Cross-Lingual Alignment of Contextual Word Embeddings

This repo will contain the code and models for the NAACL19 paper - Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

More pieces of the code will be released soon.

Updates:

A script to compute anchors for a Bert model is now available.
The Multilingual ELMo is now merged to the AllenNLP framework. You can install their repo from source or wait for their next release. Anchors for other models can be computed using the code here.
~~We are working on merging the Multilingual ELMo to the AllenNLP framework. Hopefully we will get to finish it soon.~~
~~The multi-lingual parser code is now available at this allennlp fork (requirements.txt file of this repo is updated accordingly). See more details in the Usage section below.~~

Aligned Multi Lingual Deep Contextual Word Embeddings

Embeddings

The following models were trained on Wikipedia. We provide the alignment of the first LSTM output of ELMo to English (more alignments will come soon)

Language	Model weights	Aligning matrix
English	weights.hdf5	best_mapping.pth
Spanish	weights.hdf5	best_mapping.pth
French	weights.hdf5	best_mapping.pth
Italian	weights.hdf5	best_mapping.pth
Portuguese	weights.hdf5	best_mapping.pth
Swedish	weights.hdf5	best_mapping.pth
German	weights.hdf5	best_mapping.pth

options file (for all models) - options.json

To download all the ELMo models in the table, use get_models.sh
To download all of the alignment matrices in the table, use get_alignments.sh.
Alternatively, If you are interested in applying it in an Allennlp model, you can just add the path to the configuration (check the examples in allen_configs)

Generating anchors

In order to generate your own anchors - use the gen_anchors.py script to generate your own anchors. You will need a trained ELMo model, text files with one sentence per line, and vocab file with token per line containing the tokens that you wish to calculate for. run gen_anchors.py -h for more details.

Usage

Generating aligned contextual embeddings

Given the output of a specific layer from ELMo (the contextual embeddings), run:

aligning  = torch.load(aligning_matrix_path)
aligned_embeddings = np.matmul(embeddings, aligning.transpose())

An example can be seen in demo.py.

Replicating the zero-shot cross-lingual dependency parsing results

Create an environment to install our fork of allennlp:

virtualenv -p /usr/bin/python3.6 allennlp_multilang

or, if you are using conda:

conda create -n allennlp_multilang python=3.6

Activate the environment and install:

source allennlp_multilang/bin/activate
pip install -r requirements.txt

Download the uni-dep-tb dataset (version 2) and follow the instructions to generate the English PTB data
Update the allen_configs/multilang_dependency_parser.jsonnet file with the path to dataset.
Train the model (the provided configuration is for 'es' as a target language):

allennlp train training_config/multilang_dependency_parser.jsonnet -s path_to_output_dir

Using in any model

The aligments can be used with the AllenNLP framework by simply using any model with ELMo embeddings and replacing the paths in the configuration with our provided models.

Each ELMo model was trained on Wikipedia of the relevant language. To align the models, you will need to add the following code to your model:

Load the alignment matrix in the __init__() function:

aligning_matrix_path = ... (pth file)
self.aligning_matrix = torch.FloatTensor(torch.load(aligning_matrix_path))
self.aligning = torch.nn.Linear(self.aligning_matrix[0], self.aligning_matrix[1], bias=False)
self.aligning.weight = torch.nn.Parameter(self.aligning_matrix, requires_grad=False)

Then, simply apply the alignment on the embedded tokens in the forward() pass:

embedded_text = self.aligning(embedded_text)

Note that our alignments were done on the second layer of ELMo but it can be learned and applied for each layer separately to preserve the original weighted sum of layers in the ELMo embedder. To fix the weighted sum across languages, use fixed parameters for scalar_mix (see our multi-lingual parser for an example).

Citation

If you find this repo useful, please cite our paper.

@InProceedings{Schuster2019,
    title = "Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing",
    author = "Schuster, Tal  and
      Ram, Ori  and
      Barzilay, Regina  and
      Globerson, Amir",
    booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/N19-1162",
    pages = "1599--1613"
}

so2jia/CrossLingualELMo