With this code you can:
- Use our pre-trained models to represent sentential contexts of target words and target words themselves with low-dimensional vector representations.
- Learn your own context2vec models with your choice of a learning corpus and hyperparameters.
Please cite the following paper if using the code:
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
Oren Melamud, Jacob Goldberger, Ido Dagan. CoNLL, 2016 [pdf].
- Python 3.6
- Chainer 4.2 (chainer)
- NLTK 3.0 (NLTK) - optional (only required for the AWE baseline and MSCC evaluation)
Note: Release 1.0 includes the original code that was used in the context2vec paper and has different dependencies (Python 2.7 and Chainer 1.7).
- Download the code
python setup.py install
- Download pre-trained context2vec models from [here]
- Unzip a model into MODEL_DIR
- Run:
python context2vec/eval/explore_context2vec.py MODEL_DIR/MODEL_NAME.params
>> this is a [] book
- This will embed the entire sentential context 'this is a __ book' and will output the top-10 target words whose embeddings are closest to that of the context.
- Use this as sample code to help you integrate context2vec into your own application.
- CORPUS_FILE needs to contain your learning corpus with one sentence per line and tokens separated by spaces.
- Run:
python context2vec/train/corpus_by_sent_length.py CORPUS_FILE [max-sentence-length]
- This will create a directory CORPUS_FILE.DIR that will contain your preprocessed learning corpus
- Run:
python context2vec//train/train_context2vec.py -i CORPUS_FILE.DIR -w WORD_EMBEDDINGS -m MODEL -c lstm --deep yes -t 3 --dropout 0.0 -u 300 -e 10 -p 0.75 -b 100 -g 0
- This will create WORD_EMBEDDINGS.targets file with your target word embeddings, a MODEL file, and a MODEL.params file. Put all of these in the same directory MODEL_DIR and you're done.
- See usage documentation for all run-time parameters.
NOTE:
- The current code lowercases all corpus words
- Use of a gpu and mini-batching is highly recommended to achieve good training speeds
Some users have noted that this configuration can cause exploding gradients
(see issue #6). One option
is to turn down the learning rate, by reducing the Adam optimizer's alpha from
0.001 to something lower, e.g. by specifying -a 0.0005. As an extra safety
measure, you can enable gradient clipping which could be set to 5 by using the
very scientific method of using the value everyone else seems to be using -gc 5.
- Download the train and test datasets from [here].
- Split the test files into dev and test if you wish to do development tuning.
- Download the pre-trained context2vec model for MSCC from [here];
- Or alternatively train your own model as follows:
- Run
context2vec/eval/mscc_text_tokenize.py INPUT_FILE OUTPUT_FILEfor every INPUT_FILE in the MSCC train set. - Concatenate all output files into one large learning corpus file.
- Train a model as explained above.
- Run
- Run:
python context2vec/eval/sentence_completion.py Holmes.machine_format.questions.txt Holmes.machine_format.answers.txt RESULTS_FILE MODEL_NAME.params
- Download the 'English lexical sample' train and test datasets from [here].
- Download the senseval scorer script(scorer2) from [here] and build it.
- Train your own context2vec model or use one of the pre-trained models provided.
- For development runs do:
python context2vec/eval/wsd/wsd_main.py EnglishLS.train EnglishLS.train RESULTS_FILE MODEL_NAME.params 1
scorer2 RESULTS_FILE EnglishLS.train.key EnglishLS.sensemap
- For test runs do:
python context2vec/eval/wsd/wsd_main.py EnglishLS.train EnglishLS.test RESULTS_FILE MODEL_NAME.params 1
scorer2 RESULTS_FILE EnglishLS.test.key EnglishLS.sensemap
The code for the lexical substitution evaluation is included in a separate repository [here].
- All words are converted to lowercase.
- Using gpu and/or mini-batches is not supported at test time.
Apache 2.0