DISCLAIMER: This is not an officially supported Google product.
This repository contains implementations on word embedding and quick-thought sentence embedding models.
A subset of wikipedia articles: enwiki9.
Preprocessing script is at:
data/wiki9.py
.
from data.wiki9 import write_wiki9_articles
write_wiki9_articles()
In total 140,000 articles. For experiments, 70,000 were used for training word embedding models.
Word embedding models are usually lookup tables that map a word to a low -dimension vector. Available models includes Word2Vec, FastText and Glove. Models were trained using public library gensim and glove-python.
Hyper-parameters were kept as in the original papers. To launch training for
Word2Vec for example, run: shell python train_word_embedding.py --model=w2v
model option can be w2v for Word2Vec, ft for FastText and glove for Glove.
A tensorflow implementation of Word2Vec is also available, and has an option for training Word2Vec with differential privacy. For training with DP, run:
CUDA_VISIBLE_DEVICES="0" python train_word_embedding_dp.py --dpsgd \
--noise_multiplier=0.1 --l2_norm_clip=0.25 --batch_size=512
Explanation for the options is detailed in tensorflow-privacy.
Trained models are evaluated using standard evaluation
questions.
Commandline for evaluation: python eval_word_embedding.py --model=w2v
A collection of books crawled using scripts from
https://github.com/soskek/bookcorpus. After preprocessing, there are more than
14,000 books and 30,000,000 sentences. Preprocessing script is at:
data/bookcorpus.py
.
from data.bookcorpus import preprocess_pipeline
preprocess_pipeline()
Sentence embedding models are usually neural networks that takes a sequence of words as input and output a low-dimension vector. We train QuickThought locally on half of all books. The model is trained by predicting the sentences before and after given a input sentence. Implementation is based on https://github.com/lajanugen/S2V with slight modification. To train QuickThought on books, run:
CUDA_VISIBLE_DEVICES="0" python train_quick_thought.py ----batch_size=500 \
--emb_dim=620 --encoder_dim=1200 --cell_type=LSTM --epochs=1
Evaluation of QuickThought is done by using the model as feature extractor for downstream sentence classification tasks. Currently supports evaluation on TREC and MSRP. Run evaluation with:
CUDA_VISIBLE_DEVICES="0" python eval_quick_thought.py --eval_data=trec