wanghia/awesome-sentence-embedding

A curated list of pretrained sentence and word embedding models

awesome-sentence-embedding

A curated list of pretrained sentence and word embedding models

Table of Contents

• About This Repo
• General Framework
• Word Embeddings
• OOV Handling
• Contextualized Word Embeddings
• Pooling Methods
• Encoders
• Evaluation
• Misc
• Vector Mapping
• Articles

About This Repo

• well there are some awesome-lists for word embeddings and sentence embeddings, but all of them are outdated and more importantly incomplete
• this repo will also be incomplete, but I'll try my best to find and include all the papers with pretrained models
• this is not a typical awesome list because it has tables but I guess it's ok and much better than just a huge list
• if you find any mistakes or find another paper or anything please send a pull request and help me to keep this list up to date
• enjoy!

General Framework

• Almost all the sentence embeddings work like this:
• Given some sort of word embeddings and an optional encoder (for example an LSTM) they obtain the contextualized word embeddings.
• Then they define some sort of pooling (it can be as simple as last pooling).
• Based on that they either use it directly for the supervised classification task (like infersent) or generate the target sequence (like skip-thought).
• So, in general, we have many sentence embeddings that you have never heard of, you can simply do mean-pooling over any word embedding and it's a sentence embedding!

Word Embeddings

• Note: don't worry about the language of the code, you can almost always (except for the subword models) just use the pretrained embedding table in the framework of your choice and ignore the training code

date	paper	citation count	training code	pretrained models
-	GloVe: Global Vectors for Word Representation	N/A	C	GloVe
-	Dependency-Based Word Embeddings	N/A	C++	word2vecf
-	Learning Word Meta-Embeddings	N/A	-	Meta-Emb(broken)
-	Dict2vec : Learning Word Embeddings using Lexical Dictionaries	N/A	C++	Dict2vec
-	WebVectors: A Toolkit for Building Web Interfaces for Vector Semantic Models	N/A	-	RusVectōrēs
-	SensEmbed: Learning Sense Embeddings for Word and Relational Similarity	N/A	-	SensEmbed
2013/01	Efficient Estimation of Word Representations in Vector Space	999+	C	Word2Vec
2014/12	Word Representations via Gaussian Embedding	133	Cython	-
2014/??	A Probabilistic Model for Learning Multi-Prototype Word Embeddings	89	DMTK	-
2015/06	From Paraphrase Database to Compositional Paraphrase Model and Back	0	Theano	PARAGRAM
2015/06	Non-distributional Word Vector Representations	42	Python	WordFeat
2015/06	Sparse Overcomplete Word Vector Representations	81	C++	-
2015/??	Joint Learning of Character and Word Embeddings	112	C	-
2015/??	Topical Word Embeddings	159	Cython
2016/02	Swivel: Improving Embeddings by Noticing What's Missing	47	TF	-
2016/03	Counter-fitting Word Vectors to Linguistic Constraints	130	Python	counter-fitting(broken)
2016/05	Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec	38	Chainer	-
2016/06	Siamese CBOW: Optimizing Word Embeddings for Sentence Representations	112	Theano	Siamese CBOW
2016/06	Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations	34	Go	lexvec
2016/07	Enriching Word Vectors with Subword Information	999+	C++	fastText
2016/08	Morphological Priors for Probabilistic Neural Word Embeddings	23	Theano	-
2016/11	A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks	198	C++	charNgram2vec
2016/12	ConceptNet 5.5: An Open Multilingual Graph of General Knowledge	188	Python	Numberbatch
2017/02	Offline bilingual word vectors, orthogonal transformations and the inverted softmax	171	Python	-
2017/04	Multimodal Word Distributions	30	TF	word2gm
2017/05	Poincaré Embeddings for Learning Hierarchical Representations	156	Pytorch	-
2017/06	Context encoders as a simple but powerful extension of word2vec	4	Python	-
2017/06	Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints	69	TF	Attract-Repel
2017/08	Learning Chinese Word Representations From Glyphs Of Characters	18	C	-
2017/08	Making Sense of Word Embeddings	24	Python	sensegram
2017/09	Hash Embeddings for Efficient Word Representations	7	Keras	-
2017/10	BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages	23	Gensim	BPEmb
2017/11	SPINE: SParse Interpretable Neural Embeddings	19	Pytorch	SPINE
2017/??	Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics	14	C	-
2017/??	Joint Embeddings of Chinese Words, Characters, and Fine-grained Subcharacter Components	26	C	-
2017/??	AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP	42	Gensim	AraVec
2018/04	Dynamic Meta-Embeddings for Improved Sentence Representations	19	Pytorch	DME/CDME
2018/04	Representation Tradeoffs for Hyperbolic Embeddings	38	Pytorch	h-MDS
2018/05	Analogical Reasoning on Chinese Morphological and Semantic Relations	38	-	ChineseWordVectors
2018/06	Probabilistic FastText for Multi-Sense Word Embeddings	13	C++	Probabilistic FastText
2018/09	Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks	1	TF	SynGCN
2018/09	FRAGE: Frequency-Agnostic Word Representation	30	Pytorch	-
2018/12	Wikipedia2Vec: An Optimized Tool for LearningEmbeddings of Words and Entities from Wikipedia	3	Cython	Wikipedia2Vec
2018/??	cw2vec: Learning Chinese Word Embeddings with Stroke n-gram Information	18	C++	-
2018/??	Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings	21	-	ChineseEmbedding
2019/02	VCWE: Visual Character-Enhanced Word Embeddings	0	Pytorch	VCWE
2019/05	Learning Cross-lingual Embeddings from Twitter via Distant Supervision	1	Text	-
2019/08	ViCo: Word Embeddings from Visual Co-occurrences	0	Pytorch	ViCo
2019/08	An Unsupervised Character-Aware Neural Approach to Word and Context Representation Learning	2	TF	-
2019/11	Spherical Text Embedding	N/A	C	-
2019/??	Unsupervised word embeddings capture latent knowledge from materials science literature	6	Gensim	-

OOV Handling

• Drop OOV words!
• One OOV vector(unk vector)
• Use subword models(ngram, bpe, char)
• ALaCarte: A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
• Mimick: Mimicking Word Embeddings using Subword RNNs
• CompactReconstruction: Subword-based Compact Reconstruction of Word Embeddings

Contextualized Word Embeddings

• Note: all the unofficial models can load the official pretrained models

date	paper	citation count	code	pretrained models
-	Language Models are Unsupervised Multitask Learners	N/A	TF Pytorch, TF2.0 Keras	GPT-2
2017/08	Learned in Translation: Contextualized Word Vectors	276	Pytorch Keras	CoVe
2018/01	Universal Language Model Fine-tuning for Text Classification	71	Pytorch	ULMFit(English, Zoo)
2018/02	Deep contextualized word representations	999+	Pytorch TF	ELMO(AllenNLP, TF-Hub)
2018/04	Efficient Contextualized Representation:Language Model Pruning for Sequence Labeling	11	Pytorch	LD-Net
2018/07	Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation	43	Pytorch	ELMo
2018/08	Direct Output Connection for a High-Rank Language Model	11	Pytorch	DOC
2018/10	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	999+	TF Keras Pytorch, TF2.0 MXNet PaddlePaddle TF Keras	BERT(BERT, ERNIE, KoBERT)
2018/??	Contextual String Embeddings for Sequence Labeling	109	Pytorch	Flair
2018/??	Improving Language Understanding by Generative Pre-Training	473	TF Keras Pytorch, TF2.0	GPT
2019/01	Cross-lingual Language Model Pretraining	109	Pytorch Pytorch, TF2.0	XLM
2019/01	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	127	TF Pytorch Pytorch, TF2.0	Transformer-XL
2019/01	BioBERT: pre-trained biomedical language representation model for biomedical text mining	70	TF	BioBERT
2019/01	Multi-Task Deep Neural Networks for Natural Language Understanding	79	Pytorch	MT-DNN
2019/02	Efficient Contextual Representation Learning Without Softmax Layer	1	Pytorch	-
2019/03	SciBERT: Pretrained Contextualized Embeddings for Scientific Text	27	Pytorch, TF	SciBERT
2019/04	Publicly Available Clinical BERT Embeddings	16	Text	clinicalBERT
2019/04	ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission	8	Pytorch	ClinicalBERT
2019/05	ERNIE: Enhanced Language Representation with Informative Entities	17	Pytorch	ERNIE
2019/05	Unified Language Model Pre-training for Natural Language Understanding and Generation	21	Pytorch	UniLM
2019/05	HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization	4		-
2019/06	XLNet: Generalized Autoregressive Pretraining for Language Understanding	112	TF Pytorch, TF2.0	XLNet
2019/06	Pre-Training with Whole Word Masking for Chinese BERT	3	Pytorch, TF	BERT-wwm
2019/07	ERNIE 2.0: A Continual Pre-training Framework for Language Understanding	1	PaddlePaddle	ERNIE 2.0
2019/07	SpanBERT: Improving Pre-training by Representing and Predicting Spans	12	Pytorch	SpanBERT
2019/07	RoBERTa: A Robustly Optimized BERT Pretraining Approach	67	Pytorch Pytorch, TF2.0	RoBERTa
2019/09	MultiFiT: Efficient Multi-lingual Language Model Fine-tuning	0	Pytorch	-
2019/09	Extreme Language Model Compression with Optimal Subwords and Shared Projections	0		-
2019/09	UNITER: Learning UNiversal Image-TExt Representations	0		-
2019/09	MULE: Multimodal Universal Language Embedding	0		-
2019/09	Knowledge Enhanced Contextual Word Representations	0		-
2019/09	K-BERT: Enabling Language Representation with Knowledge Graph	0		-
2019/09	Subword ELMo	0	Pytorch	-
2019/09	Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks	1		-
2019/09	TinyBERT: Distilling BERT for Natural Language Understanding	1		-
2019/09	ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	3	TF	-
2019/10	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	0	TF	T5
2019/10	DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	4	Pytorch, TF2.0	DistilBERT
2019/11	ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations	N/A	Pytorch	-

Pooling Methods

• {Last, Mean, Max}-Pooling
• Special Token Pooling (like BERT and OpenAI's Transformer)
• SIF: A Simple but Tough-to-Beat Baseline for Sentence Embeddings
• TF-IDF: Unsupervised Sentence Representations as Word Information Series: Revisiting TF--IDF
• P-norm: Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• DisC: A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs
• GEM: Zero-Training Sentence Embedding via Orthogonal Basis
• SWEM: Baseline Needs More Love: On Simple Word-Embedding-Based Modelsand Associated Pooling Mechanisms
• VLAWE: Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation
• Efficient Sentence Embedding using Discrete Cosine Transform

Encoders

date	paper	citation count	code	model_name
-	Incremental Domain Adaptation for Neural Machine Translation in Low-Resource Settings	N/A	Python	AraSIF
2014/05	Distributed Representations of Sentences and Documents	999+	Pytorch Python	Doc2Vec
2014/11	Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models	505	Theano Pytorch	VSE
2015/06	Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books	314	Theano TF Pytorch, Torch	SkipThought
2015/11	Order-Embeddings of Images and Language	200	Theano	order-embedding
2015/11	Towards Universal Paraphrastic Sentence Embeddings	262	Theano	ParagramPhrase
2015/??	From Word Embeddings to Document Distances	541	C, Python	Word Mover's Distance
2016/02	Learning Distributed Representations of Sentences from Unlabelled Data	251	Python	FastSent
2016/07	Charagram: Embedding Words and Sentences via Character n-grams	87	Theano	Charagram
2016/11	Learning Generic Sentence Representations Using Convolutional Neural Networks	46	Theano	ConvSent
2017/03	Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features	148	C++	Sent2Vec
2017/04	Learning to Generate Reviews and Discovering Sentiment	177	TF Pytorch Pytorch	Sentiment Neuron
2017/05	Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings	40	Theano	GRAN
2017/05	Supervised Learning of Universal Sentence Representations from Natural Language Inference Data	518	Pytorch	InferSent
2017/07	VSE++: Improving Visual-Semantic Embeddings with Hard Negatives	72	Pytorch	VSE++
2017/08	Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm	167	Keras Pytorch	DeepMoji
2017/09	StarSpace: Embed All The Things!	57	C++	StarSpace
2017/10	DisSent: Learning Sentence Representations from Explicit Discourse Relations	38	Pytorch	DisSent
2017/11	Dual-Path Convolutional Image-Text Embedding with Instance Loss	22	Matlab	Image-Text-Embedding
2017/11	Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations	48	Theano	para-nmt
2018/03	Universal Sentence Encoder	127	TF-Hub	USE
2018/03	An efficient framework for learning sentence representations	69	TF	Quick-Thought
2018/04	End-Task Oriented Textual Entailment via Deep Explorations of Inter-Sentence Interactions	7	Theano	DEISTE
2018/04	Learning general purpose distributed sentence representations via large scale multi-task learning	98	Pytorch	GenSen
2018/06	Embedding Text in Hyperbolic Spaces	19	TF	HyperText
2018/07	Representation Learning with Contrastive Predictive Coding	95	Keras	CPC
2018/08	Context Mover’s Distance & Barycenters: Optimal transport of contexts for building representations	2	Python	CMD
2018/09	Learning Universal Sentence Representations with Mean-Max Attention Autoencoder	3	TF	Mean-MaxAAE
2018/10	Improving Sentence Representations with Consensus Maximisation	1	-	Multi-view
2018/10	BioSentVec: creating sentence embeddings for biomedical texts	13	Python	BioSentVec
2018/10	Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model	5	TF-Hub	USE-xling
2018/11	Word Mover's Embedding: From Word2Vec to Document Embedding	13	C, Python	WordMoversEmbeddings
2018/11	A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks	23	Pytorch	HMTL
2018/12	Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond	45	Pytorch	LASER
2018/??	Convolutional Neural Network for Universal Sentence Embeddings	0	Theano	CSE
2019/01	No Training Required: Exploring Random Encoders for Sentence Classification	14	Pytorch	randsent
2019/02	CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model	0	Pytorch	CMOW
2019/07	GLOSS: Generative Latent Optimization of Sentence Representations	0	-	GLOSS
2019/07	Multilingual Universal Sentence Encoder	1	TF-Hub	MultilingualUSE
2019/08	Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks	1	Pytorch	Sentence-BERT

Evaluation

• decaNLP: The Natural Language Decathlon: Multitask Learning as Question Answering
• SentEval: SentEval: An Evaluation Toolkit for Universal Sentence Representations
• GLUE: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Exploring Semantic Properties of Sentence Embeddings
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Word Embeddings Benchmarks: How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
• MLDoc: A Corpus for Multilingual Document Classification in Eight Languages
• LexNET: Olive Oil Is Made of Olives, Baby Oil Is Made for Babies: Interpreting Noun Compounds Using Paraphrases in a Neural Model
• wordvectors.net: Community Evaluation and Exchange of Word Vectors at wordvectors.org
• jiant: Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling
• jiant: What do you learn from context? Probing for sentence structure in contextualized word representations
• Evaluation of sentence embeddings in downstream and linguistic probing tasks
• QVEC: Evaluation of Word Vector Representations by Subspace Alignment
• Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
• EQUATE : A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference
• Evaluating Word Embedding Models: Methods andExperimental Results
• How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions
• Linguistic Knowledge and Transferability of Contextual Representations: contextual-repr-analysis
• LINSPECTOR: Multilingual Probing Tasks for Word Representations
• Pitfalls in the Evaluation of Sentence Embeddings
• Probing Multilingual Sentence Representations With X-Probe: xprobe

Misc

• Word Embedding Dimensionality Selection: On the Dimensionality of Word Embedding
• Half-Size: Simple and Effective Dimensionality Reduction for Word Embeddings
• magnitude: Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package
• To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
• Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors: fuzzymax
• The Pupil Has Become the Master: Teacher-Student Model-BasedWord Embedding Distillation with Ensemble Learning: EmbeddingDistillation
• Improving Distributional Similarity with Lessons Learned from Word Embeddings: hyperwords
• Misspelling Oblivious Word Embeddings: moe
• Single Training Dimension Selection for Word Embedding with PCA
• Compressing Word Embeddings via Deep Compositional Code Learning: neuralcompressor
• UER: An Open-Source Toolkit for Pre-training Models: UER-py
• Situating Sentence Embedders with Nearest Neighbor Overlap

Vector Mapping

• Cross-lingual Word Vectors Projection Using CCA: Improving Vector Space Word Representations Using Multilingual Correlation
• vecmap: A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• MUSE: Unsupervised Machine Translation Using Monolingual Corpora Only
• CrossLingualELMo: Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Articles

• Comparing Sentence Similarity Methods
• The Current Best of Universal Word Embeddings and Sentence Embeddings
• On sentence representations, pt. 1: what can you fit into a single #$!%@*&% blog post?
• Deep-learning-free Text and Sentence Embedding, Part 1
• Deep-learning-free Text and Sentence Embedding, Part 2
• An Overview of Sentence Embedding Methods
• Word embeddings in 2017: Trends and future directions
• A Walkthrough of InferSent – Supervised Learning of Sentence Embeddings
• A survey of cross-lingual word embedding models
• Introducing state of the art text classification with universal language models
• Document Embedding Techniques