SusannaWull/awesome-sentence-embedding

A curated list of pretrained sentence and word embedding models

Github please do not ban me!

awesome-sentence-embedding

A curated list of pretrained sentence and word embedding models

Table of Contents

• About This Repo
• General Framework
• Word Embeddings
• OOV Handling
• Contextualized Word Embeddings
• Pooling Methods
• Encoders
• Evaluation
• Misc
• Vector Mapping
• Articles

About This Repo

• well there are some awesome-lists for word embeddings and sentence embeddings, but all of them are outdated and more importantly incomplete
• this repo will also be incomplete, but I'll try my best to find and include all the papers with pretrained models
• this is not a typical awesome list because it has tables but I guess it's ok and much better than just a huge list
• if you find any mistakes or find another paper or anything please send a pull request and help me to keep this list up to date
• enjoy!

General Framework

• Almost all the sentence embeddings work like this:
• Given some sort of word embeddings and an optional encoder (for example an LSTM) they obtain the contextualized word embeddings.
• Then they define some sort of pooling (it can be as simple as last pooling).
• Based on that they either use it directly for the supervised classification task (like infersent) or generate the target sequence (like skip-thought).
• So, in general, we have many sentence embeddings that you have never heard of, you can simply do mean-pooling over any word embedding and it's a sentence embedding!

Word Embeddings

• Note: don't worry about the language of the code, you can almost always (except for the subword models) just use the pretrained embedding table in the framework of your choice and ignore the training code

date	paper	citation count	training code	pretrained models
-	WebVectors: A Toolkit for Building Web Interfaces for Vector Semantic Models	N/A	-	RusVectōrēs(broken)
-	Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings	N/A	-	ChineseEmbedding
2013/01	Efficient Estimation of Word Representations in Vector Space	999+	C	Word2Vec
2014/12	Word Representations via Gaussian Embedding	127	Cython	-
2014/??	Dependency-Based Word Embeddings	519	C++	word2vecf
2014/??	A Probabilistic Model for Learning Multi-Prototype Word Embeddings	81	DMTK	-
2014/??	GloVe: Global Vectors for Word Representation	998	C	GloVe
2015/06	From Paraphrase Database to Compositional Paraphrase Model and Back	0	Theano	PARAGRAM
2015/06	Non-distributional Word Vector Representations	37	Python	WordFeat
2015/06	Sparse Overcomplete Word Vector Representations	70	C++	-
2015/??	Topical Word Embeddings	134	Cython
2015/??	SensEmbed: Learning Sense Embeddings for Word and Relational Similarity	143	-	SensEmbed
2015/??	Joint Learning of Character and Word Embeddings	95	C	-
2016/02	Swivel: Improving Embeddings by Noticing What's Missing	35	TF	-
2016/03	Counter-fitting Word Vectors to Linguistic Constraints	108	Python	counter-fitting(broken)
2016/05	Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec	26	Chainer	-
2016/06	Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations	31	Go	lexvec
2016/06	Siamese CBOW: Optimizing Word Embeddings for Sentence Representations	92	Theano	Siamese CBOW
2016/07	Enriching Word Vectors with Subword Information	999+	C++	fastText
2016/08	Morphological Priors for Probabilistic Neural Word Embeddings	18	Theano	-
2016/11	A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks	154	C++	charNgram2vec
2016/12	ConceptNet 5.5: An Open Multilingual Graph of General Knowledge	136	Python	Numberbatch
2016/??	Learning Word Meta-Embeddings	19	-	Meta-Emb(broken)
2017/02	Offline bilingual word vectors, orthogonal transformations and the inverted softmax	131	Python	-
2017/04	Multimodal Word Distributions	30	TF	word2gm
2017/05	Poincaré Embeddings for Learning Hierarchical Representations	115	Pytorch	-
2017/06	Context encoders as a simple but powerful extension of word2vec	3	Python	-
2017/06	Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints	55	TF	Attract-Repel
2017/08	Making Sense of Word Embeddings	16	Python	sensegram
2017/08	Learning Chinese Word Representations From Glyphs Of Characters	N/A	C	-
2017/09	Hash Embeddings for Efficient Word Representations	6	Keras	-
2017/10	BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages	11	Gensim	BPEmb
2017/11	SPINE: SParse Interpretable Neural Embeddings	12	Pytorch	SPINE
2017/??	Dict2vec : Learning Word Embeddings using Lexical Dictionaries	15	C++	Dict2vec
2017/??	Joint Embeddings of Chinese Words, Characters, and Fine-grained Subcharacter Components	16	C	-
2017/??	AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP	22	Gensim	AraVec
2017/??	Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics	9	C	-
2018/04	Representation Tradeoffs for Hyperbolic Embeddings	22	Pytorch	h-MDS
2018/04	Dynamic Meta-Embeddings for Improved Sentence Representations	9	Pytorch	DME/CDME
2018/05	Analogical Reasoning on Chinese Morphological and Semantic Relations	19	-	ChineseWordVectors
2018/06	Probabilistic FastText for Multi-Sense Word Embeddings	9	C++	Probabilistic FastText
2018/09	Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks	1	TF	SynGCN
2018/09	FRAGE: Frequency-Agnostic Word Representation	12	Pytorch	-
2018/12	Wikipedia2Vec: An Optimized Tool for LearningEmbeddings of Words and Entities from Wikipedia	0	Cython	Wikipedia2Vec
2018/??	cw2vec: Learning Chinese Word Embeddings with Stroke n-gram Information	10	C++	-
2019/02	VCWE: Visual Character-Enhanced Word Embeddings	0	Pytorch	VCWE
2019/05	Learning Cross-lingual Embeddings from Twitter via Distant Supervision	0	Text	-

OOV Handling

• Drop OOV words!
• One OOV vector(unk vector)
• Use subword models(ngram, bpe, char)
• ALaCarte: A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
• Mimick: Mimicking Word Embeddings using Subword RNNs
• CompactReconstruction: Subword-based Compact Reconstruction of Word Embeddings

Contextualized Word Embeddings

• Note: all the unofficial models can load the official pretrained models

date	paper	citation count	code	pretrained models
-	Language Models are Unsupervised Multitask Learners	N/A	TF Pytorch Keras	GPT-2
2017/08	Learned in Translation: Contextualized Word Vectors	188	Pytorch Keras	CoVe
2018/01	Universal Language Model Fine-tuning for Text Classification	190	Pytorch	ULMFit(English, Zoo)
2018/02	Deep contextualized word representations	803	Pytorch TF	ELMO(AllenNLP, TF-Hub)
2018/04	Efficient Contextualized Representation:Language Model Pruning for Sequence Labeling	3	Pytorch	LD-Net
2018/07	Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation	10	Pytorch	ELMo
2018/08	Direct Output Connection for a High-Rank Language Model	6	Pytorch	DOC
2018/10	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	719	TF Keras Pytorch MXNet PaddlePaddle TF Keras	BERT(BERT, ERNIE)
2018/??	Improving Language Understanding by Generative Pre-Training	222	TF Keras Pytorch	GPT
2018/??	Contextual String Embeddings for Sequence Labeling	25	Pytorch	Flair
2019/01	BioBERT: pre-trained biomedical language representation model for biomedical text mining	15	TF	BioBERT
2019/01	Cross-lingual Language Model Pretraining	28	Pytorch	XLM
2019/01	Multi-Task Deep Neural Networks for Natural Language Understanding	33	Pytorch	MT-DNN
2019/01	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	36	TF Pytorch Pytorch	Transformer-XL
2019/03	SciBERT: Pretrained Contextualized Embeddings for Scientific Text	5	Pytorch, TF	SciBERT
2019/04	ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission	3	Pytorch	ClinicalBERT
2019/04	Publicly Available Clinical BERT Embeddings	7	Text	clinicalBERT
2019/05	ERNIE: Enhanced Language Representation with Informative Entities	3	Pytorch	ERNIE
2019/06	Pre-Training with Whole Word Masking for Chinese BERT	0	Pytorch, TF	BERT-wwm
2019/06	XLNet: Generalized Autoregressive Pretraining for Language Understanding	5	TF	XLNet

Pooling Methods

• {Last, Mean, Max}-Pooling
• Special Token Pooling (like BERT and OpenAI's Transformer)
• SIF: A Simple but Tough-to-Beat Baseline for Sentence Embeddings
• TF-IDF: Unsupervised Sentence Representations as Word Information Series: Revisiting TF--IDF
• P-norm: Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• DisC: A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs
• GEM: Zero-Training Sentence Embedding via Orthogonal Basis
• SWEM: Baseline Needs More Love: On Simple Word-Embedding-Based Modelsand Associated Pooling Mechanisms
• VLAWE: Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation

Encoders

date	paper	citation count	code	model_name
2014/05	Distributed Representations of Sentences and Documents	998	Pytorch Python	Doc2Vec
2014/11	Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models	452	Theano Pytorch	VSE
2015/06	Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books	232	Theano TF Pytorch, Torch	SkipThought
2015/11	Order-Embeddings of Images and Language	171	Theano	order-embedding
2015/11	Towards Universal Paraphrastic Sentence Embeddings	231	Theano	ParagramPhrase
2015/??	From Word Embeddings to Document Distances	445	C, Python	Word Mover's Distance
2016/02	Learning Distributed Representations of Sentences from Unlabelled Data	223	Python	FastSent
2016/07	Charagram: Embedding Words and Sentences via Character n-grams	76	Theano	Charagram
2016/11	Learning Generic Sentence Representations Using Convolutional Neural Networks	35	Theano	ConvSent
2017/03	Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features	119	C++	Sent2Vec
2017/04	Learning to Generate Reviews and Discovering Sentiment	143	TF Pytorch Pytorch	Sentiment Neuron
2017/05	Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings	34	Theano	GRAN
2017/05	Supervised Learning of Universal Sentence Representations from Natural Language Inference Data	373	Pytorch	InferSent
2017/07	VSE++: Improving Visual-Semantic Embeddings with Hard Negatives	35	Pytorch	VSE++
2017/08	Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm	132	Keras Pytorch	DeepMoji
2017/09	StarSpace: Embed All The Things!	N/A	C++	StarSpace
2017/10	DisSent: Learning Sentence Representations from Explicit Discourse Relations	33	Pytorch	DisSent
2017/11	Dual-Path Convolutional Image-Text Embedding with Instance Loss	1	Matlab	Image-Text-Embedding
2017/11	Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations	34	Theano	para-nmt
2018/03	An efficient framework for learning sentence representations	47	TF	Quick-Thought
2018/03	Universal Sentence Encoder	78	TF-Hub	USE
2018/04	End-Task Oriented Textual Entailment via Deep Explorations of Inter-Sentence Interactions	2	Theano	DEISTE
2018/04	Learning general purpose distributed sentence representations via large scale multi-task learning	74	Pytorch	GenSen
2018/06	Embedding Text in Hyperbolic Spaces	15	TF	HyperText
2018/07	Representation Learning with Contrastive Predictive Coding	61	Keras	CPC
2018/08	Context Mover’s Distance & Barycenters: Optimal transport of contexts for building representations	1	Python	CMD
2018/09	Learning Universal Sentence Representations with Mean-Max Attention Autoencoder	2	TF	Mean-MaxAAE
2018/10	Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model	1	TF-Hub	USE-xling
2018/10	BioSentVec: creating sentence embeddings for biomedical texts	5	Python	BioSentVec
2018/11	A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks	10	Pytorch	HMTL
2018/11	Word Mover's Embedding: From Word2Vec to Document Embedding	8	C, Python	WordMoversEmbeddings
2018/12	Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond	16	Pytorch	LASER
2018/??	Convolutional Neural Network for Universal Sentence Embeddings	0	Theano	CSE
2019/01	No Training Required: Exploring Random Encoders for Sentence Classification	9	Pytorch	randsent
2019/02	CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model	0	Pytorch	CMOW
2019/07	Multilingual Universal Sentence Encoder	1	TF-Hub	MultilingualUSE

Evaluation

• decaNLP: The Natural Language Decathlon: Multitask Learning as Question Answering
• SentEval: SentEval: An Evaluation Toolkit for Universal Sentence Representations
• GLUE: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Exploring Semantic Properties of Sentence Embeddings
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Word Embeddings Benchmarks: How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
• MLDoc: A Corpus for Multilingual Document Classification in Eight Languages
• LexNET: Olive Oil Is Made of Olives, Baby Oil Is Made for Babies: Interpreting Noun Compounds Using Paraphrases in a Neural Model
• wordvectors.net: Community Evaluation and Exchange of Word Vectors at wordvectors.org
• jiant: Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling
• jiant: What do you learn from context? Probing for sentence structure in contextualized word representations
• Evaluation of sentence embeddings in downstream and linguistic probing tasks
• QVEC: Evaluation of Word Vector Representations by Subspace Alignment
• Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
• EQUATE : A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference
• Evaluating Word Embedding Models: Methods andExperimental Results
• How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions
• Linguistic Knowledge and Transferability of Contextual Representations: contextual-repr-analysis
• LINSPECTOR: Multilingual Probing Tasks for Word Representations

Misc

• Word Embedding Dimensionality Selection: On the Dimensionality of Word Embedding
• Half-Size: Simple and Effective Dimensionality Reduction forWord Embeddings
• magnitude: Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package
• To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
• Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors: fuzzymax
• The Pupil Has Become the Master: Teacher-Student Model-BasedWord Embedding Distillation with Ensemble Learning: EmbeddingDistillation
• Improving Distributional Similarity with Lessons Learned from Word Embeddings: hyperwords

Vector Mapping

• Cross-lingual Word Vectors Projection Using CCA: Improving Vector Space Word Representations Using Multilingual Correlation
• vecmap: A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• MUSE: Unsupervised Machine Translation Using Monolingual Corpora Only
• CrossLingualELMo: Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Articles

• Comparing Sentence Similarity Methods
• The Current Best of Universal Word Embeddings and Sentence Embeddings
• On sentence representations, pt. 1: what can you fit into a single #$!%@*&% blog post?
• Deep-learning-free Text and Sentence Embedding, Part 1
• Deep-learning-free Text and Sentence Embedding, Part 2
• An Overview of Sentence Embedding Methods
• Word embeddings in 2017: Trends and future directions
• A Walkthrough of InferSent – Supervised Learning of Sentence Embeddings
• A survey of cross-lingual word embedding models
• Introducing state of the art text classification with universal language models