/awesome-nlp

:book: A curated list of resources dedicated to Natural Language Processing (NLP)

Creative Commons Zero v1.0 UniversalCC0-1.0

awesome-nlp

Awesome

A curated list of resources dedicated to Natural Language Processing

Awesome NLP Logo

Read this in English, Traditional Chinese

Please read the contribution guidelines before contributing. Please add your favourite NLP resource by raising a pull request

Contents

Research Summaries and Trends

Prominent NLP Research Labs

Back to Top

Tutorials

Back to Top

Reading Content

General Machine Learning

  • Machine Learning 101 from Google's Senior Creative Engineer explains Machine Learning for engineer's and executives alike
  • AI Playbook - a16z AI playbook is a great link to forward to your managers or content for your presentations
  • Ruder's Blog by Sebastian Ruder for commentary on the best of NLP Research

Introductions and Guides to NLP

Blogs and Newsletters

Videos and Online Courses

Back to Top

Books

Libraries

Back to Top

  • Node.js and Javascript - Node.js Libaries for NLP | Back to Top

    • Twitter-text - A JavaScript implementation of Twitter's text processing library
    • Knwl.js - A Natural Language Processor in JS
    • Retext - Extensible system for analyzing and manipulating natural language
    • NLP Compromise - Natural Language processing in the browser
    • Natural - general natural language facilities for node
    • Poplar - A web-based annotation tool for natural language processing (NLP)
    • NLP.js - An NLP library for building bots
  • Python - Python NLP Libraries | Back to Top

    • TextBlob - Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language Toolkit (NLTK) and Pattern, and plays nicely with both 👍
    • spaCy - Industrial strength NLP with Python and Cython 👍
      • textacy - Higher level NLP built on spaCy
    • gensim - Python library to conduct unsupervised semantic modelling from plain text 👍
    • scattertext - Python library to produce d3 visualizations of how language differs between corpora
    • GluonNLP - A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a wide range of NLP tasks.
    • AllenNLP - An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
    • PyTorch-NLP - NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU
    • Rosetta - Text processing tools and wrappers (e.g. Vowpal Wabbit)
    • PyNLPl - Python Natural Language Processing Library. General purpose NLP library for Python. Also contains some specific modules for parsing common NLP formats, most notably for FoLiA, but also ARPA language models, Moses phrasetables, GIZA++ alignments.
    • jPTDP - A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages.
    • BigARTM - a fast library for topic modelling
    • Snips NLU - A production ready library for intent parsing
    • Chazutsu - A library for downloading&parsing standard NLP research datasets
    • Word Forms - Word forms can accurately generate all possible forms of an English word
    • Multilingual Latent Dirichlet Allocation (LDA) - A multilingual and extensible document clustering pipeline
    • NLP Architect - A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU
    • Flair - A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings.
    • Kashgari - Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.
    • FARM - FARM makes cutting-edge transfer learning simple and helps you to leverage pretrained language models for your own NLP tasks.

Services

NLP as API with higher level functionality such as NER, Topic tagging and so on | Back to Top

Annotation Tools

  • GATE - General Architecture and Text Engineering is 15+ years old, free and open source
  • Anafora is free and open source, web-based raw text annotation tool
  • brat - brat rapid annotation tool is an online environment for collaborative text annotation
  • doccano - doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence
  • tagtog, costs $
  • prodigy is an annotation tool powered by active learning, costs $
  • LightTag - Hosted and managed text annotation tool for teams, costs $
  • rstWeb - open source local or online tool for discourse tree annotations
  • GitDox - open source server annotation tool with GitHub version control and validation for XML data and collaborative spreadsheet grids

Techniques

Text Embeddings

Word Embeddings

Sentence and Language Model Based Word Embeddings

Back to Top

Question Answering and Knowledge Extraction

Back to Top

Datasets

Back to Top

Multilingual NLP Frameworks

Back to Top

  • UDPipe is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and reliable solution for multilingual NLP processing.
  • NLP-Cube : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).

NLP in Korean

Back to Top

Libraries

  • KoNLPy - Python package for Korean natural language processing.
  • Mecab (Korean) - C++ library for Korean NLP
  • KoalaNLP - Scala library for Korean Natural Language Processing.
  • KoNLP - R package for Korean Natural language processing

Blogs and Tutorials

Datasets

NLP in Arabic

Back to Top

Libraries

  • goarabic - Go package for Arabic text processing
  • jsastem - Javascript for Arabic stemming
  • PyArabic - Python libraries for Arabic
  • RFTokenizer - trainable Python segmenter for Arabic, Hebrew and Coptic

Datasets

  • Multidomain Datasets - Largest Available Multi-Domain Resources for Arabic Sentiment Analysis
  • LABR - LArge Arabic Book Reviews dataset
  • Arabic Stopwords - A list of Arabic stopwords from various resources

NLP in Chinese

Back to Top

Libraries

  • jieba - Python package for Words Segmentation Utilities in Chinese
  • SnowNLP - Python package for Chinese NLP
  • FudanNLP - Java library for Chinese text processing

NLP in German

  • German-NLP - Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

NLP in Spanish

Back to Top

Data

Word and Sentence Embeddings

NLP in Indic languages

Back to Top

Data, Corpora and Treebanks

Corpora/Datasets that need a login/access can be gained via email

Language Models and Word Embeddings

Libraries and Tooling

NLP in Thai

Back to Top

Libraries

  • PyThaiNLP - Thai NLP in Python Package
  • JTCC - A character cluster library in Java
  • CutKum - Word segmentation with deep learning in TensorFlow
  • Thai Language Toolkit - Based on a paper by Wirote Aroonmanakun in 2002 with included dataset
  • SynThai - Word segmentation and POS tagging using deep learning in Python

Data

  • Inter-BEST - A text corpus with 5 million words with word segmentation
  • Prime Minister 29 - Dataset containing speeches of the current Prime Minister of Thailand

NLP in Danish

NLP in Vietnamese

Libraries

  • underthesea - Vietnamese NLP Toolkit
  • vn.vitk - A Vietnamese Text Processing Toolkit
  • VnCoreNLP - A Vietnamese natural language processing toolkit

Data

NLP in Indonesian

Datasets

Libraries & Embedding

Other Languages

  • Russian: pymorphy2 - a good pos-tagger for Russian
  • Asian Languages: Thai, Lao, Chinese, Japanese, and Korean ICU Tokenizer implementation in ElasticSearch
  • Ancient Languages: CLTK: The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages
  • Dutch: python-frog - Python binding to Frog, an NLP suite for Dutch. (pos tagging, lemmatisation, dependency parsing, NER)
  • Hebrew: NLPH_Resources - A collection of papers, corpora and linguistic resources for NLP in Hebrew

Back to Top

Credits for initial curators and sources

License

License - CC0