aceyiyuan/NLP_Transformer

Natural Language Processing Specialization on Coursera (offered by deeplearning.ai)

Programming assignments from all courses in the Coursera Natural Language Processing Specialization offered by deeplearning.ai.

Credits

This repo contains my work for this specialization. The code base, quiz questions and diagrams are taken from the Natural Language Processing Specialization, unless specified otherwise.

Courses

The Natural Language Processing Specialization on Coursera contains four courses:

• Course 1: Natural Language Processing with Classification and Vector Spaces
• Course 2: Natural Language Processing with Probabilistic Models
• Course 3: Natural Language Processing with Sequence Models
• Course 4: Natural Language Processing with Attention Models

Specialization Info

• Natural Language Processing (NLP) uses algorithms to understand and manipulate human language. This technology is one of the most broadly applied areas of machine learning. As AI continues to expand, so will the demand for professionals skilled at building models that analyze speech and language, uncover contextual patterns, and produce insights from text and audio.

• By the end of this specialization, you will be ready to design NLP applications that perform question-answering and sentiment analysis, create tools to translate languages and summarize text, and even build chatbots. These and other NLP applications are going to be at the forefront of the coming transformation to an AI-powered future.

• This Specialization is designed and taught by two experts in NLP, machine learning, and deep learning. Younes Bensouda Mourri is an Instructor of AI at Stanford University who also helped build the Deep Learning Specialization. Łukasz Kaiser is a Staff Research Scientist at Google Brain and the co-author of Tensorflow, the Tensor2Tensor and Trax libraries, and the Transformer paper.

Topics Covered

This Specialization will equip you with the state-of-the-art deep learning techniques needed to build cutting-edge NLP systems:

• Use logistic regression, naïve Bayes, and word vectors to implement sentiment analysis, complete analogies, and translate words, and use locality sensitive hashing for approximate nearest neighbors.

• Use dynamic programming, hidden Markov models, and word embeddings to autocorrect misspelled words, autocomplete partial sentences, and identify part-of-speech tags for words.

• Use dense and recurrent neural networks, LSTMs, GRUs, and Siamese networks in TensorFlow and Trax to perform advanced sentiment analysis, text generation, named entity recognition, and to identify duplicate questions.

• Use encoder-decoder, causal, and self-attention to perform advanced machine translation of complete sentences, text summarization, question-answering and to build chatbots. Models covered include T5, BERT, transformer, reformer, and more! Enjoy!

Programming Assignments

Course 1: Natural Language Processing with Classification and Vector Spaces

• Week 1
  • Assignment:
    • Sentiment Analysis with Logistic Regression
  • Labs:
    • Natural language Preprocessing
    • Visualizing word frequencies
    • Visualizing tweets and Logistic Regression models
• Week 2
  • Assignment:
    • Naive Bayes
  • Labs:
    • Visualizing likelihoods and confidence ellipses
• Week 3
  • Assignment:
    • Word Embeddings: Hello Vectors
  • Labs:
    • Linear algebra in Python with Numpy
    • Manipulating word embeddings
    • Another explanation about PCA
• Week 4
  • Assignment:
    • Word Translation
  • Labs:
    • Rotation matrices in R2
    • Hash tables

Course 2: Natural Language Processing with Probabilistic Models

• Week 1
  • Assignment:
    • Autocorrect
  • Labs:
    • Building the vocabulary
    • Candidates from edits
• Week 2
  • Assignment:
    • Part of Speech Tagging
  • Labs:
    • Working with text data
    • Working with tags and NumPy
• Week 3
  • Assignment:
    • Autocomplete
  • Labs:
    • Corpus preprocessing for N-grams
    • Building the language model
    • Language model generalization
• Week 4
  • Assignment:
    • Word Embeddings
  • Labs:
    • Data Preparation
    • Intro to CBOW model
    • Training the CBOW model
    • Word Embeddings
    • Word Embeddings Step by Step

Course 3: Natural Language Processing with Sequence Models

• Week 1
  • Assignment:
    • Sentiment with Deep Neural Networks
  • Labs:
    • Introduction to Trax
    • Classes and Subclasses
    • Data Generators
• Week 2
  • Assignment:
    • Deep N-grams
  • Labs:
    • Hidden State Activation
    • Working with JAX NumPy and Calculating Perplexity
    • Vanilla RNNs, GRUs and the scan function
    • Creating a GRU model using Trax
• Week 3
  • Assignment:
    • Named Entity Recognition (NER)
  • Labs:
    • Vanishing Gradients
• Week 4
  • Assignment:
    • Question duplicates
  • Labs:
    • Creating a Siamese Model using Trax
    • Modified Triplet Loss
    • Evaluate a Siamese Model

Course 4: Natural Language Processing with Attention Models

• Week 1
  • Assignment:
    • NMT with Attention
  • Labs:
    • Stack Semantics
    • BLEU Score
• Week 2
  • Assignment:
    • Transformer Summarizer
  • Labs:
    • Attention
    • The Transformer Decoder
• Week 3
  • Assignment:
    • Question Answering
  • Labs:
    • SentencePiece and BPE
    • BERT Loss
    • T5
• Week 4
  • Assignment:
    • Chatbot
  • Labs:
    • Reformer LSH
    • Revnet