/BERT-Question-Answering

A study on encoding english sentences to tensorflow vectors or tensors using pre-trained BERT model from the Hugging Face Library.

Primary LanguageJupyter Notebook

DistilBERT Encoding and SQuAD Question Answering Model

A study on encoding english sentences to tensorflow vectors or tensors using pre-trained BERT model from the Hugging Face Library.

Overview

BERT (Bi-directional Encoder representation from Transformers) is a modern language model which can be used for sentence-encoding purposes for a wide variety of applications like sentiment analysis, embeddings, chatbots, text classification, etc. developed by researchers at Google Language AI. BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks. Hence, BERT effectively paves a way for Transfer Learning in the domain of Natural Language Processing.

BERT Models

We have used the BERT implementations on the popular HuggingFace Library

A Timeline of Various BERT Implementations

Timeline of the Implementations of BERT

A Functional Comparison of Some Modern BERT Implementations

Comparison of Various BERT Models

Fine-tuning BERT

The pre-training of a BERT model is done by 2 main methods :

  • Masked Language Model (MLM)
  • Next Sentence Prediction (NSP)

Then, the weights of the pre-trained model are fine-tuned as required for a particular use case. We can refer this paper to learn more about fine-tuning BERT for various specific tasks.

BERT Pre-Training and Fine-tuning

Study of BERT Encoding

Here we have studied the uncased DistilBERT model in detail, and how to implement it in practice to obtain the standard 768-dimensional feature vectors of any English sentence encoded using the pre-trained model.

SQuAD (Standford Question Answering Dataset) Model

SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.

Here, we have examined in detail this BERT-based model fine-tuned for the specific task of Question Answering. We have used the SQuAD implementation on the Huggingface library.

References