This repository has been authored by Arash Mohamadpour, an undergraduate researcher at the University of Tehran. It contains an implementation of the 'Memory Model of Neural Language Models' paper, which was developed under the guidance of Dr. Hadi Amiri.
In this repository we cover the following areas:
- Requirements
- Datasets
- Sequence Embeddings
- Neural Networks
- Data Prepration
- Evaluating
- Conclusion
You can execute the .ipynb files in Google Colab to obtain valid results. However, for the .py files, you will need to install certain Python libraries as specified in the utils file.
For this paper, we have gathered four main datasets with varying context lengths. Additional information about the vocabulary of this repository can be found here. You can access the dataset here.
In this section, we delve into five types of embeddings as described in the paper. These include:
- GloVe: Global Vectors for Word Representation
- Bert: Bidirectional Encoder Representations from Transformers
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- GPT: Generative Pre-training Transformer
- T5: Text-to-Text Transfer Transformer
- LSTM: Long Short-Term Memory
We'll get into each one, one by one.