A Generatively Pretrained Transformer that generates Shakespeare-eque quotes.
This project was created to accomplish the following goals:
- Gain a deeper understanding of the transformer architecture
- Implement multi-headed attention.
- Get insight into the power of transformers over standard NLP techniques.
I refered to Andrej Karpathy's demo of a GPT model as a foundation for this project: https://www.youtube.com/watch?v=kCc8FmEb1nY&t=5042s
There are a bunch of things pertaining to this project that I would like to experiment with in future.
- Currently the model works on word-level, thus the sentences generated may appear nonsensical. The next thing for me is to explore more advanced tokenization methods.
- Try out SentencePiece
- I also wish to train this model on different datasets to see how it performs.
