/transformers-gen-ai

Developing generative language models using transformers.

Primary LanguageJupyter NotebookMIT LicenseMIT

Transformers and Language Models

This purpose of this repository is to act as an entry-point to the world of transformer-based language modelling in PyTorch. It is pitched to those of us that want to understand implementation details and grasp theoretical insights, without having to wade through badly written research code 🙂

All code has been structured into a Python package called modelling, that is organised as follows:

└── src
    ├── modelling
    │   ├── data.py
    │   ├── rnn.py
    │   ├── transformer.py
    │   └── utils.py

We have done our best to make this as readable as possible and comprehensively documented, so this is the place to go for the implementation details. To see this in action, use the following notebooks:

notebooks
├── 0_attention_and_transformers.ipynb
├── 1_datasets_and_dataloaders.ipynb
├── 2_text_generation_with_rnns.ipynb
└── 3_text_generation_with_transformers.ipynb
└── 4_pre_trained_transformers_for_search.ipynb
└── 5_pre_trained_transformers_for_sentiment_analysis.ipynb

These will guide you through steps required to use the code contaied within the modelling package to train a language model and then use it to perform semantic search and sentiment classification tasks.

Installing

To run the notebooks and use the code within the src/modelling directory either clone this repository and install the package directly from the source code,

pip install .

Or install it directly from this repository,

pip install git+https://github.com/AlexIoannides/transformers.git@main

Presentation Slides

An HTML (and PDF) presentation of this work is contained in the presentation_slides directory.

Useful Resources

We found the following useful in our ascent up the transformer and LLMs learning curve: