A place where I'm trying to understand transformers deeply and explore concepts in mechanistic interpretability, by using the Othello World dataset. I've reimplimented much of their dataset code (simplifying greatly), and relied on Neel Nanda's Transformer tutorial and Andrej Karparthy's GPT Tutorial.
To get started:
pip install -e . in the root directory will install the project and its dependencies.
python3 entrypoints/train.py yaml/transformer.yaml runs training with the config specified in yaml/transformer.yaml.