GPT implementation in pure Python.
You've seen nanoGPT.
You've seen picoGPT.
Now, imagine you're alone on a desert island 🏝️ with a computer, Python installed and no Internet (you can't do pip install torch 🥲).
A crazy idea occurs to you: what if I created a GPT model... from scratch?
Here is the result of this work: femtoGPT, a GPT implementation in pure Python, without any dependencies... and with it's own autograd engine (inspired by micrograd).
- No tensor ❌
- Not fast ❌
- No batch (one sample at a time) ❌
- No GPU support ❌
A pure learning experience.
Enjoy!
Install the package:
git clone https://github.com/cthiriet/femtoGPT
pip install .Train a femto GPT model on a dataset, that is actually...just a string:
# Train on the "hello world" sentence
python femtogpt/train.py --dataset "hello world"pytest