Just playing around with Andrej Karpathy's Nano-GPT implementation.
To setup the environment, you will need conda,
# create a conda env
conda create -n nano-gpt python=3.8
conda activate nano-gpt
pip install -r requirements.txt
To run the code, you can use the following command,
python model.py
- The use of
torch.tril
and matrix multiplication to ensure autoregressive property for the decoder block. This property is not needed for the encoder block of a transformer. - TODO: Add more details here.