SushritPasupuleti/LLM-From-Scratch

Building an LLM from Scratch.

LLM from Scratch

Building an LLM from Scratch. This repo also hopes to serve as a guide to understanding the transformer architecture as well, and hence will include a lot of comments and explanations.

Setup

For NixOS users, the shell.nix file will setup your environment.

nix-shell ./shell.nix

Use the Makefile to setup the project or follow the steps below.

make help

Data Sources

• Project Gutenberg

References

• Attention is all you need

• The Illustrated Transformer - A great blog post with illustrations to understand the transformer architecture.