/commented-transformers

Highly commented implementations of Transformers in PyTorch

Primary LanguagePythonMIT LicenseMIT

Commented Transformers

Highly commented implementations of Transformers in PyTorch for Creating a Transformer From Scratch series:

  1. The Attention Mechanism
  2. The Rest of the Transformer

The layers folder contains implementations for Bidirectional Attention, Causal Attention, and CausalCrossAttention.

The models folder contains single file implementations for GPT-2 and BERT. Both models are compatible with torch.compile(..., fullgraph=True).