jamesparsloe/TinyLM

Tiny language modeling playground

TinyLM

My playground for language modeling.

Data

I'll by using the TinyStories dataset from TinyStories: How Small Can Language Models Be and Still Speak Coherent English?.

Models

• GPT
• Mamba
• Based
• ...

Experiments

• https://wandb.ai/jamesparsloe/TinyLM

Just training the LayerNorms

• Inspired by Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs (Frankle et al., 2020)

• https://wandb.ai/jamesparsloe/TinyLM/runs/tury2cp5

• ./runs/tury2cp5/TinyLM-006000.pt