rsn870/grokking_experiments

modded version of an unofficial re-implementation of "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets"

Experiments with Gtokking

We test if the observations of the paper stand in a multimodal setting.

--------------------------- DIVIDING LINE --------------------------------------------------------------------------------------

GROKKING: GENERALIZATION BEYOND OVERFITTING ON SMALL ALGORITHMIC DATASETS

unofficial re-implementation of this paper by Power et al.

Based on the code written by Charlie Snell

pull and install:

git clone https://github.com/rsn870/grokking_experiments.git
cd grokking_experiments/
pip install -r requirements.txt

To roughly re-create Figure 1 in the paper run:

export PYTHONPATH=$(pwd)/grokk_replica/
cd scripts/
python train_grokk.py

Running the above command should give curves like this.

Try different operations or learning / architectural hparams by modifying configurations in the config/ directory. I use Hydra to handle the configs (see their documentation to learn how to change configs in the commandline etc...).

Training uses Weights And Biases by default to generate plots in realtime. If you would not like to use wandb, just set wandb.use_wandb=False in config/train_grokk.yaml or as an argument when calling train_grokk.py