Foundation Model Gym
纸上得来终觉浅，绝知此事要躬行。
Step 0. pytorch warmup
Step 1. Transformer (1 week)
targets
- a transformer implementation from scratch
requirements
- pytorch Transformer API (fine-grained)
- benchmarks (comparing with pytorch's implementation)
- the Transformer paper
outputs
- a transformer matching reported performances
- implementation notes/wiki
Step 2. core models (1 week)
targets
- the BERT implementation
- optimization skills (1 GPU + accumulated gradient)
requirements
- data pre-processing pipeline:
- tokenizers (Huggingface) + raw data --> pre-processed data --> truncated samples -->MLM/NSP training data (json)
- Huggingface BERT-base retrained with 10 million samples (reference full BERT base training requires 8*V100*4 days), and their learning curves/MLM accuracies/NSP accuracies
- 1 GPU with small batch sizes or accumulated gradients
- 8 GPUs with the official BERT setting
- the BERT paper
outputs
- BERT base model matching reported performances on benchmarks
- implementation notes/wiki
Step 3. core peripherals (2 weeks)
targets
- strategies on building vocabulary (tokenizers (word piece/sentence piece/bpe))
- strategies on positional embeddings
- masking, sampling, ...
requirements
- raw data
- Huggingface tokenizer API
- data preprocessing papers
outputs
- data pre-processing pipelines
- implementation notes/wiki
Step 4. fine tune (1 week)
targets
- fine-tuning skills
requirements
- GLUE evaluation toolkits
outputs
- fine tuned BERTs with reported GLUE performances
Step 5. fm with decoders (2 weeks)
targets
- fm with decoders (GPT, unilm, BART, T5)
requirements
- raw data
- Hugging face API
outputs
- fm models matching reported performances on benchmarks
- implementation notes/wiki
Step 6. useful extensions (2 weeks)
more fm models
- XLNet (different pre-training objectives)
- tinyBERT, ALBERT (parameter sharing and compressing)
- roBERTa (data scale-up)
more pre-training, fine-tuning tricks
- learning rates (layer-wise learning rates, warmup)
- training with resource constraints (early exit, accumulate gradient (batch size))
more data pre-processing
- backdoor injection