/fastai_timm

Exploring fastai and timm integration for model finetune

Primary LanguageJupyter Notebook

A bunch of experiments on integrating fastai with timm

The main idea of this repo is finding the best way to integrate timm backbones into fastai so they fine tune good on downstream tasks.

Implementing a splitter for timm models

Timm models are different than torchvision ones, and need a proper splitter of parameters so they can be properly finetuned.

Experiments

We use W&B for experiment tracking trough the WandbCallback in fastai. The task we are trying to optimize is fine tunning a pretrained backbone on a new dataset.

The fine tunning is performed with fine_tune for a fixed number of epochs.

$ python fine_tune.py --help                                                                      
usage: fine_tune.py [-h] [--batch_size BATCH_SIZE] [--epochs EPOCHS] [--num_experiments NUM_EXPERIMENTS] [--learning_rate LEARNING_RATE]
                    [--img_size IMG_SIZE] [--resize_method RESIZE_METHOD] [--model_name MODEL_NAME] [--split_func SPLIT_FUNC] [--pool POOL] [--seed SEED]
                    [--wandb_project WANDB_PROJECT] [--wandb_entity WANDB_ENTITY] [--dataset DATASET]

options:
  -h, --help            show this help message and exit
  --batch_size BATCH_SIZE
  --epochs EPOCHS
  --num_experiments NUM_EXPERIMENTS
  --learning_rate LEARNING_RATE
  --img_size IMG_SIZE
  --resize_method RESIZE_METHOD
  --model_name MODEL_NAME
  --split_func SPLIT_FUNC
  --pool POOL
  --seed SEED
  --wandb_project WANDB_PROJECT
  --wandb_entity WANDB_ENTITY
  --dataset DATASET

Running a Sweep with W&B

  • Create a W&B account on https://wandb.ai/signup

  • Setup your machine for fastai, you will need to install latest fastai and wandb. We can provide credtis on jarvislabs.ai if needed.

  • Clone this repo

Time to run the experiment, We can perform hyperparameter sweep using W&B.

  • Setting your sweep YAML file: To run a sweep you will need first to configure what parameters you want to explore, this is done on the sweep.yaml file. In this file, you will specify the parameters the sweep will explore. For instance, setting:
learning_rate:
    values: [0.002, 0.008]

tells the sweep to choose the learning_rate param from the 2 values: 0.002 and 0.008. What happens if we want a distribution?

learning_rate:
    distribution: uniform
    min: 0.0001
    max: 0.01

You can check all the possibilities for defining parameters here.

  • Creating the sweep: In a terminal window, run:
wandb sweep sweep.yaml
  • Running the actual training of the sweep (you can paste the output of the previous command here)
wandb agent <SWEEP_ID>

References: