CAML: Context-Aware Meta-Learning

Updates:

March 22, 2024: Initial release.

This repository contains the official code for CAML, an in-context learning algorithm for few-shot image classification. CAML is designed for the universal meta-learning setting.

Universal meta-learning measures a model's capacity to quickly learn new image classes. It evaluates models across a diverse set of meta-learning benchmarks spanning many different image classification tasks without meta-training on any of the benchmarks' training sets or fine-tuning on the support set during inference.

Context-Aware Meta-Learning
Christopher Fifty, Dennis Duan, Ronald G. Junkins,
Ehsan Amid, Jure Leskovec, Christopher Ré, Sebastian Thrun
ICLR 2024
arXiv

Approach

CAML learns new visual concepts during inference without meta-training on related concepts or fine-tuning on the support set during inference. It is competitive with state-of-the-art meta-learning algorithms that meta-train on the training set of each benchmark in our testing framework.

Code environment

This code requires Pytorch 1.13 or higher with cuda support. It has been tested on Ubuntu 20.04.

You can create a conda environment with the correct dependencies using the following command lines:

cd CAML
conda env create -f environment.yml
conda activate caml

Setup

The directory structure for this project should look like:

Outer_Directory
│
│───caml_pretrained_models/
│   │   CAML_CLIP/
│   │   ...
│
│───caml_train_datasets/
│   │   fungi/
│   │   ...
│
│───caml_universal_eval_datasets/
│   │   Aircraft_fewshot/
│   │   ...
│
└───CAML
│   │   assets/
│   │   qpth_local/
│   │   src/
│   │   .gitignore
│   │   .enrionment.yml
│   │   .README.md

Downloading The Datasets and Pre-Trained Models

We offer four downloads:

The pre-trained model checkpoints. [Download Link]
The pre-training datasets as CLIP, Laion-2b, and ResNet34 image embeddings. [Download Link]
- Using image embeddings makes the training substantively faster.
The universal eval datasets. [Download Link]
qpth_local.zip: this is needed to train/test MetaOpt. [Download Link]

After unzipping a file, move it to the location specified by the directory diagram in Setup.

Training a Model

To pre-train CAML with a CLIP image encoder, ELMES label encoder, run the following:

python src/train.py \
     --opt adam \
     --lr 1e-5 \
     --epoch 100 \
     --val_epoch 1 \
     --batch_sizes 525 \
     --detailed_name \
     --encoder_size large \
     --dropout 0.0  \
     --fe_type cache:timm:vit_base_patch16_clip_224.openai:768 \
     --schedule custom_cosine \
     --fe_dtype float32 \
     --model CAML \
     --label_elmes \
     --save_dir test_CAML_repo \
     --gpu 0

Additional training details are located in a comment at the top of CAML/src/train.py.

Evaluating a Model in the Universal Setting

To evaluate CAML in the universal setting on Aircraft:

python src/evaluation/test.py --model CAML --gpu 0 --eval_dataset Aircraft  \
--fe_type timm:vit_base_patch16_clip_224.openai:768

You can also evaluate on any of the following datasets:

[Aircraft, pascal_paintings, mini_ImageNet, meta_iNat, ChestX, tiered_ImageNet, CUB_fewshot, 
tiered_meta_iNat, cifar, paintings, pascal]

Additional evaluation details are located in a comment at the top of CAML/src/evaluation/test.py.

Citation

This work builds on the codebase of Few-Shot Classification with Feature Map Reconstruction Networks, and uses the following datasets for pre-training.

We release CLIP, Laion-2b, and ResNet34 image embeddings as a download link under the "Downloading The Datasets and Pre-Trained Models" section.

The following datasets are used to evaluate universal meta-learning performance.

We release a zip file containing the test set as a download link under the "Downloading The Datasets and Pre-Trained Models" section.

If you use this codebase or otherwise found the ideas useful, please reach out to let us know. You can contact Chris at fifty@cs.stanford.edu.

You can also cite our work:

@inproceedings{fifty2023context,
  title={Context-Aware Meta-Learning},
  author={Fifty, Christopher and Duan, Dennis and Junkins, Ronald Guenther and Amid, Ehsan and Leskovec, Jure and Re, Christopher and Thrun, Sebastian},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2023}
}

snenyl/CAML