Chip-Tuning is a part of QQ MLLM project and this is the repository for paper Chip-Tuning: Classify Before Language Models Say with code and scripts.
To use and evaluate chip-tuning, you have to install the dependencies by:
pip install -e requirements.txt
And download related datasets and models by excecuting python code files in ./transformer_chips/download
.
An example for downloading the MMLU dataset is:
python ./transformer_chips/download/dataset/download_MMLU.py
This will download the MMLU dataset and save the dataset to ./data/datasets
.
You can also manually download the resources, then put datasets under ./data/datasets
and models under ./model
.
The scripts for experiments in the paper are stored in ./scripts
:
./scripts/benchmark
: the main experiment (Section 4.2);./scripts/benchmark_mllm
: the multimodal model experiments (Section 4.3);./scripts/benchmark_lora
&./scripts/benchmark_lora_chips
: combination with finetuning (Section 4.4);./scripts/benchmark_validation
: chip selection strategy (Section 5.2);./scripts/benchmark/data_influence
: impact of training dataset scale (Section 5.3);./scripts/benchmark_llama3
: Llama3 experiments (Appendix E);
See README.md
under these folders for more details.
We also provide examples of solely evaluating trained chips on certain benchmark or making prediction with selected chips.
See ./scripts/benchmark/misc
for examples.
You can add your custom tasks with two steps:
- Define your classification task with the format in
./data/yaml
; - Implement your own recipe and data process function in
recipe.py
anddata.py
.
You can also directly use the ChipTrainer
class in ./transformer_chips/chip_trainer.py
by passing your own task_dict
, train_dataset
and eval_dataset
;
or use the ChipedTransformer
class in ./transformer_chips/ChipedTransformer.py
to create your own trainer.
Different from classification chips (ChipedTransformer
), the generation chips (ChipedTransformerForGeneration
) currently need further exploration.
Using them could lead to unexpected results.