This is the implementation of the paper "MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models".
base_model
: MeteoRA modelckpt
: the datasets and dataset processing code.eval
: the evaluation results and evaluation code.MoELoRA
: MeteoRA module and adapted PEFT code.
- Install necessary packages:
pip install -r requirements.txt
- Download all JSON format BIG-bench benchmarks, please refer to here.
- Change
bigbench_dataset_dir
path inconfigs/config.yaml
. - Prepare datasets:
cd data
python create_dataset.py --task all
If you just want to create a specific dataset, run:
cd data
python create_dataset.py --task <task_name>
- Prepare composite-n tasks:
python create_composite.py --n <n>
We prepared n=3
, n=5
and n=10
few-shot dataset generating code. Before generating, please ensure that the sub-tasks to composite composite-n task have been included in data/datasets
.
- Prepare LoRA adapters checkpoint and MeteoRA model checkpoint. You can train by yourself or download ours(LlaMA2 and LlaMA3 as base model) by:
python download_ckpt.py
- Change other paths in
configs/config.yaml
. Example:
base_model_path: 'meta-llama3/Meta-Llama-3-8B'
meteora_ckpt_path: 'ckpt/llama3_8b/llama3_8b_meteora/top_2'
adapter_dir: 'ckpt/llama3_8b/llama3_8b_peft'
Running a benchmark with MeteoRA model:
python eval_model.py --task <task_name> --batch_size <batch_size>
For example:
python eval_model.py --task composite_10 --batch_size 4
Note: If you want to run a composite-n task, please set a larger temperature value (self.T
in MoELoRA/layer.py
). As a reference, 15
, 20
and 30
for n=3
, n=5
and n=10
.
Save the evaluation result:
python eval_model.py --task <task_name> --batch_size <batch_size> --save
Debug mode (model output and ground truth will be shown in the console):
python eval_model.py --task <task_name> --batch_size <batch_size> --debug
Running a benchmark with PEFT model:
python eval_model.py --task <task_name> --batch_size <batch_size> --model <adapter_name>
-
Prepare LoRA adapters and corresponding datasets in JSONL format. Ensure each LoRA adapter has a corresponding dataset. Place all LoRA adapters and datasets in their respective folders with matching subfolder names:
- lora_adapters - adapter_name1 - adapter_name2 - ... - datasets - dataset_name1 - dataset_name2 - ...
-
Change file paths in
run_meteora_train_fsdp.sh
. -
Train MeteoRA model:
sh run_meteora_train_fsdp.sh