This repository can help to instruct-tune LLaMA (1 & 2), Open LLaMA, RedPajama, Falcon or StableLM models on consumer hardware using QLoRA (Original implementation here). It's mostly based on the original alpaca-lora repo which can be found here. Please note that this has only been tested on following models, but should work with other models. Contributions are welcome!
1. RedPajama
2. StableLM
3. Open LLaMA/LLaMA (1 & 2)
4. Falcon
5. Codegen
6. gpt_bigcode models
-
Install dependencies
pip install -r requirements.txt
-
If bitsandbytes doesn't work, install it from source. Windows users can follow these instructions.
This file contains a straightforward application of QLoRA PEFT to the Open LLaMA / RedPajama / Falcon / StableLM model, as well as some code related to prompt construction and tokenization. PRs adapting this code to support larger models are always welcome.
Example usage:
For Open LLaMa
python finetune.py \
--base_model 'openlm-research/open_llama_3b_600bt_preview' \
--data_path '../datasets/dolly.json' \
--num_epochs=3 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./dolly-lora-3b' \
--lora_r=16 \
--lora_target_modules='[q_proj,v_proj]'
For RedPajama
python finetune.py \
--base_model='togethercomputer/RedPajama-INCITE-Base-3B-v1' \
--data_path='../datasets/dolly.json' \
--num_epochs=3 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./dolly-lora-rp-3b-t1' \
--lora_r=16 \
--lora_target_modules='["query_key_value"]'
For StableLM
python finetune.py \
--base_model='stabilityai/stablelm-base-alpha-3b' \
--data_path='../datasets/dolly.json' \
--num_epochs=3 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./dolly-lora-st-3b-t1' \
--lora_r=16 \
--lora_target_modules='["query_key_value"]'
For Pythia
python finetune.py \
--base_model='EleutherAI/pythia-6.9b-deduped' \
--data_path='../datasets/dolly.json' \
--num_epochs=1 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./dolly-lora-pyt-6b-t1' \
--lora_r=8 \
--lora_target_modules='["query_key_value"]'
For Falcon
python finetune.py \
--base_model='tiiuae/falcon-7b' \
--data_path='../datasets/dolly.json' \
--num_epochs=1 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./dolly-lora-falcon-7b-t1' \
--lora_r=8 \
--lora_target_modules='["query_key_value"]'
For codegen
python finetune.py \
--base_model='Salesforce/codegen-350M-mono' \
--data_path='../datasets/code_alpaca_20k.json' \
--num_epochs=1 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./ca-cg-350m-t1' \
--lora_r=8 \
--lora_target_modules='["qkv_proj"]'
For gpt_bigcode
python finetune.py \
--base_model='bigcode/gpt_bigcode-santacoder' \
--data_path='../datasets/code_alpaca_20k.json' \
--num_epochs=1 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./ca-big_code-santa-t1' \
--lora_r=8 \
--lora_target_modules='["c_proj"]'
We can also tweak our hyperparameters (similar to alpaca-lora):
CUDA_VISIBLE_DEVICES=0,1 python3 finetune.py \
--base_model meta-llama/Llama-2-70b-hf \
--data_path 'otherside_train.json' \
--output_dir './otherside_f' \
--batch_size 128 \
--micro_batch_size 4 \
--num_epochs 3 \
--learning_rate 0.0004 \
--cutoff_len 4096 \
--val_set_size 0 \
--lora_r 16 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--lora_target_modules '[gate_proj, down_proj, up_proj]' \
--train_on_inputs \
--group_by_length
torchrun --nproc_per_node=8 --master_port=3434 finetune.py \
--base_model meta-llama/Llama-2-70b-hf \
--data-path otherside_train.json \
--output_dir ./llama2 \
--batch_size 8 \
--micro_batch_size 1 \
--num_epochs 3 \
--learning_rate 0.0004 \
--cutoff_len 4096 \
--val_set_size 0 \
--lora_r 16 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--lora_target_modules '[gate_proj, down_proj, up_proj]' \
--train_on_inputs False \
--group_by_length True \
--prompt_template_name alpaca \
--lr_scheduler 'cosine' \
--warmup_steps 100 \
--add_eos_token True \
--save_strategy "epoch" \
--fsdp "full_wrap auto_wrap offload" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer'
This file reads the foundation model from the Hugging Face model hub and the LoRA weights from trained peft model, and runs a Gradio interface for inference on a specified input. Users should treat this as example code for the use of the model, and modify it as needed.
Example usage:
For Open LLaMa
python generate.py \
--base_model 'openlm-research/open_llama_3b_600bt_preview' \
--lora_weights './lora-alpaca'
For RedPajama
python generate.py \
--base_model 'togethercomputer/RedPajama-INCITE-Base-3B-v1' \
--lora_weights './dolly-lora-rp-3b-t1/'
For StableLM
python generate.py \
--base_model 'stabilityai/stablelm-base-alpha-3b' \
--lora_weights './dolly-lora-st-3b-t1'
For Pythia
python generate.py \
--base_model 'EleutherAI/pythia-6.9b-deduped' \
--lora_weights './dolly-lora-pyt-6b-t1'
For Falcon
python generate.py \
--base_model 'tiiuae/falcon-7b' \
--lora_weights './dolly-lora-falcon-7b-t1'
For Codegen
python generate.py \
--base_model 'Salesforce/codegen-350M-mono' \
--lora_weights './ca-cg-350m-t1'
For gpt_bigcode
python generate.py \
--base_model 'bigcode/gpt_bigcode-santacoder' \
--lora_weights './ca-big_code-santa-t1'
AemonAlgiz's walkthrough video here
We would like to express our heartfelt gratitude to Meta for releasing LLaMA . Without this pioneering technology, the foundations of projects like Open Llama and Alpaca wouldn't exist. We sincerely appreciate the immense contributions you've made to the field.
Our acknowledgements also extend to the teams behind Open LLaMA, Together Computer, Alpaca and Alpaca LoRA.. You can find more about their excellent work on their respective GitHub repositories:
Lastly, we would like to express our thanks to the developers of QLoRA and bitsandbytes Your efforts have been instrumental in advancing the field, and we're grateful for your contributions. More information about these projects can be found at:
Thank you all for your commitment to innovation and for making these projects possible.