Efficient Alpaca

English | 中文

The aim of Efficient Alpaca is to utilize LLaMA to build and enhance the LLM-based chatbots, including but not limited to reducing resource consumption (GPU memory or training time), improving inference speed, and more facilitating researchers' use (especially for fairseq users). This project will be constantly updated and maintained. Please feel free to use it!

**************************** Updates ****************************

4/5 We support Fine-tuning using FSDP to reduce GPU memory with extra RAM memory !
3/17 We support model parallel to reduce GPU memory using Megatron-LM !
3/15 We support LoRA (Efficient-finetuning) to reproduce Stanford Alpaca !

Supported Inference Devices

we can choose following device to support inference, even 12G 1080.

Method	Support	Device	GPU	Inference Speed
Original	all	1 24G 3090	14G
Megatron	megatron_lora	2 12G 1080	8G

Supported Training Methods and Devices

We can choose the following available methods and combinations: for example, I have 2 24G 3090 and a lot of memory, at this time you can have two options: 1. Use Megatron-LM for Efficient-Finetuning (does not use a lot of memory) . 2. Use FSDP for Fine-tuning (it will use a lot of extra memory).

Method	Type	Support	Data Para	Model Para	Device	GPU	Memory Limit	Training Speed
LoRA	Efficient Fine-tuning	lora	✓	✓	1 40G A100	30G	No	90 sec / 100 step
Megatron-LoRA	Efficient Fine-tuning	megatron_lora	✗	✓	2 24G 3090	21G	No	190 sec / 100 step
FSDP	Fine-tuning	fsdp	✓	✓	1 40G A100	32G	128G +	1600 sec / 100 step
					8 40G A100	32G	No	400 sec / 100 step
					2 24G 3090	13G	128G +	900 sec / 100 step
					8 24G 3090	22G	128G +	800 sec / 100 step
Megatron	Fine-tuning	megatron	✗	✓	4 40G A100	25G	No	130 sec / 100 step
					8 24G 3090	14G	No	130 sec / 100 step

Some Explanation about Support Table.

All evaluation used hyper-parameter --max-tokens 2048.

Data Para: Whether to support data parallel.
Model Para: Whether to support model parallel.
GPU: GPU Memory usage of each node during training.
Memory Limit: RAM Memory usage during training.
Training Speed: Only represents training speed rather than training time, because data parallel supports to accelerate training.

Web Interface

We support web interface using Gradio。

bash  alpaca_lora/scripts/run_webapp.sh

Setup

Ensure the pytorch and cuda environment available, and install fllowing dependences:

pip install fairseq
pip install fairscale

We have to install sentencepiece from official repo to process data or hack for your specific task.

git clone https://github.com/google/sentencepiece.git 
cd sentencepiece
mkdir build
cd build
cmake ..
make -j $(nproc)
make install

Prepare Model and Data

Data and Model Preprocess

Training Step:

Efficient-Finetuning

Fine-tuning

Some Case Sampled by Our Model:

Question: what is the capital of china?

Answer: Beijing is the capital of China.

Question: please give a pieces of python code about socket connection.

Answer:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print("Constructing a socket connection")
print(s)

s.bind(('', 5000)) # Bind the socket to the '(' and 5000'

print("Listening for connections")
while True:
    connection, address = s.accept()
    print("Got connection from", address)
    print("Connected!")
    connection.close() # Close the connection.

print("Exiting the program")

Question: 请讲一个武侠故事.

Answer: The legendary warrior Guan Yu Once upon a time, there was a legendary warrior named Guan Yu. He was a highly skilled fighter and commander and was known for his great bravery and loyalty. He was the loyal servant of Emperor Liu Bei, and fought in the Three Kingdoms civil war alongside him. He was brave enough to challenge the powerful enemy warriors, and eventually helped Liu Bei's army defeat the Wei Kingdom, the strongest of the three kingdoms. Guan Yu was a great warrior and a hero in Chinese history.
Question: what is an alpaca? how is it different form a llama?

Answer: Alpacas and llamas are both members of the Camelidae family. They are both herbivorous mammals that are domesticated for their coats, but they are distinctly different species. Llamas are generally larger than alpacas and have more pronounced spiral horns, while alpacas have smaller horns and more delicate coats. Alpacas also have smaller teeth and a narrower neck than llamas, and they are able to reproduced year-round. Llamas are also unable to reproduce during the winter months and have a more social behavior than alpacas.

Question: write a e-mail to congratulate Lee and mention that you are exited about meeting all of them in person.

Answer:

Dear Lee,

I want to congratulate you on your success and I'm looking forward to meeting you and the other team members in person. 
It's exciting to see the progress you've made and I'm sure the hard work you have put in has paid off. 
I'm very proud of you and I'm looking forward to meeting you in person and seeing all of your progress.

Sincerely,
[Your Name]

Question: write a well-thought our abstract for a machine learning parper that proves that 42 is the optimal seed for training neural network.

Answer: This paper investigates the effect of different seeds on the performance of neural networks. Through experiments, we demonstrate that the optimal seed is 42, which leads to the best performance on a wide range of tasks. We analyze the effects of different seeds on the model's accuracy, robustness, and other metrics. Our results show that 42 is the best seed for neural networks, and we discuss the implications of this finding.

Reference

Fairseq: a sequence modeling toolkit [github]

@inproceedings{ott2019fairseq,
  title = {fairseq: A Fast, Extensible Toolkit for Sequence Modeling},
  author = {Myle Ott and Sergey Edunov and Alexei Baevski and Angela Fan and Sam Gross and Nathan Ng and David Grangier and Michael Auli},
  booktitle = {Proceedings of NAACL-HLT 2019: Demonstrations},
  year = {2019},
}

FairScale: is a PyTorch extension library for high performance and large scale training. [github]

@Misc{FairScale2021,
  author =       {{FairScale authors}},
  title =        {FairScale:  A general purpose modular PyTorch library for high performance and large scale training},
  howpublished = {\url{https://github.com/facebookresearch/fairscale}},
  year =         {2021}
}

LLaMA: Open and Efficient Foundation Language Models [paper][github]

@article{touvron2023llama,
  title={LLaMA: Open and Efficient Foundation Language Models},
  author={Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth{\'e}e and Rozi{\`e}re, Baptiste and Goyal, Naman and Hambro, Eric and Azhar, Faisal and Rodriguez, Aurelien and Joulin, Armand and Grave, Edouard and Lample, Guillaume},
  journal={arXiv preprint arXiv:2302.13971},
  year={2023}
}

Stanford Alpaca: An Instruction-following LLaMA model [github]

@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}

dropreg/efficient_alpaca