/Alpaca-3B-Fine-Tuned

In this project, I have provided code and a Colaboratory notebook that facilitates the fine-tuning process of an Alpaca 3B parameter model originally developed at Stanford University. The model was adapted using LoRA to run with fewer computational resources and training parameters and used HuggingFace's PEFT library.

Primary LanguageJupyter NotebookMIT LicenseMIT

Alpaca-3B-Fine-Tuned

Professional work-related project

In this project, I have provided code and a Colaboratory notebook that facilitates the fine-tuning process of an Alpaca 3B parameter model originally developed at Stanford University. The particular model that is being fine-tuned has around 3 billion parameters, which is one of the smaller Alpaca models.

The model uses low-rank adaptation LoRA to run with fewer computational resources and training parameters. We use bitsandbytes to set up and run in an 8-bit format so it can be used on colaboratory. Furthermore, the PEFT library from HuggingFace was used for fine-tuning the model.

Hyper Parameters:

  1. MICRO_BATCH_SIZE = 4 (4 works with a smaller GPU)
  2. BATCH_SIZE = 256
  3. GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
  4. EPOCHS = 2 (Stanford's Alpaca uses 3)
  5. LEARNING_RATE = 2e-5 (Stanford's Alpaca uses 2e-5)
  6. CUTOFF_LEN = 256 (Stanford's Alpaca uses 512, but 256 accounts for 96% of the data and runs far quicker)
  7. LORA_R = 4
  8. LORA_ALPHA = 16
  9. LORA_DROPOUT = 0.05

Credit for Original Model: Xinyang (Young) Geng and Hao Liu from OpenLM Research

Fine-Tuned Model: RyanAir/alpaca-3b-fine-tuned (HuggingFace)