/training-fine-tuning-large-language-models-workshop-dhs2024

This repository will contain all the presentations, content, hands-on notebooks for a full day Generative AI workshop on Training, Fine-tuning Large Language Models for the DataHack Summit 2024 conference.

Primary LanguageJupyter NotebookGNU General Public License v3.0GPL-3.0

Mastering Large Language Models: Training and Fine-Tuning Large Language Models Workshop (DHS 2024)

This repository contains all the presentations and hands-on code notebooks for the "Training and Fine-Tuning Large Language Models" workshop held at Analytics Vidhya DataHack Summit 2024. The workshop is divided into several modules, each focusing on different aspects of working with large language models (LLMs) from prompting to fine-tuning

Workshop instructor: Dipanjan (DJ) Sarkar


Tools and Frameworks Used





Workshop Overview

This workshop is divided into 5 modules, the following graphic clearly showcases what is covered in each module. It is recommended to navigate sequentially through each module to get the best experience. Also do go through the presentation decks to learn the conceptual aspects of the topics besides trying out the hands-on python notebooks.

Note: For environment setup each module has its own requirements.ipynb notebook with all the necessary libraries needed to be installed. We use Runpod.io A40 Servers which give us a 48GB VRAM GPU which is enough for these experiments. Make sure you have a separate disk volume of at least 30GB to store the Llama 3 LLM weight files after fine-tuning, else you will get an error.

)


Modules Overview

Module 01: Transformers, LLMs, and Generative AI Essentials


This module serves as the foundational introduction to Transformers, Foundation models, and fine-tuned Large Language Models (LLMs). You'll learn about both simple and contextual embedding models and their applications in real-world scenarios. Practical exercises include building a custom search engine and leveraging Transformer-based LLMs for tasks such as sentiment analysis, Q&A, summarization, and zero-shot classification. A special focus is given to using Microsoft Phi-3 Mini Instruct locally for prompt engineering and comparing performance across models like GPT-4o mini and Meta Llama 3.1 8B.




Module 02: Pre-training and Fine-tuning LLMs


This module dives deep into the processes of pre-training and fine-tuning LLMs. You'll explore the key steps for building LLMs or SLMs (Small Language Models) from scratch, with hands-on experience in customizing models like GPT-2 through pre-training on unlabeled datasets. The module also covers the process of full fine-tuning of pre-trained BERT models for specific tasks like text classification, helping you understand the key workflows of adapting LLMs to specialized applications.


Module 03: Parameter Efficient Fine-tuning of LLMs


In this module, you'll learn about Parameter Efficient Fine-Tuning (PEFT) techniques, which allow you to fine-tune large language models (LLMs) with fewer resources. The module includes practical exercises on fine-tuning a BERT model using PEFT methods like QLoRA (Quantized Low-Rank Adaptation) for text classification and named entity recognition. You'll also explore advanced methods like merging and switching LoRA adapters on the same model, making your fine-tuning efforts more flexible and efficient.


Module 04: Instruction Fine-tune LLMs with Supervised Fine-tuning

This module deep dives into instruction-based fine-tuning of LLMs by leveraging Supervised fine-tuning. You'll start by working on a simple, yet challenging task of instruct fine-tuning a TinyLlama 1B SLM for text-to-SQL operations. Then you will work step-by-step on building a custom fine-tuned RAG pipeline. You will start by leveraging LLMs and prompting to prepare your own labeled datasets for Retrieval-Augmented Generation (RAG). Next you will work on fine-tuning your own embedder model using your prepared dataset. Then you will fine-tune a Llama 3 LLM on your labeled dataset for RAG Q&A response generation using instruction tuning and PEFT. Finally you will leverage all these components and build your own end-to-end custom fine-tuned RAG pipeline.


Module 05: Aligning Fine-tuned LLMs to Human Preferences - RLHF & Beyond

The final module focuses on aligning fine-tuned LLMs with human preference data using advanced reinforcement learning alignment and reinforcement learning free alignment techniques. You'll explore Reinforcement Learning with Human Feedback (RLHF) and Proximal Policy Optimization (PPO) by aligning LLMs like GPT-2 to generate positive content. The module also covers RL free methods like Direct Preference Optimization (DPO) and Odds-Ratio Preference Optimization (ORPO) to align LLMs like Llama 3 with human preference data, ensuring that your models are helpful, harmless and algined to human preferences.


Citation

@misc{dipanjan-sarkar-mastering-llms-workshop-2024,
  author = {Dipanjan (DJ) Sarkar},
  title = {Mastering Large Language Models: Training and Fine-Tuning Large Language Models Workshop},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/dipanjanS/training-fine-tuning-large-language-models-workshop-dhs2024}}
}

References and Acknowledgements