Generative Artificial Intelligence (Generative AI) has emerged as a transformative force in creating realistic data, images, and content. By leveraging deep learning (DL) techniques, generative AI models learn patterns and structures from large datasets and autonomously produce novel outputs without explicit programming for each possible outcome. Even though these models are usually pre-trained on massive datasets, fine-tuning for specific tasks or performing inference still requires considerable number of computational resources, necessitating strategic optimizations to unlock their full potential.
This tutorial, powered by EuroCC 2, aims to equip participants with a comprehensive understanding of the computational challenges inherent in Generative AI. Moreover, it seeks to present strategies for identifying and mitigating bottlenecks, exclusively leveraging PyTorch’s native features without resorting to additional languages like C++ or CUDA. The tutorial encompasses a diverse array of topics, including hands-on sessions on setting up and optimizing a generic DL pipeline on a supercomputer, an exploration of mainstream Generative AI models, and optimization strategies for inference specifically tailored to Generative AI. Additionally, it investigates advanced subjects such as sparsification and model parallelism for models with a large number of parameters. By the end of this tutorial, participants will be capable at navigating the challenges of accelerating Generative AI using the powerful tools and techniques inherent in PyTorch.
- Chris Stylianou
- Research Engineer, The Cyprus Institute, EuroCC Cyprus
- Robert Jan Schlimbach
- Technical Advisor ML/HPC, SURF BV, EuroCC Netherlands
- Ivan Gentile
- Data Scientist, IFAB, EuroCC Italy
- Charalambos Chrysostomou
- Research Scientist, The Cyprus Institute, EuroCC Cyprus
- Constantine Dovrolis
- Professor; Director of CaSToRC, The Cyprus Institute, EuroCC Cyprus
- Boris Velichkov
- Researcher, Sofia University, EuroCC Cyprus
| Time Slot | Topic | Presenter(s) |
|---|---|---|
| 09:00 – 09:30 | Introduction and setup | Boris Velichkov |
| 09:30 – 10:15 | Efficient Scaling of Machine Learning Models with PyTorch: Distributed Learning | Charalambos Chrysostomou |
| 10:30 – 11:00 | PyTorch and Profiling | Robert Jan Schlimbach, Boris Velichkov |
| 11:00 – 11:30 | Coffee Break | N/A |
| 11:30 - 11:45 | A gentle introduction to LLMs and LLaMa | Ivan Gentile |
| 11:45 – 12:30 | Optimizing Llama: Enhancing Efficiency and Scalability of Large Language Models with PyTorch | Chris Stylianou |
| 12:30 – 13:00 | An overview of methods for efficient generative AI training & inference | Constantine Dovrolis |