Yet Another LLM Repository
I am a Machine Learning Engineer/Software Developer exploring LLM for last few months. Here are the different resources on Large Language Model (LLM) which I am using or planning to refer to in future. This list doesn't include academic papers or theoretical courses. It mostly focuses on resources (code, tools, blogs, short videos) which will help an engineer to build applications around LLM. Please note, I am NOT doing AI; I am just building a layer around AI.
Not sure if you should explore LLM? Start with this highly motivational post :The Rise of the AI Engineer
What is LLM/GPT?
- Catching up on the weird world of LLMs ⭐⭐: First thing first. Lot of things have already happened around LLM. This post (or youtube video, if you prefer) by Simon Willson will help you in catching up from the end user perspective.
- State of GPT by Andrej Karpathy - Microsoft Build ' 2023 Talk ⭐⭐: This talk captures most of the progress made from the technical perspective.
- Large language models, explained with a minimum of math and jargon ⭐⭐: If you want understand how language model works (intutively), check this out. No harm, if you skip it.
Getting Started
If you are already familiar with ChatGPT and want to get started with LLM (to build an application around it or to understand its capabilities), you might want to try OpenAI Playground and APIs. On signing up at OpenAI, you will be allocated with certain amount of free quota (to use the service) which is more than sufficient to build your first prototype. Save your API Key.
- Getting Started using OpenAI API ⭐: This will in turn take you to playground. You don't need to write any code. Start here!
- openai-cookbook⭐: Example code for accomplishing common tasks with OpenAI. If you are a developer, this is going to be your go-to place.
If you are located in US/UK, Claude-2 by Anthropic is another option. As of Aug'2023, I have not used Claude, since it's not available in India.
Third option is to use Azure OpenAI. This is only available for Azure Enterprise Customers and Most Valuable Bloggers at this moment. You can request for it here.
Online Courses
- Short Courses by deeplearning.ai ⭐: If you are a developer, these super short courses, will help you get started immediately. Start with the PromptEngineering course, then based on your area of interest, go through other courses. You will need the OpenAI API Keys to try out the tutorials.
- [A Hackers' Guide to Language Models] by Jeremy Howard(https://www.youtube.com/watch?v=jkrNMKz9pWU) - 90 minutes introduction to LLM ⭐
- Full Stack LLM Bootcamp ⭐: A bunch of session recorded in April 2023. Highly recommended.
Prompt Engineering Resources
- GPT best practices by OpenAI ⭐
- Prompt engineering techniques by Azure ⭐
- Prompting guides suggested by openai-cookbook
My suggestion would be not to spend lot of time going through different prompt engineering courses and guides. "GPT best practices by OpenAI" is more than enough. Browse through other resources on the need basis.
Tools
There are a bunch of Open Source libraries, frameworks that have been developed around LLM. Here are few which I have used:
What are different LLMs available and how ?
GPT's by OpenAI are not the only LLMs available. There are other commercial and Open Source models which you might be interested in. Following are the leaderboards to compare the performance of various LLMs:
- Leaderboard by lmsys ⭐
- FastEval ⭐
- Open Source LLM Leaderboard by huggingface ⭐
- MMLU Leaderboard
- AlpacaEval: An Automatic Evaluator for Instruction-following Language Models
Use Cases for LLM
langchain documentation captures most of the popular use cases: link
Use Case : Retrieval-Augmented Generation / Document QA
Probably the most popular application of LLM: Build a ChatBot on your private data. But remember, semantic based search doesn't always work
Blogs:
Courses
- LLM Bootcamp: Augmented Language Models ⭐
- deeplearning.ai: LangChain: Chat with Your Data
- deeplearning.ai: Large Language Models with Semantic Search
Code Examples
- openai-cookbook: Question answering over your data using embedding ⭐
- Question answering using a search API and re-ranking
Tools/Frameworks
- Langchain ⭐
- LlamaIndex ⭐
Case Study
Use Case : Agents
Blog
Courses
Code Examples
- openai-cookbook: How to call functions with chat models
- openai-cookbook: How to build a tool-using agent with LangChain
- Agents Documentation by langchain
Automatic Speech Recognition
- whisper by openai (openai doc, github, model-card)
- Massively Multilingual Speech by facebook research (CC-BY-NC 4.0)
- SeamlessM4T
Evaluation of LLMs/LLM based systems
The way we evaluate a LLM and a system built using LLM are different. Here I am going to focus mostly on evaluation methodologies used for LLM based systems
Blogs/Courses
- Evaluating LLM-based Applications: Workshop by Josh Tobin ⭐
- How to Evaluate, Compare, and Optimize LLM Systems - Weights & Biases ⭐
- Using LLMs To Evaluate LLMs
Tools
- Azure Promptflow: Service to evaluate and track LLM Experiments
- openai-evals: A framework for evaluating LLMs (large language models) or systems built using LLMs as components
- langchain-evals: Evaluation using LangChain as a framework. Make sure to check the examples. Also read the source code for the example prompts
- lang-kit - Library
- langsmith
LLM In Production
- Building LLM applications for production by Chip Huyen ⭐
- Generative AI Strategy ⭐
- All the Hard Stuff Nobody Talks About when Building Products with LLMs
- Blogs from replit: here & there
- Numbers every LLM Developer should know
Azure OpenAI
- Documentation
- E2E solutions using Azure OpenAI and various other services
- Deployment of an Enterprise Azure OpenAI reference architecture
- Bring your own data: Concepts
- Azure Promptflow: Service to evaluate and track LLM Experiments
Interesting Links
- What Is ChatGPT Doing … and Why Does It Work? by Stephen Wolfram
- RLHF: Reinforcement Learning from Human Feedback
- Patterns for Building LLM-based Systems & Products
- What are embeddings?