awesome-LLM-toolkit
Collect information about all the necessary tools and dataset required for building a high-quality customized LLM
Core Model/Training Techniques
- FlashAttention: Github, Paper
- ALiBi: Paper
- Faster Inference
- FasterTransformer: Github
- bitsandbytes: 8-bit optimizer, Github
- PEFT: Github, LoRA Paper
- QLoRA: GPU memory efficient fine-tuning by training LoRA on quantized models
- BLIP-2/LAVIS: Connect image encoder with LLM to enable multi-modal, Paper
- RLHFs:
Data
Name | Type | Language | License | Note |
---|---|---|---|---|
mc4 | Raw Text | multilingual (100+) | ODC-By | |
bloom | Raw Text | multilingual (46) | Depend on data | |
wmt22 | translation | multilingual | Depend on data | |
RedPajama | Raw Text | mostly EN | Apache 2.0 | |
WuDaoCorpora | Raw Text | zh-CN | 5TB | |
The Stack | Code | |||
The Flan Collection | Instruction | |||
openwebtext | Raw Text | EN | CC0 1.0 | used to train GPT-2 |
self-instruct-seed | Instruction | EN | Apache 2.0 | |
Stanford Alpaca | Instruction | EN | CC BY-NC 4.0 | |
Alpaca Cleaned | Instruction | EN | CC BY-NC 4.0 | |
ShareGPT Vicuna | Instruction | EN | 🤐 | Collected from sharegpt |
evol_instruct_70k | Instruction | EN | CC BY-NC ? | Generated by Evol-Instruct |
HH-RLHF | RLHF | EN | MIT | |
databricks-dolly-15k | Instruction | EN | CC BY-SA 3.0 | |
GuanacoDataset | Instruction | EN, zh-CN, zh-TW, JA, DE | GPL 3.0 | desgined for multilingual |
dolly_hhrlhf | Instruction | EN | CC BY-SA 3.0 | MosaicAI's filtered version of HH-RLHF and databricks-dolly-15k |
HC3, Chinese | RLHF | EN, CN | CC BY-SA 3.0 | Paper, Github |
Lamini | Instruction | EN | CC-BY-4.0 | |
the_pile_books3 | Raw Text | mostly EN | MIT | part of the pile |
CoNaLa | Coding | EN | MIT | |
GPTeacher | Instruction | EN | MIT | Generated by GPT-4 |
Alpaca-CoT | Instruction | EN, CN | Apache 2.0 | Collection of instruction datasets |
OpenAssistant/oasst1 | Conversation | multilingual | Apache 2.0 | Paper, Collected from Open Assistant |
Others
- Stack Exchange
- wikihow
- Pushshift Reddit API
- awesome-legal-nlp
- Papers on data-selection / instruction fine-tuning
Evaluation
Serving
- huggingface/text-generation-inference: production LLM inference server
Community
Others
- LMSYS Arena & Leaderboard
- text-generation-webui
- LlamaChat MacOS Client
- llama.cpp