wnma3mz/Together-LLM

Together-LLM

Build your personal language assistant and co-evolve with others assistants.

LLM

Model Architecture

Encoder

• Bert
• RoBERTa
• DistilBERT
• ALBERT
• DeBERTa
• Longformer

Decoder

• GPT

• GPT-J

• OPT

• BLOOM

• llama

Encoder-Decoder

• BART
• T5
• Pegasus

Dataset

Unlabelled

Labelled

Deployment

Quantification

• llama.cpp

API Deployment

• llama.cpp2python
• chatgpt-web
• Gradio

Processing

• Pre-processing

• Post-processing

Together

Training

Fine-tuning

Dataset

• self-instruct
• Belle & Github

Method

• PEFT

  • LoRA: LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
  • Prefix Tuning: Prefix-Tuning: Optimizing Continuous Prompts for Generation, P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
  • P-Tuning: GPT Understands, Too
  • Prompt Tuning: The Power of Scale for Parameter-Efficient Prompt Tuning

• Self-Instruct

  • Self-Instruct: Aligning Language Model with Self Generated Instructions

• FLAN

  • Scaling Instruction-Finetuned Language Models

Evolution

• together
• nebullvm
• Distributed Learning
• Federated Learning

Reference

• https://huggingface.co/

Reading List

• What Is ChatGPT Doing … and Why Does It Work?
• A simple Python implementation of the ReAct pattern for LLMs
• Language Models are Changing AI: The Need for Holistic Evaluation