/h2ogpt

Private Q&A and summarization of documents+images or chat with local GPT, 100% private, Apache 2.0. Supports LLaMa2, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://codellama.h2o.ai/

Primary LanguagePythonApache License 2.0Apache-2.0

h2oGPT

Turn ★ into ⭐ (top-right corner) if you like the project!

Query and summarize your documents or just chat with local private GPT LLMs using h2oGPT, an Apache V2 open-source project.

  • Private offline database of any documents (PDFs, Excel, Word, Images, Code, Text, MarkDown, etc.)
    • Persistent database (Chroma, Weaviate, or in-memory FAISS) using accurate embeddings (instructor-large, all-MiniLM-L6-v2, etc.)
    • Efficient use of context using instruct-tuned LLMs (no need for LangChain's few-shot approach)
    • Parallel summarization and extraction, reaching an output of 80 tokens per second with the 13B LLaMa2 model
    • HYDE (Hypothetical Document Embeddings) for enhanced retrieval based upon LLM responses
  • Variety of models supported (LLaMa2, Mistral, Falcon, Vicuna, WizardLM. With AutoGPTQ, 4-bit/8-bit, LORA, etc.)
    • GPU support from HF and LLaMa.cpp GGML models, and CPU support using HF, LLaMa.cpp, and GPT4ALL models
    • Attention Sinks for arbitrarily long generation (LLaMa-2, Mistral, MPT, Pythia, Falcon, etc.)
  • UI or CLI with streaming of all models
    • Upload and View documents through the UI (control multiple collaborative or personal collections)
    • Bake-off UI mode against many models at same time
    • Easy Download of model artifacts and control over models like LLaMa.cpp through the UI
    • Authentication in the UI by user/password
    • State Preservation in the UI by user/password
  • Linux, Docker, macOS, and Windows support
  • Inference Servers support (HF TGI server, vLLM, Gradio, ExLLaMa, Replicate, OpenAI, Azure OpenAI)
  • OpenAI-compliant Python client API for client-server control
  • Web-Search integration with Chat and Document Q/A
  • Agents for Search, Document Q/A, Python Code, CSV frames (Experimental, best with OpenAI currently)
  • Evaluate performance using reward models
  • Quality maintained with over 1000 unit and integration tests taking over 4 GPU-hours

Getting Started

GitHub license Linux macOS Windows Docker

To quickly try out h2oGPT on CPU with limited document Q/A capability using LLaMa2 7B Chat, create a fresh Python 3.10 environment and run:

git clone https://github.com/h2oai/h2ogpt.git
cd h2ogpt
pip install -r requirements.txt
pip install -r reqs_optional/requirements_optional_langchain.txt
pip install -r reqs_optional/requirements_optional_gpt4all.txt
python generate.py --base_model='llama' --prompt_type=llama2 --model_path_llama=https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q8_0.bin --max_seq_len=4096

then go to your browser by visiting http://127.0.0.1:7860 or http://localhost:7860. Choose 13B for a better model than 7B. If you encounter issues with llama-cpp-python or other packages that try to compile and fail, try binary wheels for your platform as linked in the detailed instructions below.

We recommend quantized (e.g. GGML) models for most small-GPU systems, e.g. LLaMa-2-7B-Chat-GGML for 9GB+ GPU memory or larger models like LLaMa-2-13B-Chat-GGML if you have 16GB+ GPU memory.

Windows 10/11 64-bit with full document Q/A capability

  • One-click Installers Oct 06, 2023:

    Aug 19, 2023:

    The installers include all dependencies for document Q/A except for models (LLM, embedding, reward), which you can download through the UI. After installation, go to start and run h2oGPT, and a web browser will open for h2oGPT. To use LLaMa model, go to Models tab, select llama base model, then click load to download from preset URL. Then use as normal. To terminate the app, in task manager kill the Python process named pythonw.exe as will also show up in nvidia-smi if using GPUs. Set environment variables (in system properties->advanced->environment variables) to control things:

    • n_jobs: number of cores for various tasks
    • OMP_NUM_THREADS thread count for LLaMa
    • CUDA_VISIBLE_DEVICES which GPUs are used. Recommend set to single fast GPU, e.g. CUDA_VISIBLE_DEVICES=0 if have multiple GPUs. Note that UI cannot control which GPUs (or CPU mode) for LLaMa models.
    • Any CLI argument from python generate.py --help with environment variable set as h2ogpt_x, e.g. h2ogpt_h2ocolors to False.
    • Set env h2ogpt_server_name to actual IP address for LAN to see app, e.g. h2ogpt_server_name to 192.168.1.172 and allow access through firewall if have Windows Defender activated.

  • Windows 10/11 Manual Install and Run Docs

Linux (CPU/CUDA) with full document Q/A capability

macOS (CPU/M1/M2) with full document Q/A capability

Example Models

CPU mode uses GPT4ALL and LLaMa.cpp, e.g. gpt4all-j, requiring about 14GB of system RAM in typical use.

Live Demos

Inference Benchmarks for Summarization & Generation

Resources

Partners

Video Demo

demo2.mp4

YouTube 4K version: https://www.youtube.com/watch?v=_iktbj4obAI

Docs Guide

Experimental features

These are not part of normal installation instructions and are experimental.

  • SerpAPI web search augmentation -- in Beta testing. Working well but hits file closed failure under intense use.
  • Agents -- in Alpha testing. Optimal for OpenAI, but that also fails sometimes.

Roadmap

  • Integration of code and resulting LLMs with downstream applications and low/no-code platforms
  • Complement h2oGPT chatbot with other APIs like ToolBench
  • Enhance the model's code completion, reasoning, and mathematical capabilities, ensure factual correctness, minimize hallucinations, and avoid repetitive output
  • Add better agents for SQL and CSV question/answer

Development

  • To create a development environment for training and generation, follow the installation instructions.
  • To fine-tune any LLM models on your data, follow the fine-tuning instructions.
  • To run h2oGPT tests: 
    wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q8_0.bin
pip install requirements-parser pytest-instafail
pytest --instafail -s -v tests
# for client tests
make -C client setup
make -C client build
pytest --instafail -s -v client/tests
    or tweak/run tests/test4gpus.sh to run tests in parallel.

Help

Acknowledgements

Why H2O.ai?

Our Makers at H2O.ai have built several world-class Machine Learning, Deep Learning and AI platforms:

We also built platforms for deployment and monitoring, and for data wrangling and governance:

  • H2O MLOps to deploy and monitor models at scale
  • H2O Feature Store in collaboration with AT&T
  • Open-source Low-Code AI App Development Frameworks Wave and Nitro
  • Open-source Python datatable (the engine for H2O Driverless AI feature engineering)

Many of our customers are creating models and deploying them enterprise-wide and at scale in the H2O AI Cloud:

We are proud to have over 25 (of the world's 280) Kaggle Grandmasters call H2O home, including three Kaggle Grandmasters who have made it to world #1.

Disclaimer

Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.

  • Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
  • Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
  • Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
  • Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
  • Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
  • Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.

By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.

Star History

Star History Chart