/h2ogpt

Join us at H2O.ai to make the world's best open-source GPT with document and image Q&A, 100% private chat, no data leaks, Apache 2.0 https://arxiv.org/pdf/2306.08161.pdf Live Demo: https://gpt.h2o.ai/

Primary LanguagePythonApache License 2.0Apache-2.0

h2oGPT

Turn ★ into ⭐ (top-right corner) if you like the project!

Watch these YouTube Videos:

For questions, discussing, or just hanging out, come and join our Discord!

Technical Paper: https://arxiv.org/pdf/2306.08161.pdf

h2oGPT is a large language model (LLM) fine-tuning framework and chatbot UI with document(s) question-answer capabilities. Documents help to ground LLMs against hallucinations by providing them context relevant to the instruction. h2oGPT is fully permissive Apache V2 open-source project for 100% private and secure use of LLMs and document embeddings for document question-answer.

Welcome! Join us and make an issue or a PR, and contribute to making the best fine-tuned LLMs, chatbot UI, and document question-answer framework!

Supported OS and Hardware

GitHub license Linux macOS Windows Docker

GPU mode requires CUDA support via torch and transformers. A 6.9B (or 12GB) model in 8-bit uses 8GB (or 13GB) of GPU memory. 8-bit or 4-bit precision can further reduce memory requirements down no more than about 6.5GB when asking a question about your documents (see low-memory mode).

CPU mode uses GPT4ALL and LLaMa.cpp, e.g. gpt4all-j, requiring about 14GB of system RAM in typical use.

GPU and CPU mode tested on variety of NVIDIA GPUs in Ubuntu 18-22, but any modern Linux variant should work. MACOS support tested on Macbook Pro running Monterey v12.3.1 using CPU mode.

Apache V2 ChatBot with LangChain Integration

  • LangChain equipped Chatbot integration and streaming responses
  • Persistent database using Chroma or in-memory with FAISS
  • Original content url links and scores to rank content against query
  • Private offline database of any documents (PDFs, Images, and many more)
  • Upload documents via chatbot into shared space or only allow scratch space
  • Control data sources and the context provided to LLM
  • Efficient use of context using instruct-tuned LLMs (no need for many examples)
  • API for client-server control
  • CPU and GPU support from variety of HF models, and CPU support using GPT4ALL and LLaMa cpp
  • Linux, MAC, and Windows support

Light mode with soft colors talking to cat image:

Talk to Cat

Dark mode with H2O.ai colors: VectorDB

Apache V2 Data Preparation code, Training code, and Models

  • Variety of models (h2oGPT, WizardLM, Vicuna, OpenAssistant, etc.) supported
  • Fully Commercially Apache V2 code, data and models
  • High-Quality data cleaning of large open-source instruction datasets
  • LoRA and QLoRA (low-rank approximation) efficient 4-bit, 8-bit and 16-bit fine-tuning and generation
  • Large (up to 65B parameters) models built on commodity or enterprise GPUs (single or multi node)
  • Evaluate performance using RLHF-based reward models
Screen.Recording.2023-04-18.at.4.10.58.PM.mov

All open-source datasets and models are posted on 🤗 H2O.ai's Hugging Face page.

Also check out H2O LLM Studio for our no-code LLM fine-tuning framework!

Roadmap

  • Integration of code and resulting LLMs with downstream applications and low/no-code platforms
  • Complement h2oGPT chatbot with search and other APIs
  • High-performance distributed training of larger models on trillion tokens
  • Enhance the model's code completion, reasoning, and mathematical capabilities, ensure factual correctness, minimize hallucinations, and avoid repetitive output
  • Add other tools like search
  • Add agents for SQL and CSV question/answer

Getting Started

First one needs a Python 3.10 environment. For help installing a Python 3.10 environment, see Install Python 3.10 Environment. On newer Ubuntu systems and environment may be installed by just doing:

sudo apt-get install -y build-essential gcc python3.10-dev
virtualenv -p python3 h2ogpt
source h2ogpt/bin/activate

or use conda:

conda create -n h2ogpt -y
conda activate h2ogpt
conda install python=3.10 -c conda-forge -y

Check your installation by doing:

python --version # should say 3.10.xx
pip --version  # should say pip 23.x.y ... (python 3.10)

On some systems, pip still refers back to the system one, then one can use python -m pip or pip3 instead of pip or try python3 instead of python.

TLDR

For MACOS and Windows 10/11 please follow their instructions.

On Ubuntu, after Python 3.10 environment installed do:

git clone https://github.com/h2oai/h2ogpt.git
cd h2ogpt
# fix any bad env
pip uninstall -y pandoc pypandoc pypandoc-binary
# broad support, but no training-time or data creation dependencies
for fil in requirements.txt reqs_optional/requirements_optional_langchain.txt reqs_optional/requirements_optional_gpt4all.txt reqs_optional/requirements_optional_langchain.gpllike.txt reqs_optional/requirements_optional_langchain.urls.txt ; do pip install -r $fil ; done
# Optional: support docx, pptx, ArXiv, etc.
sudo apt-get install -y libmagic-dev poppler-utils tesseract-ocr libreoffice
# Optional: for supporting unstructured package
python -m nltk.downloader all

Place all documents in user_path or upload in UI.

UI using GPU with at least 24GB with streaming:

python generate.py --base_model=h2oai/h2ogpt-oasst1-512-12b --load_8bit=True  --score_model=None --langchain_mode='UserData' --user_path=user_path

UI using CPU

wget https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML/resolve/main/WizardLM-7B-uncensored.ggmlv3.q8_0.bin
python generate.py --base_model='llama' --prompt_type=wizard2 --score_model=None --langchain_mode='UserData' --user_path=user_path

If using OpenAI for the LLM is ok, but you want documents to be parsed and embedded locally, then do:

python generate.py  --inference_server=openai_chat --base_model=gpt-3.5-turbo --score_model=None

and perhaps you want better image caption performance and focus local GPU on that, then do:

python generate.py  --inference_server=openai_chat --base_model=gpt-3.5-turbo --score_model=None --captions_model=Salesforce/blip2-flan-t5-xl

Add --share=True to make gradio server visible via sharable URL. If you see an error about protobuf, try:

pip install protobuf==3.20.0

Development

  • To create a development environment for training and generation, follow the installation instructions.
  • To fine-tune any LLM models on your data, follow the fine-tuning instructions.
  • To create a container for deployment, follow the Docker instructions.
  • To run h2oGPT tests, run pip install requirements-parser ; pytest -s -v tests client/tests

Help

Acknowledgements

Why H2O.ai?

Our Makers at H2O.ai have built several world-class Machine Learning, Deep Learning and AI platforms:

We also built platforms for deployment and monitoring, and for data wrangling and governance:

  • H2O MLOps to deploy and monitor models at scale
  • H2O Feature Store in collaboration with AT&T
  • Open-source Low-Code AI App Development Frameworks Wave and Nitro
  • Open-source Python datatable (the engine for H2O Driverless AI feature engineering)

Many of our customers are creating models and deploying them enterprise-wide and at scale in the H2O AI Cloud:

We are proud to have over 25 (of the world's 280) Kaggle Grandmasters call H2O home, including three Kaggle Grandmasters who have made it to world #1.

Disclaimer

Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.

  • Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
  • Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
  • Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
  • Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
  • Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
  • Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.

By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.

Star History

Star History Chart