/ggify

Tool to download models from Huggingface Hub and convert them to GGML/GGUF for llama.cpp

Primary LanguagePythonMIT LicenseMIT

ggify

A small tool that downloads models from the Huggingface Hub and converts them into GGML for use with llama.cpp.

Usage

  • Download and compile llama.cpp.
  • Set up a virtualenv using the requirements from llama.cpp.
  • Install the requirements from this repo in that virtualenv.
  • Run e.g. python ggify.py databricks/dolly-v2-12b (nb.: I haven't tried with that particular repo)
  • You'll end up with GGML models under models/....

You can set --llama-cpp-dir (or the LLAMA_CPP_DIR environment variable) to point to the directory where you've compiled llama.cpp.