This project demonstrates how to download a model from Hugging Face, convert it to GGUF format, and upload it back to Hugging Face using a Colab notebook.
- Setup
- Downloading the Model
- Cloning llama.cpp Repository
- Converting the Model
- Uploading to Hugging Face Hub
- First, install the necessary libraries:
!pip install huggingface_hub
- Use the huggingface_hub library to download a model snapshot:
from huggingface_hub import snapshot_download
model_id = "Telugu-LLM-Labs/Indic-gemma-2b-finetuned-sft-Navarasa-2.0"
snapshot_download(repo_id=model_id, local_dir="Indic-gemma-2b-finetuned-sft-Navarasa",
local_dir_use_symlinks=False, revision="main")
!ls -lash Indic-gemma-2b-finetuned-sft-Navarasa
- Clone the llama.cpp repository and install its requirements to access conversion tools:
!git clone https://github.com/ggerganov/llama.cpp.git
!pip install -r llama.cpp/requirements.txt
- Convert the downloaded Hugging Face model to GGUF format using the provided conversion script:
!python llama.cpp/convert-hf-to-gguf.py -h
!python llama.cpp/convert-hf-to-gguf.py Indic-gemma-2b-finetuned-sft-Navarasa \
--outfile Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf \
--outtype q8_0
!ls -lash Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf
- convert-hf-to-gguf.py -h: Displays the help message for the conversion script.
- convert-hf-to-gguf.py: Converts the specified Hugging Face model to GGUF format. Parameters:
- --outfile: Specifies the output file name.
- --outtype: Specifies the output type (e.g., q8_0).
- Upload the converted model back to the Hugging Face Hub:
from huggingface_hub import HfApi
import os
# Set the token as an environment variable (recommended for security)
os.environ["HUGGING_FACE_HUB_TOKEN"] = "your_hugging_face_hub_token"
api = HfApi()
model_id = "your-username/Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf"
api.create_repo(model_id, exist_ok=True, repo_type="model") # Use the api object to create the repo
api.upload_file(
path_or_fileobj="Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf",
path_in_repo="Indic-gemma-2b-finetuned-sft-Navarasa-2.0.gguf",
repo_id=model_id,
)
This notebook provides a step-by-step guide to convert a Hugging Face model to GGUF format and upload it back to the Hugging Face Hub. Modify the notebook as needed for your specific models and conversion requirements.