How to convert to 4bit gptq
c-seeger opened this issue · 1 comments
c-seeger commented
I've seen there is a 4bit gptq version of stablelm and i'm curious if someone could lead me to some resources describing how to convert the current AI model to 4bit gptq. Any hint would be much appreciated.
Ph0rk0z commented
Try to convert with https://github.com/0cc4m/GPTQ-for-LLaMa
It is GPT-neox
CUDA_VISIBLE_DEVICES=0 python gptneox.py "model path" c4 --wbits 4 --save_safetensors mode_name.safetensors
You can try also adding act-order and true-sequential or group size, just don't put act order + group size together.