How to support _Falcon-40b-Instruct.ggmlv3.q4_0.bin_ ?

[benoit-cty]

Thanks for this tool !

How can I use it with Falcon-40b-Instruct.ggmlv3.q4_0.bin ?

I can run Falcon-40b-Instruct.ggmlv3.q4_0.bin locally with:

git clone https://github.com/jploski/ggml falcon-ggml
cd falcon-ggml
git checkout falcon40b
mkdir build && cd build && cmake .. && cmake --build . --config Release
bin/falcon -m ~/work/Falcon-40b-Instruct.ggmlv3.q4_0.bin -t 10 -n 200 -p "Hello AGI"

[CSnowRules]

That should be possible via CTransformers, which is currently broken. Please close this issue and refer to issue 309.

[benoit-cty]

Thanks, waiting for #309

[ParisNeo]

All falcon models are now supported on lollms-webui through ctransformers bindings. You can either install the models I have put in the ui or just use the url install if it is not on the zoo. Just grub the link to the .bin file from a repo like @TheBloke's hugging face, paste it in url and press install.
The ctransformers binding supports loads of models. It is my favorite.

[benoit-cty]

Yes, it works, thanks ! But seems too slow for the UI : #310

[ParisNeo]

If you use 40B, it is big for most gpus so many parts are handled by cpu. Use the 7B model. With gpu support that works fast.