Command R+

[severian42]

Awesome work with this project, it's so great to have when testing models. I was trying to load the Command R+ with the latest version and am getting this:

File "/Users/anima/mlx-ui/venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 584, in _run_script
    exec(code, module.__dict__)
File "/Users/anima/mlx-ui/app.py", line 56, in <module>
    supports_system_role = "system role not supported" not in chat_template.lower()

Thought I could figure it out but I just keep making the app messier. Any thoughts? Thanks!!

[da-z]

Hi. Glad you find the app useful.

Hmm. I don't remember having any issues on that line when I tried Command R+

Is that the whole error? Could you paste the full thing?

Also, does the model work fine from command line?

python3 -m mlx_lm.generate --model mlx-community/c4ai-command-r-plus-4bit --max-tokens 100 --prompt "the capital of France is"

[severian42]

Thanks for the quick reply! Yeah, that is the whole error I get after the model loads from the cache. I just tried the script you provided, it seems to run from the command line just fine. Strange. I'll just rebuild everything from scratch again and try, could just be an odd thing my env did or something

Also, side note, I discovered this little hack through HF that speeds it up a bit

sudo sysctl iogpu.wired_lwm_mb=100000

Thread: https://huggingface.co/mlx-community/c4ai-command-r-plus-4bit/discussions/2#6613dfdebf1904adf1ef89b9

[da-z]

If still does not work, try replacing that line locally with

'supports_system_role = True'

[severian42]

Awesome, thanks for the help! Very much appreciated.