OpenNMT/CTranslate2

Mistral-Nemo not working

Closed this issue · 3 comments

BBC-Esq commented

Mistral-nemo model is converting but not working. Likely related to the issue identified here:

#1743

    results_batch = generator.generate_batch(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: axis 2 has dimension 6144 but expected 7680
minhthuc2502 commented

It will be fixed in the next release

BBC-Esq commented

It will be fixed in the next release

Cool! BTW, I verified that thew new Mistral-Small model does in-fact work. Unfortunately, even running at int8 it spills over into system memory despite my 24GB.

https://huggingface.co/mistralai/Mistral-Small-Instruct-2409

Also, I'd love to try it using AWQ. Any chance the docs can give examples of how to use the new 4-bit mode of ctranslate2? Thanks. Feel free to close the issue whenever.

BBC-Esq commented

I confirmed that this issue is fixed when I converted the model with this change:

https://github.com/OpenNMT/CTranslate2/pull/1785/files

Closing for now unless you think others might want to reference this issue before the next release.