huggingface/swift-chat

Cannot import Mistral7b

juliusmlr opened this issue · 9 comments

juliusmlr commented

On my Macbook Air m3 with MacOS 15.1 (24B83) i cannot import the mistral model downloaded from this repo: https://huggingface.co/apple/mistral-coreml

I'm running the preview branch.
This is the XCode error i'm running in:

.bundle/main/main_mps_graph/main_mps_graph.mpsgraphpackage/model_0.mpsgraph":0:0): error: attempting to parse a byte at the end of the bytecode
Models/LanguageModel.swift:39: Fatal error: Cannot obtain shape information

Any idea?

rudrankriyam commented

Facing the same issue

kokluch commented

same

joaomelo81 commented

Switch the branch to main

OneMHz commented

If you look at model.modelDescription.inputDescriptionsByName, it contains key = inputIds. But the code is looking for input_ids. But if you fix that, then it complains about the causal mask because that's not added as an MLFeatureValue. And if you fix that, it complains that mistral needs an MLState (though, not in the most clear message), which is only iOS 18+/macOS15+. And if you fix that, it complains that the shape.count = 0 != stride.count = 3 or something (I had to update to the latest and need to re-apply some local changes to get the exact error back).

But I'm kind of naively fixing the error messages locally and hoping for the best so some guidance would be great.

pcuenca commented

Hi @OneMHz, are you using the preview branch?

OneMHz commented

No. Main. I did manage to code my way to getting a result. But it’s coming back as a MLMultiArray so that doesn’t work for the return type

pcuenca commented

You need the preview branch of this project, which also pulls the preview branch from swift-transformers. It adds supports for MLState and a few other things. Please, let us know if that works!

pcuenca commented

Sorry about that, we should unify soon!

OneMHz commented

Thanks! That is working much better!

  • 👍1