Not usable on Intel macs
longseespace opened this issue · 9 comments
Overview
I couldn't use this on an Intel Mac. For some reason, it could not load custom models (miniqa or distilbert), the only model that works is NativeEmbeddings
Here is the logs:
2023-08-09 17:28:57.802134+0800 BasicExample [50044:803119] Metal API Validation Enabled
2023-08-09 17:28:59.012154+0800 BasicExample[50044:8031191 [espresso
[Espresso::handle ex plan] exception=Espresso exception: "Invalid state": inner product kernel p.has biases is true, but biases were not found! [Exception from Layer: 24:x_9_cast.
2023-08-09 17:28:59.012343+0800 BasicExample[50044:803119] [coremll Error computing NN outputs -1
2023-08-09 17:28:59.012400+0800 BasicExample[50044:803119] [coreml] Failure in
-executePlan:error:.
Failed to generate a test input vector
I'm totally clueless at this point. Can you give some pointer @ZachNagengast
PS: I'm using a remote mac to test. Happy to give you access for debugging.
EDIT: I did some more testing and looks like the CoreML model can only run on CPU or Neural Engine, not GPU.
I don't understand why the tests all pass with Github Actions, on x86_64 machine 😅
My uneducated guess is that in the tests, it uses CPU only and not GPU (Metal). What do you think? @ZachNagengast
Thanks for the info, investigating.
@longseespace I'm having trouble replicating this on my machine, can you try out the PDFExample and see if it persists? The BasicExample is really pared down and may be missing some stuff needed to run distilbert on it, but the PDFExample uses distilbert by default.
Also any other info on the machine you're using? I'm testing on a 2016 MBP on macOS 13.4.1.
@ZachNagengast I'm going to share the remote mac I used to replicate the issue (via DM on Twitter). It's not the problem with distilbert but any coreml model converted from huggingface (I think). I tested with miniqa and got the same issue.
Follow up: appeared to be an issue with xcode in macOS 12, recommended using macOS 13 or above to build apps with this lib.
@ZachNagengast can you share more info on that? I've been building PDF Pals on macOS 13.
@longseespace It was just that remote mac you sent to test out, it was running macOS 12. The target is ok to be macOS 13 just as long as the underlying operating system is also macOS 13 or above.
Hello @ZachNagengast @longseespace
This error is still present, how to overcome this error or how to fix it
please, help
023-11-08 15:02:34.694158+0200 ChatWithFilesExample[17397:4904410] [coreml] Error computing NN outputs -1
2023-11-08 15:02:34.694214+0200 ChatWithFilesExample[17397:4904410] [coreml] Failure in -executePlan:error:.
Failed to generate query embedding for 'hello'.