bosun-ai/swiftide

HuggingFace models support (mistralrs)

timonv opened this issue · 4 comments

timonv commented

Recently https://github.com/EricLBuehler/mistral.rs released, a really nice solution to, besides mistral, add support for a lot of huggingface models.

By implementing SimplePrompt, existing transformers that prompt should be able to use it right away.

timonv commented

Errors currently, parked for now

jac-cbi commented

@timonv what errors did you get integrating mistral.rs? I've encountered a couple of bugs during testing, and @EricLBuehler has been very responsive

  • 👀1
EricLBuehler commented

@jac-cbi @timonv I would be happy to help integrating mistral.rs :)

timonv commented

@EricLBuehler Thanks for reaching out and love your work! I believe at the time there were multiple things going on, I don't recall exactly what was going on. Back then the only hardware I had to try it out properly was running WSL, and candle was having trouble with the cuda drivers (I think it recently got fixed that they can use dynamically linked).

#70 was done on a much older version of the Swiftide architecture (and mistral.rs), although it might just work with a little moving around. I have some time next week (although EuroRust) to try again.

Have you considered publishing mistral.rs as a crate? It would help with the amount of surport I'd have to provide for different versions.