Can I use the SentencePiece Model file?

Question

Can I use the SentencePiece Model file?

tylike opened this issue a year ago · 1 comments

hi!

I have an LLM model in Onnx format and a sentencepiece.model file, and I have used HuggingFace and SentencePiece in Python. Now I plan to do inference in C# + OnnxRuntime, but I haven't found a suitable version of the SentencePiece library for C#. Can I use the SentencepieceTokenizer with SentencePiece compatibility in C#?

thanks!

Answer 1 · 2023-06-29T16:00:13.000Z

I believe you can use https://github.com/microsoft/BlingFire.