[BUG] 🐛 - Missing a tokenizer. Probably need to add Dart bindings for HF tokenizer?

[BrutalCoding]

Describe the bug
I couldn't wrap my head around last couple of days on what's missing in order to have a conversational AI. A short while ago, I added the AI evaluation feature and successfully performed an eval on a 3B model. In other words, I was able to load it in and run a test against it.

Great, eval done. Chat can be done in less than 15 minutes of work. Right? Yes but no. You see, the problem is that I'm missing an important component: a tokenizer. I've been too blind to notice this until today thanks to a good friend of mine.

Now, that's the theory at least. Will put this to practice and see if I can finish it this week.

PS. I'm not an AI expert, more of an AI noob that's good enough to grasp concepts and simply able to read instructions to run them.

[BrutalCoding]

Closing now because my recent changes includes llama.cpp directly which comes with a tokenizer built-in.

Hooray!