TypeError: e.split is not a function

[flatsiedatsie]

System Info

V3 Alpha 17

Mac OS
Brave

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

An issue applying template for Llama 3.2 it 3B. Most likely an issue in my code, but just to be safe / in case you want to harden against this.

Config / Tokenizer data: https://huggingface.co/BoscoTheDog/llama_3_2_it_3b_q4_k_m_chunked/tree/main

Reproduction

It's happening consistently. I'll try loading tokenizer.json from another repo.

[flatsiedatsie]

I tried pointing to the ONNX Community repo, but that resulted in a different error.

[flatsiedatsie]

Third time's the charm. alpindale/Llama-3.2-3B-Instruct worked.

[xenova]

This is due to a new tokenizers format for v0.20.0 - where merges are stored as [string, string][] instead of string[]. I fixed it here, and should work for alpha>=18