huggingface/tokenizers

Precompiled: Error("invalid type: null, expected a borrowed string", line : 1, column: 28)

vicantwin opened this issue · 2 comments

use tokenizers::Tokenizer;

fn main() {
    let _tokenizer = Tokenizer::from_file("tokenizer.json");
}
thread 'main' panicked at /home/vi/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.20.0/src/normalizers/mod.rs:143:26:
Precompiled: Error("invalid type: null, expected a borrowed string", line: 1, column: 28)

tokenizer.json is from Xenova/speecht5_tts

Not sure what's causing this. This model works flawlessly on Transformers.js so it isn't a problem with the tokenizer.json

Seems like its a duplicate of #1627.

Closing...

Thanks for reporting will have a look!