WorksApplications/SudachiTra

tokenizer.model_max_length is incorrect

mh-northlander opened this issue · 1 comments

mh-northlander commented

The parameter model_max_length of chitra tokenizer seems too large.

> sudachitra.sudachitra.BertSudachipyTokenizer.from_pretrained("chiTra-1.0").model_max_length
1000000000000000019884624838656
mh-northlander commented

Adding "model_max_length": 512 to the tokenizer_config.json will solve this problem.