Issues
- 2
Can I use a user dictionary?
#65 opened by mumumu09chi - 4
sudachitra and other custom tokenizers no longer compatible with transformers later than 4.34
#66 opened by mingboiz - 0
- 0
Evaluate model with JGLUE
#52 opened by mh-northlander - 0
- 1
- 0
setup.py install is deprecated.
#62 opened by mh-northlander - 0
Add changelog file
#56 opened by mh-northlander - 0
Allow empty line in the vocab file
#53 opened by mh-northlander - 0
Vocabulary file handling
#57 opened by mh-northlander - 1
tokenizer.model_max_length is incorrect
#51 opened by mh-northlander - 0
Update README for pretraing
#44 opened by t-yamamura - 0
- 0
Add to the test for alignments of encoded tokens by `JapaneseBertWordPieceTokenizer`
#42 opened by t-yamamura - 0
- 0
use `pathlib` instead of `os.path`
#41 opened by t-yamamura - 0
Add preprocessing for cleaning up corpus
#23 opened by t-yamamura - 0
Replace SudachiPy with sudachi.rs
#22 opened by t-yamamura - 0
remove slow tokenizer
#28 opened by t-yamamura - 0
use NFKC as preprocessing
#26 opened by t-yamamura - 0
Remove lowercase normalizer
#24 opened by t-yamamura