huggingface/tokenizers

Issues

• Strange warnings with tokenizer for some models

  #1528 opened 2 days ago by EricLBuehler
  1

• Converting `tokenizers` tokenizers into `tiktoken` tokenizers

  #1530 opened 5 days ago by umarbutler
  2

• How to Batch-Encode Paired Input Sentences with Tokenizers: Seeking Clarification

  #1531 opened 4 days ago by insookim43
  0

• Extended vocab tokenizer merging text into a single string without spaces while decoding

  #1501 opened a month ago by savanth14
  3

• Train tokenizer on integer lists, not strings

  #1471 opened 2 months ago by rteehas
  4

• Bug with `CodeQwen1.5`: `data did not match any variant of untagged enum PyPreTokenizerTypeWrapper`

  #1529 opened 5 days ago by QwertyJack
  0

• error: casting `&T` to `&mut T` is undefined behavior

  #1485 opened 7 days ago by Jipok
  3

• Deepseeker model completely loses performance after using tokenizer.add_tokens(special_tokens)

  #1490 opened a month ago by bin123apple
  1

• Is it possible to pass a tokenizer from Python into Rust?

  #1487 opened 7 days ago by albertsgarde
  1

• Special token handling breaks idempotency of sentencepiece due to extra spaces

  #1527 opened 8 days ago by cat-state
  0

• Link to download the training text in `docs/source/quicktour.rst` is broken

  #1526 opened 9 days ago by 14jdelap
  0

• How to write custom Wordpiece class?

  #1525 opened 9 days ago by xinyinan9527
  0

• Convert huggingface tokenizer into sentencepiece format

  #1524 opened 10 days ago by RRaphaell
  0

• Loading `tokenizer.model` with Rust API

  #1518 opened 19 days ago by EricLBuehler
  4

• Breaking changes in v0.19.1 for tiktoken/llama3

  #1512 opened 24 days ago by sanderland
  3

• ❓Get stats (e.g. counts) about the merged pairs

  #1523 opened 11 days ago by pietrolesci
  0

• BPE Trainer doesn't respect the `vocab_size` parameter when dataset size is increased

  #1514 opened 23 days ago by Abhinay1997
  1

• Discrepancy Between GitHub Release and NPM Package Version & Missing Dependencies

  #1489 opened a month ago by superBertBerg
  2

• Error: Cannot find module 'tokenizers/bindings/tokenizer'

  #1522 opened 12 days ago by meichangsu1
  0

• tokenizers-linux-x64-musl is not found when running inside node apline docker

  #1480 opened 13 days ago by madhurjya-acko
  1

• How to convert tokenizers.tokenizer to XXTokenizerFast in transformers?

  #1468 opened 16 days ago by rangehow
  3

• BPE Decoder cleanup option

  #1474 opened 16 days ago by w-zygmuntowicz
  2

• BpeTrainer seems to ignore max_token_length=1

  #1461 opened 17 days ago by geajack
  2

• Llama3 tokenizer with Incorrect offset_mapping

  #1517 opened 21 days ago by justin-shao
  1

• How to load tokenizer trained by sentencepiece or tiktoken

  #1469 opened 18 days ago by jordane95
  5

• Why are 'unknown' tokens randomly added to my tokenized input?

  #1520 opened 18 days ago by tshmak
  2

• Why the tokenizer is slower than tiktoken?

  #1519 opened 18 days ago by BigBinnie
  1

• Assign `<unusedXX>` tokens with `special_tokens` without growing vocab size

  #1473 opened 20 days ago by jacobwjs
  3

• Issue merging across whitespaces

  #1475 opened 21 days ago by henrycharlesworth
  1

• Tokens Removed from Trained Custom BPE Tokenizer

  #1516 opened 22 days ago by rteehas
  0

• UnigramTrainer: byte_fallback is false.

  #1515 opened 23 days ago by Moddus
  0

• Cross-compilation fails for custom target

  #1509 opened 24 days ago by semaraugusto
  0

• New Update causes add_special_tokens not recognized

  #1466 opened 25 days ago by sravell
  3

• `cargo build` fails for python bindings when `--locked` is passed for `v0.15.1` and `v0.15.2`

  #1477 opened 2 months ago by CobaltCause
  2

• Treatment of hyphenated words

  #1507 opened a month ago by rattle99
  0

• Failing to build bindings with 0.19.1

  #1505 opened a month ago by bryteise
  1

• Python Binding: Tokenizer.from_file() cannot parse JSON file of tokens

  #1506 opened a month ago by dwash96
  1

• Unsound use of unsafe in `src/utils/parallelism.rs`

  #1491 opened a month ago by albertsgarde
  0

• Tokens display issues

  #1470 opened a month ago by jordane95
  1

• offline installation

  #1502 opened a month ago by HankLiu10
  1

• Issue in installing rudalle on google colab, !pip install rudalle

  #1500 opened a month ago by deepanshh786
  0

• Potential vulnerability: Control token injection through Jinja templates in apply_chat_template

  #1458 opened a month ago by pluiez
  2

• Training a tokenizer with limited memory

  #1460 opened a month ago by arxyzan
  3

• Tokenizer dataset is very slow

  #1464 opened a month ago by ManuSinghYadav
  2

• StripAccents doesn't work

  #1496 opened a month ago by NivinaNull
  0

• LLamaTokenizer with `use_fast=True` / and `use_fast=False` causing memory leak when used with multiprocessing / `dataset.map(num_proc)`

  #1495 opened a month ago by michaelfeil
  1

• `BertWordPieceTokenizer` not saving with `sep_token` marked

  #1481 opened 2 months ago by AngledLuffa
  2

• wish of a Summary field in the MATADATA for Python tokenizers-0.15.1 wheels

  #1455 opened 2 months ago by stonebig
  1

• Support operating computer system

  #1457 opened 2 months ago by Southpika
  2

• different output of AutoTokenizer from that of T5tokenizer

  #1463 opened 3 months ago by sm745052
  1