karpathy/minbpe

Issues

• Question about Encoder Logic

  #87 opened 5 months ago by JackxTong
  5

• One problem in the annotations of `test_wikipedia_example` in the `tests/test_tokenizer` file

  #93 opened a month ago by donglinkang2021
  0

• LLM is worse at non-English languages

  #92 opened a month ago by 7CD
  0

• Interface to remove Unseen or rare encoding paths from downloaded models given new dataset

  #36 opened 3 months ago by cmollgaard
  2

• Python API with C extensions for faster training and encoding

  #85 opened 6 months ago by benarnav
  0

• counting pairs is inaccurate for repeating tokens?

  #51 opened 10 months ago by JohannesVod
  5

• Instead of finding the one pair with the highest frequency and merging it at each step, do the highest N pairs

  #69 opened 8 months ago by hippietrail
  2

• LLM as calc

  #81 opened 7 months ago by michaelshekasta
  0

• OSS-Fuzz Integration

  #80 opened 7 months ago by ennamarie19
  0

• Huggingface already has an efficient implementation of this?

  #58 opened 9 months ago by laurislopata
  3

• BPE in Haskell

  #79 opened 7 months ago by BobMcDear
  0

• The regular expressions break all scripts with combining marks in the middle of the syllable

  #73 opened 7 months ago by ajaykg
  3

• What to support GPT-4O tokenizer？

  #77 opened 7 months ago by echo-valor
  0

• Notebook Issue In Google Colab

  #74 opened 7 months ago by kelixirr
  0

• Amplifying your courses with my digital notes

  #70 opened 8 months ago by AayushSameerShah
  0

• `minbpe-rs`: A pure Rust implementation of `minbpe`

  #66 opened 8 months ago by shubham0204
  2

• Need train_multi.py example to show use with multiple input files

  #47 opened 8 months ago by gnp
  3

• decode() method in GPT4Tokenizer does not handle special tokens

  #64 opened 9 months ago by Vakarva
  0

• Would using prompts that contain concatenated words to reduce token count negatively affect results

  #61 opened 9 months ago by hatgit
  0

• Optimizing minbpe to also support video tokenization (extract low-dimensional latent patches from video frames)

  #48 opened 9 months ago by Jaykef
  1

• Faster BPE

  #5 opened 10 months ago by zouharvi
  11

• Implementation of LlamaTokenizer (without sentencepiece)

  #60 opened 9 months ago by MaveriQ
  0

• "regex.py" file name conflict

  #59 opened 9 months ago by mogomaa79
  0

• Alternative to bpe

  #50 opened 10 months ago by marcov-dart
  16

• _

  #57 opened 9 months ago by momonga-ml
  0

• Using minBPE token encoded sentence vectors need to be padded

  #56 opened 9 months ago by elevateclub
  1

• Minbpe as a potential course

  #19 opened 10 months ago by ViswanathaReddyGajjala
  4

• Alternative to „Vector Representation Pre-training“ possible?

  #30 opened 10 months ago by Heavy02011
  1

• how to deal with special tokens for multiple files

  #44 opened 10 months ago by IamExperimenting
  0

• A thanks from self-learners community

  #45 opened 10 months ago by IamExperimenting
  0

• Vectorized BasicTokenizer.train?

  #29 opened 10 months ago by kuprel
  2

• Byte2Byte Tokenizer

  #37 opened 10 months ago by loretoparisi
  0

• `regex.py` is feature-engineering and will probably degrade performance when scaling to many languages

  #31 opened 10 months ago by domschl
  3

• A probably faster way for training the tokenizer (pure Python)

  #25 opened 10 months ago by ReinforcedKnowledge
  3

• Steal token visualisation code

  #11 opened 10 months ago by hauntsaninja
  5

• What is the difference about the bbpe vocab decode method in minbpe against huggingface transformers?

  #15 opened 10 months ago by lovekittynine
  0

• Saving/Loading tokenizer from disk

  #2 opened 10 months ago by cyrilzakka
  4

• Loading data from disk partially

  #8 opened 10 months ago by kathir-ks
  2