Issues
- 1
Add pytests to ensure that tokenizer metrics are correct, including for larger datasets
#80 opened by snova-zoltanc - 2
Unable to install package
#105 opened by snova-varunkrishna - 1
Apply Chat Template Error
#102 opened by snova-zoltanc - 0
Support user assistant chat dataset format
#101 opened by snova-zoltanc - 1
- 2
- 1
Progress Bar
#36 opened by snova-zoltanc - 1
Tracking dataset metrics
#37 opened by snova-zoltanc - 1
Greedy Truncate Right Bug
#50 opened by snova-virens - 1
tokenization time remaining not working
#82 opened by snova-zoltanc - 1
- 1
Add pytests for EOS and BOS tokens, and ensuring Llama tokenizer works properly OOB
#78 opened by snova-zoltanc - 0
- 0
README examples default to --output_path of empty directory so that they do not fail out
#96 opened by snova-zoltanc - 1
- 1
Create FAQ section
#85 opened by snova-zoltanc - 0
Prompt_prefix not interpreted correctly
#84 opened by snova-bol - 1
- 1
- 1
Documentation About Input Packing Config
#35 opened by snova-zoltanc - 3
Creation of jsonl files
#33 opened by snova-darshang - 1
apply chat_template
#66 opened by hongfenglu - 1
Token Metrics Incorrect For Large Datasets
#53 opened by snova-zoltanc - 1
- 0
- 1