llm-jp/llm-jp-corpus

Issues

• Wiki dump 20230720 has been removed from https://dumps.wikimedia.org

  #38 opened 3 months ago by junjiechen-chris
  1

• Python 3.10 has some issues in downloading data from huggingface: use Python 3.9

  #4 opened a year ago by AkikoAizawa
  1

• Introduce category-based filtering to Wikipedia

  #30 opened a year ago by hkiyomaru
  0

• Exclude the Book3 portion in the Pile dataset

  #36 opened a year ago by hkiyomaru
  1

• Quantitatively assess the quality of the filtering

  #31 opened a year ago by hkiyomaru
  0

• Use Hojichar for better filtering of Japanese mC4

  #33 opened a year ago by hkiyomaru
  0

• Apply ethical filtering to Japanese Wikipedia

  #29 opened a year ago by hkiyomaru
  0

• Create a validation split

  #26 opened a year ago by hkiyomaru
  1

• Add the `token_ids` field

  #25 opened a year ago by hkiyomaru
  1

• Apply ethical filtering to the Japanese mC4 dataset

  #24 opened a year ago by hkiyomaru
  1

• Apply filtering to the Stack dataset

  #23 opened a year ago by hkiyomaru
  1

• Improve Wikipedia text extraction

  #22 opened a year ago by hkiyomaru
  1

• Expired links to Wikipedia dumps

  #27 opened a year ago by hkiyomaru
  1

• Construct the corpus ver. 1

  #13 opened a year ago by hkiyomaru
  1

• Use the HF datasets library for tokenization

  #14 opened a year ago by hkiyomaru
  1

• Use Python 3.11

  #15 opened a year ago by hkiyomaru
  0

• Determine the license

  #20 opened a year ago by hkiyomaru
  2

• Use the tokenizer provided by the Fugaku project

  #12 opened a year ago by hkiyomaru
  1

• Improve dataset downloading for memory efficiency

  #16 opened a year ago by hkiyomaru
  0

• Japanese Wikipedia

  #1 opened a year ago by hkiyomaru
  0

• English Wikipedia

  #2 opened a year ago by hkiyomaru
  0