how to deduplicate already save_to_disk dataset ?

[StephennFernandes]

Hey @ChenghaoMou , thanks for building such an amazing tool.

could you please link to your documentation of text-dedup.

I am looking for a way to deduplicate an already save_to_disk dataset that i am laoding using load_from_disk i see there is some flag --local to be used. could not really make this work

[ChenghaoMou]

There used to be documentation for older versions. But it is less useful now because it is primarily cli-focused. You can use --help to see all the parameters each script supports.

The best course of action I would recommend is to 1) clone this repo 2) modify the code to your needs. In this case, you can simply change the load_dataset function call in the script you want to use to load_from_disk to load your local data.

[ChenghaoMou]

I am closing this due to inactivity. Feel free to reopen if you have additional questions.