Script to remove duplicate quotes

Question

Script to remove duplicate quotes

mubaris opened this issue 7 years ago · 15 comments

Add a small script to automatically delete all the duplicate quotes in the data folder.

Check #25

Answer 1 · 2017-10-16T02:42:49.000Z

Is it to remove duplicated quotes in an unique file or among all files? Is the script allowed to edit the files in data folder?

Answer 2 · 2017-10-16T03:15:45.000Z

We want to remove duplicate quotes from all the json files in data folder. We already have a script to find duplicate entries. That script will be useful.

Yes, the script is allowed to edit the files in data folder. Whenever there's a duplicate entry in json file, it should be removed.

Answer 3 · 2017-10-16T03:52:25.000Z

Ok. I'll take this issue.

Answer 4 · 2017-10-16T11:23:53.000Z

Can't run $ motivate

TypeError: join() argument must be str or bytes, not 'PosixPath'

Answer 5 · 2017-10-16T12:19:42.000Z

@michellymenezes I think I have fixed this issue. Let me know if there's any problem. Clone and install again

Answer 6 · 2017-10-16T14:55:41.000Z

@mubaris I've made a script to create a new json file with all unique quotes #25

Answer 7 · 2017-10-16T15:10:04.000Z

@michellymenezes It would be very useful instead creating new json file, if we could remove the entry from existing json files.

Answer 8 · 2017-10-16T15:30:44.000Z

Can I take this up?

Answer 9 · 2017-10-16T15:32:58.000Z

There's already a PR. There's a small problem with it. If you can find a method to eliminate that issue, let me know.

Answer 10 · 2017-10-16T15:34:05.000Z

What's the problem?

Answer 11 · 2017-10-16T16:40:17.000Z

Hey in your find_dupes.py file,can you explain me in this line of code
dupes = sorted([x for x in quotes if x['quote'] in seen or seen.add(x['quote'])], key=lambda x:x['quote'])
What do the 'key=lambda x:x['quote']' does?

Answer 12 · 2017-10-16T17:28:07.000Z

@mubaris I think removing repeated quotes from each file will cost more than computing unique quotes and getting everything from a unique file.

Answer 13 · 2017-10-17T14:04:18.000Z

@mubaris Hey is the issue still open?I made a fix for what you want to achieve.

Answer 14 · 2017-10-21T04:36:08.000Z

#40

Issue was still open, you mentioned there were issues with previous implementation. In this case it's better to not worry about cost, but worry about ease of the database. Maintaining multiple json files only makes sense if there is a difference between them (001 = science quotes, 002 = history quotes, 007 = james bond quotes). If you wish for randomization it'll be easier if you don't have to randomize file in addition to the quote.

If you'd like I can do the same refactoring to your same quote as this one. It ends up improving readability.