/UltraFastDuplicateFilesFinder

Takes a list of file from stdin. And print the duplicate ones. Like fdupes on steroids.

Primary LanguagePython

=================================
original Ultra Fast Duplicate Files Finder
=================================
  by Gautier Portet <kassoulet gmail com>
forked and extended:
  by Gabriel Reyla <gabreyla@gmail.com> 

The extended version works on Windows too:
------------------------------------------------------------------------------
$ python fileWalker.py
which folder would you like to find the duplicates in?
.
.\fileWalker.py
      size   filename
      1306   .\README
      1306   .\test\README
      1306   .\test\README_2
      1306   .\test\README_3

7 files checked (15.24 KiB), 1 duplicates (3.83 KiB).
would you like to remove all duplicates?
yes
keeping: .\README
duplicate: .\test\README
?y
deleting: .\test\README
duplicate: .\test\README_2
?n
duplicate: .\test\README_3
?n
>>> 
------------------------------------------------------------------------------

On Unix systems the program can be used as follow:
Takes a list of file from stdin.
And print the duplicate ones.


example use:

  find ~/ -size +10M | ./UltraFastDuplicateFilesFinder.py

to find duplicates in your home folder, all files more than 10MB.

UltraFastDuplicateFilesFinder compares only the very beginning of the files.
Its sufficient for most uses, but use with caution.

But this way is quite useful to detect duplicates within corrupted media files...


this is public domain.



------------------------------------------------------------------------------
example run, took less than a second to answer...


gautier@quad:~/code/tmp$ find /home -size +10M | ./duplicate.py
      size   filename
  12467906   /home/gautier/Photos/pict4614.mov
  12467906   /home/gautier/Photos/Videos/PICT4614.MOV

  13068570   /home/gautier/Photos/pict4588.mov
  13068570   /home/gautier/Photos/Videos/PICT4588.MOV

[...]

  20865498   /home/gautier/Photos/pict4695.mov
  20865498   /home/gautier/Photos/Videos/PICT4695.MOV

  28270824   /home/gautier/tmp/tsunami 1 œ ǒǑ.flac
  28270824   /home/gautier/tmp/tsunami-1.flac

136 files checked (22.75 GiB), 8 duplicates (153.45 MiB).