maffblaster/bigdiffer

Compute binary diffs between large tar files without using more memory than exists on the planet.

PURPOSE:

Need to compute a reasonably-sized delta between two 10GB tarballs without requiring 2TB of memory?  This will do the job in a brutal, ugly, effective manner.


BUILD:


gcc -o chunker chunker.c

Everything else is in Python.  You'll need the sqlite module.


USAGE:

bigdiffer.sh amalgamates all the utilities into one callable so that you can just use it.  Be aware that it will make a mess of your disk and memory depending on how big the files are, especially if you abort it early.

QUESTIONS:

Why aren't I using a rolling hash?

I tried it.  I couldn't get consistent results.  Because of the vast tracts of nulls in tar files that have lots of little files I
was getting things like good chunk sizes for the first half of the file, and then the second half of the file as a single chunk.
So I made it use the vast tracts of nulls as break points and called it a day.  It works great on tar files.  It works ok on other things.
It's modular enough that you can easily substitute your own chunker if a different algorithm will work better for the data in question.

Doesn't this use a lot more memory than it needs to?

Yep.  If you're actually using it and it's actually a problem for you, file a bug report and I'll actually fix it.  
Otherwise, it works for what I need it for.