hashing-files

Solution (bash, hash, postgres) to enumerate directories with several hashes (md5sum, crc32, xxhash) to build database of tree, sizes, etc. The main advantage is to analyze and easy handle duplicates in big backups!

Setup env

Dependencies

For run it please install next dependencies (all of them in official repos of Fedora atleast):

podman
podman-compose
md5sum
rhash
tree
du

Run container

First set password for the container and write it into .pgpass:

echo "POSTGRES_PASSWORD={my_cool_pass}" > .env
source .env
echo "127.0.0.1:5432:filedupes:filedupes_u:${POSTGRES_PASSWORD}" >> ~/.pgpass

And then just run the postgres container:

podman-compose up -d

Note Docker/docker-compose also should work if you use such old tools.

Confi

Main entrypoint