README

Deduplicate files on your APFS file system.

APFS allows cloning files instead of copying them
This script detects duplicates and replaces them with clones
More information (clones): https://developer.apple.com/documentation/foundation/file_system/about_apple_file_system

Requires python3
To download wget -O deduplicate.py https://raw.githubusercontent.com/capyvara/apfs-deduplicate/master/deduplicate.py && chmod +x deduplicate.py
Run ./deduplicate.py -h for help
To save time, this script will first compile a list of probable duplicates by computing a hash of the first 1024 bytes of data. Of the probable matches, a hash of the full file contents are then computed, and duplicates are replaced with clones via calling cp -c
Note: This script is considered experimental. Although it has been tested on a number of different data sets (git repositories, RDBMS storage, etc.) it should not be run on sensitive data

capyvara/apfs-deduplicate