exfat path normalisation results in deleting/recopying paths with trailing dots

[cdown]

Hi there!

Most modern Android devices use exfat/sdfat on the backing device. These filesystems strip trailing dots on path components as stripped as part of path resolution.

% dd if=/dev/zero of=exfat bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.313496 s, 3.3 GB/s
% sudo losetup -fP exfat
% losetup -a
/dev/loop0: []: (/tmp/exfat)
% mkfs.exfat exfat
exfatprogs version : 1.1.3
Creating exFAT filesystem(exfat, cluster size=32768)

Writing volume boot record: done
Writing backup volume boot record: done
Fat table creation: done
Allocation bitmap creation: done
Upcase table creation: done
Writing root directory entry: done
Synchronizing...

exFAT format complete!
% mkdir test
% sudo mount -o loop /dev/loop0 test
% sudo mkdir test/...t.../
% sudo touch test/...t.../...e...
% sudo ls -la test/...t.../...e...
-rwxr-xr-x 1 root root 0 Sep  7 14:28 test/...t.../...e...
% sudo ls -la test/...t.../...e..
-rwxr-xr-x 1 root root 0 Sep  7 14:28 test/...t.../...e..
% sudo ls -la test/...t.../...e.
-rwxr-xr-x 1 root root 0 Sep  7 14:28 test/...t.../...e.
% sudo ls -la test/...t../...e.
-rwxr-xr-x 1 root root 0 Sep  7 14:28 test/...t../...e.

This results in deleting the old files, and then copying them again, even though they are the same:

[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/01 Stop.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/02 Malaise.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/03 Low Concept_High Maintenance.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/04 Nonsanity 2037x.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/05 Mr. Prince.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/06 Even the Scars Forget the Wounds.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/07 Go.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/08 Filling in a City.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/09 Destination.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/10 Mobile.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/11 Aggression.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/12 Then Silence.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/13 Stop and Go All Ye Faithful.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam/14 Graduated.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing folder /storage/21D1-F045/Music/Grüvis Malt/...With the Spirit of a Traffic Jam (Base.py:57)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/01 Home Work.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/02 Mouthful.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/03 Good People Check.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/04 Poisonpit.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/05 Live Trap.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/06 Only Child Explosion.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/07 Paging Dr. Moon Or Gun.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/08 Dark Sky Demo.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/09 You Devil You.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/10 Out In The Open.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing /storage/21D1-F045/Music/Themselves/the no music/11 Hat In The Wind.mp3 (Base.py:49)
[2022-09-07 13:50:05][INFO] Removing folder /storage/21D1-F045/Music/Themselves/the no music (Base.py:57)

[...]

[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../ (Base.py:85)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../01 Stop.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../02 Malaise.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../03 Low Concept_High Maintenance.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../04 Nonsanity 2037x.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../05 Mr. Prince.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../06 Even the Scars Forget the Wounds.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../07 Go.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../08 Filling in a City.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../09 Destination.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../10 Mobile.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../11 Aggression.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../12 Then Silence.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../13 Stop and Go All Ye Faithful.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Grüvis Malt/...With the Spirit of a Traffic Jam.../14 Graduated.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./ (Base.py:85)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./01 Home Work.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./02 Mouthful.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./03 Good People Check.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./04 Poisonpit.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./05 Live Trap.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./06 Only Child Explosion.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./07 Paging Dr. Moon Or Gun.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./08 Dark Sky Demo.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./09 You Devil You.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./10 Out In The Open.mp3 (Base.py:75)
[2022-09-07 13:50:05][INFO] ./Themselves/the no music./11 Hat In The Wind.mp3 (Base.py:75)

See __exfat_resolve_path for normalisation rules. Specifically, we should compare after stripping all trailing periods on these filesystems.

The harder parts are detecting if we're on exfat/sdfat in the first place with different phones having fastly different FUSE/DM topologies. Probably the strategy should be something like:

1. Build a map of major:minor to whether this is an exfat-like mount stripping dots;
2. On Android side, store major:minor for each path;
3. Check if the path is on an exfat-like mount, and if so do custom equality check between local <-> android.

I'd usually just send out a patch, but I'm about to head out on holiday, so wanted to write this up first :-) I can take a look in a couple of weeks.

Thoughts and feedback on the idea appreciated, of course.

Thanks!