steilerDev/icloud-photos-sync

Library Version 2

Opened this issue · 2 comments

steilerDev commented

Describe the ultimate goal you want to achieve

The current local file structure has certain design goals, which initially seems to have made sense, but based on the community feedback are resulting in some challenges, when it comes to use by a wider audience, requiring more flexibility and extensibility.

Additional some challenges (around filetype extensions) have surfaced that might be impossible to solve with the current structure.

Key goals:

  • Extensibility - Support for multiple files per picture. Addresses support for Live Photos (#121) and support for different resolution (#279)
  • Removing file system flakiness that e.g. leads to modification time not matching iCloud modification time (#342)
  • Streamline loading & validating local library (#291)
  • No longer dependent on a static list of filetype extensions (#143)
  • More robust (and deduplicated) archiving support to address feature requests like #314 and #284

How do you think the feature should be implemented

Current proposal:

  • Each asset would be stored as a folder, with adjacent metadata, where a user can specify which versions of the asset they want to sync (excluding e.g. live photos videos, or edited pictures, or only sync lower res versions of the file)
  • The format of the metadata should mirror the iCloud Photos backend as of the state of the sync and should be the foundation of making diffing decisions
  • Given I currently don't understand the iCloud provided checksum, the metadata file could hold our own checksum
assetUUID/
├─ resOriginalVidComplRes
├─ resOriginalRes
├─ metadata.json
  • Those assets would be part of a raw data directory, that will also provide context on their lifecycle (either they are synced from the primary or shared library, or they were archived or even soft deleted - see #314)
library/
├─ .library-data/
│  ├─ primary/
│  │  ├─ <assetUUID1>/
│  ├─ shared/
│  │  ├─ ...
│  ├─ backup/
│  │  ├─ ...
│  ├─ deleted/
│  │  ├─ ...
  • The final piece will be the user folders and meta albums (like 'All Photos', 'Shared Photos', 'Archived Photos')
  • Each folder will contain its metadata (primarily its UUID and folder content)
  • Since all asset metadata will be stored separately, we could arbitrarily name those files and give users the option to pass their preferred formatting string as well as link a specific version of the file (and/or multiple versions of a file, depending on the user's preference)
library/
├─ .library-data/
│  ├─ ...
├─ _All Photos/
│  ├─ ...
├─ _Shared Photos/
│  ├─ ...
├─ _Archived Photos/
│  ├─ ...
├─ <userFolder1>/
│  ├─ metadata.json
│  ├─ 2023-01-01 - someFile -> ../.library-data/primary/<assetUUID1>/resOriginalRes

Timeline & Expectations

This would require a massive rework, and I would need feedback from the community if a migration path would be necessary (which would increase the amount of work that needs to be done)

I will address some sync/library related issues first before tackling this initiative as part of a v2 release. So it would be great if we could have more inputs on this topic to come up with the best possible (long term) solution