Explanation for slow sync on Android w/lots of files + request for fix
taoeffect opened this issue · 2 comments
(Note: I realize that this project is in maintenance mode. As this is not a feature request, I've decided to post this issue here, as well as in SyncThing-Fork, in case someone is able to take this on for Android/GrapheneOS users.)
Description of the issue
Hi, SyncThing is perfect for syncing large amounts of small files that don't change, e.g. photo albums and music libraries.
However, on Android, it's a well known issue that the performance is almost prohibitively bad for this use-case.
Example threads:
- https://forum.syncthing.net/t/sync-becomes-very-slow-between-mac-and-android-device/21614
- https://forum.syncthing.net/t/syncing-to-android-is-incredibly-slow/20752
I reached out to the GrapheneOS developers to see if they knew why this might be happening.
Here is what they had to say (reposted with their permission), I hope it is helpful to addressing this issue. Although their response might be taken as a bit abrasive, I ask that you please forgive them and look past it, in case what they have to say is useful for addressing this problem that users (like me and others) are facing:
It's almost certainly primarily caused by how the app is choosing to do it rather than an inherent issue with Android. Accessing files in shared storage is certainly slower than native filesystem access in the app's storage. That's why it would be best for an app like this to sync to their own storage and implement a content provider for accessing it instead of putting it directly into the user's shared home directory. It's likely the home directory performance will improve though.
The app accessing their own files is native f2fs filesystem access which has the usual performance of f2fs with SELinux. SELinux adds a bit of overhead, but not an enormous amount. There are multiple ways to access shared storage, some performing better than other methods, but it's not direct filesystem access. An app using Storage Access Framework to access a huge number of files is certainly going to be a lot slower, while lower level access to shared storage isn't high overhead.
They're blaming Android but it's how they designed their app. They chose to do it this way. They could have synced the data to their app storage by default and provided a storage provider for browsing it in the file manager. If they did that, it would appear as an entry in the file manager's list of storage providers where the top level home directory appears.
Android focused on trying to get apps to stop misusing the home directory and demanding bulk storage accessed.
The workarounds to provide backwards compatibility with apps depending on bulk storage access led to access to the home directory being slower than before. This app appears to have simply reworked their code to continue functioning with this instead of considering redesigning it to better fit into how Android intends for storage to be used. They could have native filesystem performance if they used the native filesystem access to their own app sandbox directory they have by default.
They just need to start saving files to their own directory and expose their own directory as a storage provider so users can access it from the system file manager / system file picker.
This app is primarily meant to sync a specific directory across computers, right? They should be using a directory in their app sandbox where they have native filesystem access and exposing that to users via a storage providers.
They're just not doing this in a good way for Android.
It is true that accessing the user's home folder is slower than native file access, but it shouldn't be as slow as their app, so something seems wrong with their implementation. It's also their choice to take that approach instead of the high performance approach. They ended up doing that since they were always doing it that way and the performance gap has always been there but used to be smaller. Security/privacy features slowed down home directory, but not their sandboxed directory.
They're blaming Android but it's their choice to put files in a location that has always been slower to access and the gap in performance got much wider. Android wanted developers to stop using the home directory instead of using their sandboxed storage and exposing files to users and other apps via content/storage providers. The performance hit to home directory was a consequence of adding abstraction layers to provide fine grained access control with backwards compatibility.
The reason we were able to implement our Storage Scopes feature in GrapheneOS is because Android abstracted home directory access in a way that enabled us to build that feature on top of what they provide. It was a good change. The problem is that app developers don't know the platform well and those that do don't want to rewrite significant parts of their app to work the way it always should have been done. They're paying for doing things in an invasive way to begin with.
Version Information
SyncThing-Fork 1.27.2.1 on GrapheneOS (I'm guessing the same issue exists for this repo as well though.)
Although their response might be taken as a bit abrasive, I ask that you please forgive them and look past it
It's really hard not to start a rant. This whole response has a bit of a "piss me in the face and tell me it's raining" feel to it. Guess where the user's media files are that he wants to synchronise? Exactly, they're outside our app's storage.
They could have synced the data to their app storage by default and provided a storage provider for browsing it in the file manager.
It's a valid approach for a limited use case. The problem is that most of the necessary changes would have to be made in the app itself, such as a UI that guides the user to select the correct location, and the implementation of the storage provider.
I'm also worried about the restrictions if the app is being uninstalled. Are the files simply gone?
Just for the record, you can already sync into the Syncthing app's own storage if you input the path manually using the Web GUI, however the first problem is that the path is inaccessible to other apps (hence the talk about providing a dedicated file browser), and the second problem is that when it comes to Android specifically, most users simply want to sync their photos and videos, and those are located on the shared storage, which makes the suggested approach not very feasible anyway.