Add Firefox and Cloud Sync (GitHub Gist) support
alessiosacco opened this issue · 64 comments
The automatic backup of the tab groups is a very useful feature, but currently it is only usable in a single PC configuration, due to the fact that the backups are saved in the Downloads folder. Would it be possible to add the same functionality using Firefox Sync?
I was wondering whether it would be possible to synchronise tab groups using Firefox Sync, too. My idea was to sync them as bookmarks (loosing history, but that is OK for me), potentially storing the group names in tags (allows having the same tab belong to multiple groups) or using bookmark folders (allowing just one group per tab, if that is more desirable from a design / implementation perspective).
+1
Alternatively, if someone would figure out if/how we can sync hiddent tabs, that might solve this issue. But native support by the extension would be ace, I use sync a lot - opening work content at home, and vice versa.
Hi all, I have been trying to implement this for a long time, but there is a restriction:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync
You can store up to 100KB of data using this API. If you try to store more than this, the call will fail with an error message.
If you have a lot of tabs, the data size will be more than 100KB. And then the problem, the solution of which adds only more problems ...
Is there any way to make us see the total of all tabs in Firefox sync? Because when I move tabs to groups, on my other device I only see tabs which are in the open group, tabs from other groups are not shown and hence inaccessible.
As a use case, I use groups at work to group my work-related tabs into logical parts. When I get home and want to come back to some of my work tabs, I only see the active group tabs in my synced tabs, and other tabs are only accessible from my work device.
I understand that synchronizing all tabs seems difficult, but it would be nice if at least we can synchronize groups and settings
The solution is to allow backup of groups to bookmarks as in #328, that way you have 100KB data in firefox sync to store configuration.
Hi all! I added option to export groups to bookmarks, you can sync groups/bookmarks this way
#328 (comment)
Sorry, maybe I'm dumb, but I don't understand how I should use the new bookmark export to resolve syncing use cases. Can I set it up so that all my groups are periodically automatically bookmarked? Or can I only do it manually for each group?
@hluposti you're quite right in that, this is not an automatic process.
Can I set it up so that all my groups are periodically automatically bookmarked?
Not currently, but maybe the author is working on some magic here :)
Or can I only do it manually for each group?
Currently this is the case.
The current implementation in #328 is not fully automatic. It can backup automatically, but not restore automatically. So it's not really a sync though it can kind of be used that way in a manual process. To implement sync functionality using #328, the author would need to use the functionality implemented in #328 a little bit differently.
One potential design approach would be to create a "SimpleTabGroupsSync" BookMark Folder, which STG would then automatically write all current STG Window groups. Second he would need to read this group if it exists on startup to make sure that the sync'ed bookmarks and the the current STG window groups match. To do this requires solving several hard computer science problems, something similar to a "cache invalidation" problem. He would have to keep a hash of every item in each STG window group to make sure existing items are updated and only new items are created, and vice versa. Or he can just start from scratch each time. It gets into the weeds quickly to try to implement this. He could potentially calculate a hash on the STGSync Bookmarks folder, or perhaps access a last sync'ed timestamp to know when he needs to update groups in STG from newly sync'ed bookmarks. The point is that this new implementation would require a private "STGSync BookMark Folder" which wouldn't be meant for any manual user interaction, which is how the current sync feature in #328 is implemented.
In other words, the facility to backup window groups using firefox sync has been accomplished via bookmarks (which works around firefox sync data size limitations), but that currently is a manual process, i.e. one that allows you to backup existing window groups individually.
Didn't I already do it?
Ah yes I didn't see that one! But how to import automatically from backup to new browser?
Yes of course, I see that one. But I think this feature request is asking for automatic import from bookmark folders to groups. In other works automatic synchronization of window groups between browsers by using Firefox bookmarks to transfer the data.
Computer 1 --> Browser 1 --> STG Window Groups >>>> Automatic STG BookMarks Folder Backup
Computer 1 --> Browser 2 --> STG Window Groups <<<< Automatic STG BookMarks Folder Import
Automatic import is not yet possible due to technical reasons of the browser.
What about a one click loading of some specific backup file from the drop down menu?
The steps could be:
- machine A: automatically create a backup named "machineA" (for instance) every hour (it already does that) in a shared folder (Dropbox, Syncthing or whatever the user has setup)
- machine B: click the simple tab groups toolbar icon and hit the load icon in that menu (next to the "create new group" or something like that) and automatically import the new session from file (something that this extension already does)
- machine B: automatically create a backup named "machineB" (for instance) every hour, allowing the same thing to be done on machine A when getting back to it
Automatically save the file is only possible in the download folder. And adding it to Dropbox is unlikely.
Any savvy user could do a symlink or a small script to copy that automatically to Dropbox/Syncthing/etc
I have a question about the backup to bookmark folder - it seems that when i close tabs from a tab group, they still remain in the backup and accumulate (i believe with a separator in between them). This might be the intended behavior for backups, but I tend to use this as a "sync" feature between browsers on different PCs. Therefore, it would be nice to have the option that the tabs are saved to the bookmarks "as is" - thereby replacing all the bookmarks in the previous backup. In other words, when i close a tab in the tab group, it will no longer appear in the backup, and this feature would operate as a true sync between browsers.
Is this possible as an option?
@lx0n2acl Yes, I thought to make this functionality customizable, but for some reason I didn’t)) in the near future I will make an option in the settings for the controllability of this behavior (bookmarks without tabs that are closed)
Automatic import is not yet possible due to technical reasons of the browser.
Hi Drive4ik, sorry if I reopen this old discussion.
I wonder if this could be of help, to create an auto sync from bookmarks functionality:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/bookmarks
From what I see there, it looks like it can be done. Am I missing something else?
Thanks.
I have been using Workona now. They just - finally - launched their plugin to Firefox. Works great so far, allows you to create groups much in the same manner as STG and automatically syncs across all your machines.
I had some sync problems with it on Chrome, but now on Firefox they are gone.
Hi @JackSlaterIV
Nothing to worry about
Bookmark sync already implemented
enable bookmark permission
and then you can change auto backup folder name
manually create backup of all groups
or any group of your choice
Then bookmarks will be synchronized between devices (if bookmark synchronization is enabled in the settings in the browser)
@Drive4ik
Thanks, but it does not automatically import from the bookmark folder, after the browser starts up.
Can you add this automatic import?
In one message above, you wrote that there was a technical issue in FF which blocked you from implementing this automatic import. Can you tell me more about it?
@JackSlaterIV
Bookmarks do not have attachment to FF containers, and are unlikely to have it. Therefore, you cannot restore the bookmark in the desired container.
Some users have 3000+ tabs, you can imagine how much synchronization will take.
Also, one of the main problems - the addon cannot know when FF has finished its synchronization (bookmarks, settings, passwords, etc. that you have chosen in the browser settings) and this can take a lot of time (5+ min) and it is difficult for the addon to understand when to start synchronization, and this is very important, because you need to have the most recent "snapshot" of data, but bookmarks do not have this ...
When the FF increases the capacity https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync at least to 10 MB, then I will start to think how to implement this.
It’s just what immediately comes to mind, there are more minor problems.
But so far this is not feasible.
@JackSlaterIV
Bookmarks do not have attachment to FF containers, and are unlikely to have it. Therefore, you cannot restore the bookmark in the desired container.
Some users have 3000+ tabs, you can imagine how much synchronization will take.
Also, one of the main problems - the addon cannot know when FF has finished its synchronization (bookmarks, settings, passwords, etc. that you have chosen in the browser settings) and this can take a lot of time (5+ min) and it is difficult for the addon to understand when to start synchronization, and this is very important, because you need to have the most recent "snapshot" of data, but bookmarks do not have this ...
When the FF increases the capacity https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync at least to 10 MB, then I will start to think how to implement this.
It’s just what immediately comes to mind, there are more minor problems.
But so far this is not feasible.
Hello @Drive4ik Thanks for the extension. Firefox will never have, and shouldn't have MB sync limits for the reasons you mentioned in terms of sync start-complete delays.
I do have a solution suggestion. A majority of the users are not having 3000+ tabs, and those who do should really look into using bookmarks, for the rest (and the majority of us) the following would work.
Apply simple lookup of current tab numbers and use firefox sync if they are under a safe margin (let's say 500). If they are above the safe margin, show a little alert on the top of the simple tab group window that says "Tabs Exceed Firefox Sync limits".
The sync issue is whats keeping me from adopting the plugin fully as others offer syncing. The groups really are most useful as a medium term management solution that is ideal for a desk to laptop workflow.
Let me know if I can help with UX suggestions, I am mostly a designer.
I'm also interested in this feature and I'd be willing to help implement it.
I currently have about 300 tabs in 21 groups and the latest save file in my downloads folder has 92.3kB. If I read the documentation correctly, an extension can save up to 100kB of data to Firefox Sync. That's obviously not sufficient if the data is stored in plain JSON, but I think it could be made to work: Compressing the file with gzip --best results in a file that's just 19.5kB. With brotli it's only 16.5kB. I don't think storage.sync can store binary data, so the data would have to be encoded somehow. Using base64 it's still only 26kB with gzip and 22kB with brotli. Assuming the number of tabs per kB can be extrapolated linearly, it'd be possible to store 1360 tabs with brotli + base64 in Firefox Sync.
Now, that might still not be enough for some people, but I feel it's not the majority and the extension could show an error in this case. Maybe an option to disable or only enable sync for some tab groups might be a possibility too.
Another option might be to think up some kind of binary format that's less verbose than JSON, although I'm not sure if that has a significant advantage over the compression approach while at the same time making debugging and adding new features a lot harder.
What do you think @Drive4ik ?
@maxkl there’s custom JSON encoders that can reduce the size dramatically (can’t remember the names right now)
@maxkl this is one of the ones I was thinking about: https://github.com/KilledByAPixel/JSONCrush although it’s URI-optimized, so maybe not the best match.
Now, that might still not be enough for some people, but I feel it's not the majority and the extension could show an error in this case.
uBlock Origin for example shows the amount of sync data used in relation to the totally available space using a simple bar below the cloud buttons in the settings UI (sorry it's in German):
(The light grey bar is the total space; the dark grey bar is the used space; the purple bar is the space used by my rules as a part of all synced settings.)
If FF sync support gets implemented for STG, it would be cool to indicate the fraction used of the total available space (100KiB) in the UI. So the user could reduce the number of open tabs some time ahead of running into the hard storage limit which would probably help reduce frustration when hitting the limit.
@maxkl this is one of the ones I was thinking about: https://github.com/KilledByAPixel/JSONCrush although it’s URI-optimized, so maybe not the best match.
That's a nice find! I just tried it on my 90kB file, and the result is still 51kB big, probably because of the URI encoding. But another, similar content-aware compression algorithm might work nicely. I'll try to look into that some more...
Edit: I forgot to add that compressing the JSONCrushed file with brotli results in 18kB, so it's still bigger than just brotli without the intermediate step
@salim-b Thanks, I didn't know uBlock Origin had that feature until now. I agree that that's a nice way to implement it and it's similar to what I had in mind. Their wiki page has some more info: https://github.com/gorhill/uBlock/wiki/Cloud-storage
I managed to bring it down to 15.7kB (20.9kB in base64) by flattening the tab groups array and applying brotli compression afterwards. This is what it looks in the JSON file:
"groups": [
{
"id": 1,
"title": "FPGA",
"iconColor": "hsla(277, 100%, 50%, 1)",
"iconUrl": null,
"iconViewType": "old-tab-groups",
"tabs": [
{
"url": "https://reference.digilentinc.com/reference/programmable-logic/arty-z7/reference-manual",
"title": "Arty Z7 Reference Manual [Digilent Documentation]"
},
{
"url": "https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/wp/wp-01082-quartus-ii-metastability.pdf",
"title": "Understanding Metastability in FPGAs - wp-01082-quartus-ii-metastability.pdf"
},
...
],
"catchTabRules": "",
"catchTabContainers": [],
"isSticky": false,
"muteTabsWhenGroupCloseAndRestoreWhenOpen": false,
"showTabAfterMovingItIntoThisGroup": false,
"dontDiscardTabsAfterHideThisGroup": false,
"newTabContainer": "firefox-default",
"isArchive": false,
"isMain": false,
"moveToMainIfNotInCatchTabRules": false,
"ifDifferentContainerReOpen": false,
"excludeContainersForReOpen": []
},
...
]And it's converted to this by the flattening step:
"groups": [
[
1,
"FPGA",
"hsla(277, 100%, 50%, 1)",
null,
"old-tab-groups",
"",
[],
false,
false,
false,
false,
"firefox-default",
false,
false,
false,
false,
[],
"https://reference.digilentinc.com/reference/programmable-logic/arty-z7/reference-manual",
"Arty Z7 Reference Manual [Digilent Documentation]",
"https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/wp/wp-01082-quartus-ii-metastability.pdf",
"Understanding Metastability in FPGAs - wp-01082-quartus-ii-metastability.pdf",
...
],
...
]All the other things I tried did not reduce the size after compression – gzip and brotli are dealing with common substrings really effectively after all... It might be possible to shave another kB off by also applying this flattening step to the top-level properties but I'm not sure if that's worth it because it's just a constant overhead.
Honestly, I'm not even sure if the flattening step makes sense on the tab group array. It's just a 5% improvement and would require the decompressor to be aware of the format version because the keys might change.
Let me know what you think
The first what should be done is to store the favIconUrl and own group icon urls into a separate file to use this only locally. The file size was here reduced from 230KB to 8KB without these details. For syncing groups and tabs between devices there are no favicon url data required. This could be synced locally from webpage if the tab will be reloaded.
Maybe we can save tab details with bookmarks, and only save groups and tab(bookmark) ids in the data file. It will reduce the size of the file to sync significantly.
The main problem is that we have no access to already cached favicon data in Firefox. If this bug 1315616 would be fixed (this is open already 4 years) then we could use the local stored data of Firefox self. Until this is fixed the data should be stored and ex/imported in a separate file to updata STG Favicon URL Database manually between different clients or its required to reload the webpage.
The first what should be done is to store the favIconUrl and own group icon urls into a separate file to use this only locally. The file size was here reduced from 230KB to 8KB without these details. For syncing groups and tabs between devices there are no favicon url data required. This could be synced locally from webpage if the tab will be reloaded.
Ah yes, I forgot to mention that I have already disabled backing up of favicons. I agree that they should not be synced
I'm trying to set up sync, but the extension seems to handle conflicts by just duplicating everything.
Is there a setting I'm missing that addresses this? Or some procedure to do or not do?
Looking back through this discussion, I think maybe I just misunderstood what was actually implemented. Seems like the automatic bookmarks sync behaves like firefox's own "tabs on other devices", which is the paradigm I'm trying to escape.
However I think it would be simple to get a workable solution: a button that overwrites the current session with what is in bookmarks.
It's not elegant, and would require the user manually:
- Export all groups to bookmarks & manually trigger firefox sync before leaving PC.
- Manually trigger firefox sync & import all groups from bookmarks, overwriting all local changes before using firefox.
And probably disable automatic bookmark backup so there's no chance of an accident.
Far from ideal but it'd be worth having, @Drive4ik , what do you think?
Edit: It'd probably be wise to have the backend logic only act on differences: instead of telling firefox to close all tabs and reopen all tabs, just close tabs that don't exist anymore and open new tabs as appropriate.
Oh, and I'm not concerned with syncing containers. Every container related extension for firefox has users clamoring for container syncing and it's clearly just not possible right now. It can't be done, best to just ignore it.
(As an aside, you can transport container rules between computers with this extension- it's just about the only way to do it)
I added #911 use bookmarks to store groups instead of localstorage. This would automatically add sync support.
For the beginning an automatic syncronisation with a file would be possible.
So automatic export to the backup file and when starting Firefox simply import it again. Then you could at least with Drive or Syncthing get this automatically on the other PC.
Yeah, feature of automatical synchronization (import and export both) is so demanded... It would be nice to see tab closed if it was closed on this group of tabs on another PC. Are any news how to realize it? Or any plans when it will be ready?
I'm working on this task now.
Hi @Drive4ik I see you're using Github Gists to implement the sync feature, an alternative solution to the problem indeed.
-
Another alternative solution would be to remove favicons from the cloud sync all together, compress and split the file into chunks of 8KB to be stored by Mozilla Sync. I don't know if you considered this way or even if it's a good idea, but it's an idea I had, and it indeed does not sound like a good one as the information is not obfuscated and harder to be dealt with.
-
But by just by removing the favicons from the cloud sync, my estimate is that the average user would be able to store anything from 200-1000 tabs, which should (?) be enough for most people. And you could always show for users who has sync enabled a progress bar to show how much data it's being used on the cloud sync.
-
Also, by making the default values not to be sync into the cloud (like most bool values, empty arrays, null values) you could save around 100-400 bytes per tab group. Other thing to minimize usage would be to shorten those key values for the tab groups options.
With all that, my estimate is that the average user can use the Sync without any major impacts.
Anyways, this is the best extension out there for tab management! I hope you continue to work on it. If you think my suggestions combined are a good way to do the Mozilla Sync I can try to submit a PR for it.
1 seems bad, but 2+3 seems simpler and would solve the problem.
Just discovered the addon and wow, I've been missing out! Integration with containers is amazing!
Set it up on two pcs and for some reason thought it would sync everything automagically, but alas. Glad to see the project is active and you are working on this issue 🥳 I would even pay for total sync, something that Speed Dial 2 does very well https://addons.mozilla.org/en-US/firefox/addon/new-tab-speed-dial/. Thanks for the amazing way to deal with tabs in firefox!
@Shirkit I don't think you can split it into 8KB chunks since it's a key-value storage. Here's my proposal
Storage quotas
We can have 512 items in the store. Each item can have 8192 bytes and the total storage is 102400 bytes.
Proposal
My idea goes as follows:
- Extension preferences go to
preferenceskey and we store only values that are different from the defaults. This leaves us with 511 items in the store and (102400 - size of preferences) of total storage. - Each group is stored as a separate key, this gives us the ability to store up to 511 groups (or less, depending on the total space left)
- Value for each group key should have the following structure:
{
"id": "integer",
"iconColor": "string",
"iconUrl": "string",
"iconViewType": "string",
"tabs": [
{
"id": "integer",
"url": "string",
"cookieStoreId": "string (optional)",
}
]
}I stripped the title attribute since IMO it's not really useful during sync (your newly synced device will have to reload the tab anyway) and it contributes 2.6K in my dump:
quintasan@demonbane:~$ ls -alh
-rwxr-xr-x 1 quintasan quintasan 11K Aug 1 21:57 manual-stg-backup-2023-08-01@drive4ik.json
-rwxr-xr-x 1 quintasan quintasan 8.4K Aug 1 22:12 manual-stg-backup-2023-08-01@drive4ik.json.title_strippedI removed all the extension preferences, title keys, and removed all the whitespace so my 24 tab group dump is 2.7K
quintasan@demonbane:~$ ls -alh group_payload_minimized.json.compressed
-rw-r--r-- 1 quintasan quintasan 2.7K Aug 1 22:20 group_payload_minimized.json.compressed
By compressed I mean I used jq -c . < input.json to remove whitespace.
I believe that cookieStoreId is a name of Container in which the cookies are kept so we must retain this key. I am not sure if openerTabId is mandatory to keep.
Downsides
Calculating the total storage is going to require N+1 calls to https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/StorageArea/getBytesInUse where N is the number of groups and the +1 comes from the preferences key.
Alternatively we could store everything under one key in which case we would be complicating the logic to update the tabs in groups. I belive that N+1 calls for storage calculation is the lesser problem.
What do you think @Drive4ik ?
@Quintasan your proposal makes more sense and it's simpler than mine in some regards. I did not want to change much the config file, so your idea is just better than mine in that regard.
And one can adapt your solution to split into 2 or more items if a group exceeds the 8KB, which can be stored the values for it on the preferences item if such thing is really needed (I have one group with 60 tabs that reaches just over 7KB with your proposed solution without the cookieStoreId as I do not have it on my config file