Bookmark folders not synced
Closed this issue · 30 comments
Bookmarks inside folders are not being synced correctly.
This isssue is extracted from the bottom comments of issue #49.
Two things that should be done:
- Check out if any other Sync data (e.g. passwords, form data,...) also experiences issues or if this is solely related to bookmarks.
- Does it work correctly when using older versions of Firefox (e.g. two Sync clients of both Firefox 25)? Maybe some changes were introduced by Mozilla once they activated the new Sync protocol option.
@ogasser Do you think Firefox 24.3.0 ESR would do? I am not certain how to install an earlier version of Firefox and make it -not- update to current.
Did you see my other comments "Addendum2" in #49, last message in that thread? I think they are relevant in addition to the version hypothesis you are proposing we test.
I never tried synching Thunderbird.
@craigarno You can download an older version (e.g. Firefox 25) and then disable auto updates.
I saw your addendum just now. It would be good to test a scenario with two instances of Firefox 25. Just to make sure.
@ogasser my bad. I meant Firefox 24.3.0 ESR.
My plan is use a Windows Vista x64/FirefoxESR 24.3.0 and Windows XP-SP3(32-bit)/Firefox 24.3.0 machine for this round of testing. I plan to leave the Linux and Win7 machines at Firefox 26.0(Linux) and Firefox 27.0(MS-Win7). We'll see what we get shortly.
Sounds good to me, @craigarno!
OwnCloud 5.0.14a, Mozilla_Sync 1.3.1.92
M1-Windows Vista x64/FirefoxESR 24.3.0
M2-Windows XP-SP3(32-bit)/Firefox 24.3.0
All operations performed in the Firefox "Bookmarks Menu" area.
M1(Vista) - Rearranged a sub-folder ("Atmel"), deleted some "top level"(not in Atmel) extra dividing lines and performed a Sync.
M2(XP) - Performed a Sync, the "Atmel" folder did not change to reflect the new arrangement, but dividing line changes did get Sync'ed. (Sync operations are happening and only working on top level bookmarks)
M2(XP) - Created a new folder called "Test Folder" then performed a Sync.
M1(Vista) - Performed a Sync and looked for "Test Folder". "Test Folder" never showed up.
It appears Firefox ESR 24.3.0 against OwnCloud 5.0.14a/Mozilla_Sync 1.3.1.92 exhibit the same behavior as newer versions of Firefox (26/27) [which are still setup, running, and exhibiting the same behavior].
One last check... nope, nothing changed on either system, lots of minutes later.
BTW, Sync changes are fairly immediate... i.e. as fast as I can press Sync then switch to the other machines and press "Sync", the changes which are working (top level Bookmarks) show. Also, special folders like "Recent Tags" and "Recently Bookmarked" (which appear to just be "links" to actual information) also get copied/deleted during a Sync operation. It's only actual folders and folder contents which aren't being Sync'ed. This can be tested with a simple copy/paste of "Recently Bookmarked" so you have two in the bookmark list; copy, sync, sync, observe, delete, sync, sync, observe. Apparently Firefox allows two items of identical names in this "list". So this Firefox Bookmark list does allow for duplicate identical names in the list. Probably irrelevant, but property I never noticed before.
I think I found the solution: A bug was introduced in aba3d70 when working on the debug log issue #49. Please test the code from the pull request @craigarno @alexboss.
@ogasser mixed bag results with Mozilla_Sync1.3.1.99. I'll look at it more, wouldn't mind also getting @alexboss critical feedback.
For test I used:
- Firefox 26.0 (Linux-OpenSuse 12.3 x64)
- Firefox ESR 24.3.0 (MS-Windows XP-SP3)
- Firefox 27.0 (MS-Windows Vista Ultimate x64)
What I think I see is Firefox Sync has trouble re-reporting any list item it thinks it's already reported to the non-functional Mozilla_Sync1.3.1.92 and before. i.e. if a Sync was performed once on a folder and/or it's contents, Firefox doesn't seem to report it to the Sync Server again. So previously (silently) failed Sync's don't get caught up.
Brand new folders and brand new folder content seems to Sync just fine with Mozilla_Sync1.3.1.99 and the three Firefox/OS combinations listed above.
Here's where it gets confusing...
As near as I can tell, each Bookmark knows the date-time it was made. There is no "touch" ability in Firefox Bookmarks. Once it's been reported as "Sync'ed", sometimes it will sync, sometimes it won't. As an example, I have an "Atmel" folder I mentioned earlier. It had 3 Bookmarks in the folder in one browser. In another there was an Atmel folder, but no bookmarks. No amount of Sync'ing on either Firefox would change this. I tried to Sync a third Firefox machine which already had the Atmel folder and 3 Bookmarks. Syncing it in any combination with other Sync's didn't change anything, one Firefox machine Atmel folder was still empty.
So I created a Test2 Folder and moved the three Atmel Folder Bookmarks into Test 2 Folder and performed a Sync. On another machine the Test 2 Folder and 3 Bookmarks showed. On another machine only the Test 2 Folder showed and it was empty. I moved the 3 Bookmarks back to the Atmel Folder and on one machine they showed and on another they didn't.
I then selected all three bookmarks and performed a Copy, then Paste making 6 Bookmarks (a copy of the 3) in the same folder and performed a Sync. Three Bookmarks showed up on the machine which was previously empty. I went back to the machine with 6 Bookmarks and deleted three of them. On the other machine all three disappeared again... apparently I deleted the 3 copies and not the three original Bookmarks which wouldn't sync. Ok, paying more attention this time, I made three copies again and deleted the three originals then performed a sync. On the previously empty machine, three Bookmarks (the copies) showed. On the third machine I performed a Sync and had 6 Bookmarks. Ok, which are the copies? I deleted the three I thought were copies... wrong choice, now the three on the first two machines disappeared. So I make copies again, deleted the three originals, performed a Sync on the other two machines and now the folder looks as it should.
So I don't know how to force a Sync to remedy this situation where Firefox Sync gets out of Sync with the Mozilla_Sync server. Firefox seems to remember a lot and won't allow something I think I want to call a "forced sync". This "Forced Sync" option only seems to appear asking for (Sync, Overwrite Firefox with Sync Server, Overwrite Sync Server with Firefox) if Sync is Unlinked, then restored in a fresh profile using Account:, Password:, and My Recovery Key:.
So recovering from getting out of Sync between Firefox and Sync Server with this condition is arduous and painful if you have a non trivial number of machines (like my 10+). I don't know of any solutions for this issue.
So, with Mozilla_Sync 1.3.1.99, from install time forward, this issue appears to be solved. There is a window of vulnerability between the time Sync'ing was broken and when Mozilla_Sync 1.3.1.99 is installed. I don't know how to correct Folders and Bookmarks created during this Window of vulnerability. Perhaps @alexboss will see something a little different which might clear up this issue? Maybe after we nail this down a little better, a new Sync Feature request needs to be submitted to Mozilla?
@craigarno In the Sync preferences in Firefox you can Manage Account -> Reset Sync. That's exactly the force resync scenario ;)
@ogasser Thank you, just what I needed.
I had a little trepidation about trying the "Recommended: Merge this device's data with my Sync data", but it seemed to work as I needed and no data was lost. This includes a few machines I had to unlink from Mozilla's Sync Servers and reconnect to my OwnCloud:Mozilla_Sync server.
Trepidation was caused by not finding a clear description with detailed examples of what "Recommended: Merge this device's data with my Sync data" does. Thankfully it worked as I had hoped. It appears to turn off "Remove", ignores date-time information and just shuffle what exists in both locations together. When it's done (no clear indication of completion) you resume normal Sync operation by deleting anything you don't want.
I'll give this one my thumbs up, even on a new Windows 8.1 x64/Firefox 27.0(32-bit) installation. Can't wait to see if @alexboss agrees (since he spotted this problem before I did!).
What timezones are you guys in? I'm in GMT-8
Hi guys @ogasser and @craigarno , sorry for not being able to report sooner :(
Thanks a lot for the fix provided, it has been installed on the server and I then performed new testing...
Here is what I did:
1/ Device 1 - Create new bookmark folder F1 and new bookmark B1, and subscribe AdBlock to new filters
2/ Device 1 - Perform a Sync Now
3/ Install the pull request #99
4/ Device 2 - Start a new Sync - bookmark folder F1 and bookmark B1 are not synced 👎 but AdBlock preferences are retrieved and synced successfully 👍
5/ Device 2 - Create a new bookmark folder F2 and new bookmark B2
6/ Device 2 - Perform a new Sync
7/ Device 3 - Start a new Sync - F1+B1 are not synced 👎 , F2+B2 are retrieved and synced 👍 , AdBlock preferences are retrieved and synced successfully 👍
8/ Device 1 - Perform a Sync Now
9/ Device 1 - F2+B2 are retrieved and synced 👍 , AdBlock preferences are retrieved and synced successfully 👍
10/ Device 1 - Perform a Reset Sync
11/ Device 2 - Perform a Sync Now
12/ F1+B1 are retrieved and synced 👍
So to me, it looks it's working much better with this fix. However, I had this F1+B1 I couldn't to sync until I perform a reset sync, maybe it was caused by the fact I didn't install the fix before creating these new resources. I could confirm tomorrow by applying further testing with other computers synced at work with the same account; indeed I created folders and bookmarks on these computers and made several syncs but before the pull request #99 has been installed.
But definitely it's getting better and it seems to get back to normality, either naturally (meaning not doing any Reset Sync but just using the automatic sync process or Sync Now) or requiring a little action from user (Reset Sync).
Anyway thanks a lot @ogasser for identifying the origin of the issue and fixing it so quickly, and also @craigarno for the testing and feedback provided, very helpful and appreciated 👍 👍 👍
And before I forget, I'm GMT+1 !
Best regards,
Alexandre
The reason for F1+B1 not being synched without resetting is that without the fix F1+B1 were not properly saved in the Mozilla Sync database (w/o the fix applied).
Thx for your testing efforts @alexboss and @craigarno. Waiting a day or two before merging PR 99 into master. This way you can do some more testing (if you want).
GMT+1 as well.
My test plan is to use and observe, mostly Bookmarks and History. My current activities are to setup and learn more about an embedded development IDE and GIT. While I'm doing this I'll pay attention to saved/modified Bookmark Sync actions. I'm primarily using Linux and Windows 7, both with Firefox 27.0/OwnCloud:Mozilla_Sync 1.3.1.99. It is interesting to note that -all- data associated with a Bookmark appears to sync. Apparently all data in the following window is contained in the WBO "payload" field.
If anyone else would like to pay attention to things I'm not hitting with my activities; Sync [Add-ons, Passwords, Preferences, Tabs] or DB[addons, clients, crypto, forms, meta, passwords, prefs, tabs] then we might end up with better coverage with the combined activities of three of us.
Alright,
Thanks for the explanation ogasser, indeed I performed additional testing this morning and I can confirm that all bookmarks folders and bookmarks added between the last successful sync and the installation fix #99 were not synced.
Even clicking "Sync Now" does not help for these added "in between" bookmarks, while bookmarks added after fix #99 sync perfectly.
Anyway, it's not a big deal to me as the "Reset Sync" fixes the situation and allow to re-sync all resources without any trouble.
Just like mentioned by craigarno, I didn't pay attention to other elements synced, I just got focus on bookmarks folders / bookmarks (which were obviously impacted with this bug) and AdBlock plus preferences (which apparently were not impacted). Maybe for the future, in a similar case, it worth it to have a full coverage of sync-able items.
But again, thanks for the fixing and great support,
Best regards,
Alexandre 8)
@ogasser , @alexboss , I performed a quick check on Sync'ed Passwords this morning (just now) and Sync of Passwords appears to function as I expect.
- Machine 1: MS-Windows 7 / Firefox 27.0
- Machine 2: OpenSUSE 12.3 x64 / Firefox 27.0
- URL tested is my own OwnCloud instance https://cloud.MyDomain.tld
- Machine 3 (Server): OwnCloud:Mozilla_Sync 1.3.1.99
Machine 1 - I deleted three URL/Username/Password entries verified on Sync'ed Machine 2 three entries disappeared (after a forced Sync on Machine 2). Then I restored one entry by "Remembering" a Username/Password, and after a forced Sync on Machine 2, the restored URL/Username/Password appeared.
Machine 2 - Performed same actions and URL/Username/Password entries appeared, disappeared as expected on Machine 1.
Other notes: Password "Remember"/"Remove" actions are observed to Sync immediately to OwnCloud:Mozilla_Sync. Since there is no "tracking/notification" communication mechanism for live Firefox instances, a "Sync Now" is necessary on the receiving machine to see URL/Username/Password entry changes.
Thx for the password sync test, @craigarno!
The option on when to sync is left to the client in the Sync protocol. Firefox employs some kind of push mechanism whenever data is created/changed/deleted in the browser it is (most of the time) immediately sent to the server. Other clients on the other hand check for updated data after a certain time interval or by the user forcing a Sync now.
Summarizing the current testing status for successful syncing:
- bookmarks
- passwords
- history
- tabs
Once we verified that everything is working as expected, we can merge the PR into master and release Mozilla Sync v1.4.
@ogasser I've been using the History Sync rather continuously. So this can also be checked off.
History Sync appears to follow "time interval" or Sync Now rules. i.e. each new URL visited does not force a Sync.
I have not specifically tested Addons. I can note that new installs of Firefox are observed to load my usual basic set of Addons, which haven't changed for some time. I haven't specifically removed or added an Addon to see if this action is Sync'ed. I seem to remember @alexboss saying his Addon preferences for Ad-Blocker perform Sync as he expects. Addons Sync appears to be partially covered, but not specifically tested in usecase test procedure fashion. i.e. Remove, Sync, observe, Add, Sync, observe, etc.
Yes, exactly, @craigarno: For passwords and bookmarks Sync seems to be triggered immediately, whereas there is some kind of pipelining/collecting of records before Sync is done for history and open tabs.
Updated list:
- bookmarks
- passwords
- history
- tabs
I don't use the tab syncing feature a lot but on the occassion that I needed it, it worked. Did anyone of you experience problems or do some more testing regarding tab syncing, @craigarno @alexboss?
I haven't tested Tabs. My browser is set for "Show my windows and tabs from last time", so I suspect what we want to see/test is masked by this feature-option. I'll try the "Show my home page" setting to see if Tabs behavior changes with Sync activity. My different computers tend to be used for different functions, like software development, hardware development, accounting/taxes, documentation, etc. So having the same set of tabs show on all machines isn't normally desirable behavior for how I use these machines. I'll try it for a bit to see how this feature operates and report back if I observe anything useful.
The synced tabs are not automatically opened on all browsers. They can rather be seen at about:sync-tabs.
@ogasser , thanks for the insight and tip. This is a new area for me.
_Summary_
Sync-Tabs isn't working.
_Observations_
- The used machine does not display in it's own about:sync-tabs listing.
_Things I tried_
I spent many hours reading through the Firefox site looking for definitive clues about when Tabs are synced, and all the information just says they are Sync'ed, not when.
My assumption is a Sync Now on the Firefox Tabs of interest, followed by a Sync on the receiving machine followed by a Refresh-List should update the Tabs listing for the machine under study. It doesn't. And no entries are made in the OwnCloud.log.
I used Windows 8.1 with Firefox 27.0, and when Sync-Tabs wasn't working, I decided to try with a more mature and stable Windows 7 machine, and of course OpenSUSE 12.3 x64 Linux with Firefox 27.0. Basically all three clients behaved similarly and not how I believe they should with Tabs. History and Bookmarks are still working.
Considering earlier issues with Sync, I performed a Sync Reset... Merge on all three machines. This had no new effect on the results. Since Windows 8.1 is a fresh install, the fact that one Tab entry displays says at least something is trying to happen.
Here's an example from Windows 8.1 with Sync-Tabs entries on the other two machines:
How this browser appeared under Linux:
How this Browser appeared under Windows 7:
I'm no wiser about where the problem exists, but it doesn't work. I thought I'd report back to get your thoughts before spending more time to see how Mozilla Weave servers work with this feature.
BTW, considering your list of 4 Sync itmes, here is what I stumbled across while looking for Sync-Tabs information, posted on Mozilla's site:
https://support.mozilla.org/en-US/kb/firefox-sync-take-your-bookmarks-and-tabs-with-you#w_how-much-of-my-data-is-synced
Any of the following can be synchronized with the Mozilla server:
- Bookmarks — including all tags, keywords, folders, and your custom menu order
- Passwords
- Preferences/Options of Firefox
- History of sites visited in the last 60 days
- Tabs and tab groups
- Add-ons installed on your computer or mobile device
I noticed when I change an option, like the General Option; what will display when I first start Firefox, that information is synced to other broswers. While not exhaustive, it appears Sync-Preferences/Options works.
Hello @craigarno, I'm not sure this is actually a bug, or maybe I haven't understood correctly the issue.
You say that "The used machine does not display in it's own about:sync-tabs listing." and I observe the same thing, but maybe it's an expected behavior "by design" considering it might not be necessary to display the opened-tabs in the sync-tabs, as they can be seen by the end-user directly through the browser interface.
Anyway, what I observed is that indeed the "user machine" does not display in it's own about:sync-tabs listings, but it appears on the other synced machines, and of course also display the opened-tabs of the other synced machines.
What I did observe also is that opened-tabs are not synced if you browser runs in "privacy / paranoid" mode, i.e. without saving any history. But here it also makes sens, why "save" the status of opened tabs when the requester decided not to save any history.
Alexandre 8)
@alexboss is right. It is by design, that your own client and its tabs are not displayed on the about:sync-tabs site. The title of the page already says "Tabs from other devices".
If the other clients and their tabs are displayed, it is working correctly. If this is the case I would suggest to close this issue and merge the PR. What do you say @craigarno @alexboss ?
Well, for what I tested and my needs, syncing is working fine. So it's fine by me 👍
@ogasser @alexboss I think I confused the issue.
Sync-Tabs with OwnCloud:Mozilla-Sync 1.3.1.99 is not working. Look at the Windows 8 picture 5 messages back from this one, count the Tabs in the picture, Now look at the result in the following two images... only one Tab out of 5 Sync'ed. This is a _failure_ of Sync-Tabs with OwnCloud:Mozilla-Sync 1.3.1.99.
I looked at @ogasser original problem statement, the first message in this thread, and am a little confused again. Is this thread only about Bookmarks inside folders are not being synced correctly? If this is true, then I agree, this thread can be closed and Sync-Bookmarks can be considered complete, and #99 should be merged.
However in @ogasser first message there is also a 1. suggesting other Sync-Data should be tested. Is this other testing a part of the condition for closing this thread? If Yes, then this thread can not be closed. If No, then close this thread and merge. Also, If No then another thread should be opened to address Sync-Tabs not functioning _with OwnCloud:Mozilla-Sync 1.3.1.99_ (also look 8 messages back where @ogasser shows Sync "Tabs" as a requirement. Is Tabs a Sync requirement in a General sense, or _is Tabs a requirement for this thread and this merge_?
Last message seems a little cranky... so let me put my experience to better use.
What I suggest is we treat (compartmentalize) this thread as a Bug Report raised by @alexboss for Bookmarks fixed by #99. This Bookmark Sync issue Bug Report can be closed and appears to be complete to all our satisfactions. Good work done by all!
After inspecting PHP code, I can see from a C/C++ programming background why @ogasser would expect prior code to work., but PHP apparently doesn't work exactly like C/C++ (why I haven't jumped in). All this is interesting stuff.
I suggest we end this thread and start another thread for Sync-Tabs which aren't working. Procedurally, this is how this thread started, and is how Bug Reports are typically generated (I'm trying to use Mantis for this because Mantis fits in a LAMP stack and integrates with Eclipse/Mylin).
Thoughts?
I merged PR 99 and closed this issue as it was mainly about fixing the bookmarks problem.
@craigarno Regarding not alls tabs showing up on the about:sync-tabs page: Tabs, which are already open on a Firefox instance will not show up on this page. So make sure that the tabs that are 'missing' are not already open in some tabs on this machine. If you sill feel that Tab Sync is not working as it should, you can of course open a new issue.
@ogasser with your additional explanation, and a little more investigation I will say Sync-Tabs is working. I did not find the information you provided here on the Mozilla site. I do wish this sort of information were displayed prominently in Mozilla Sync-Tab documentation. I had two assumptions about how Sync-Tabs work which were proved wrong with this exercise. This information has value.
I'm off to help figure out why Sogo Connector for Thunderbird/OwnCloud isn't Syncing (Merging) some addressbook/Contact entries. Hopefully we'll get it sorted out quickly so I can get back to other things of value. It's havoc when your infrastructure isn't working.
I found a nifty new Addon [Saved Password Editor 2.7.2] which also allowed me to test OwnCloud:Mozilla-Sync Sync-Addon and Sync-Passwords. Both Addons and Passwords are Syncing correctly under Windows 7, 8.1, and Linux.
The Saved Password Editor allows me to save all the unusual Username/Password combinations required by the new National Healthcare sites. These sites normally block Firefox from saving this information, and without Password Save, I was having trouble remembering them all (because each one is different).