hegerdes/joplin-plugin-remote-note-pull

Sync stops working

blokpardi opened this issue · 10 comments

blokpardi commented

Love this plugin, but after a while it just stops syncing. I setup the sync note and it pulls the md file. Then I can manually sync it up for a while. But after some period of time it just stops working and I have no idea why. I've confirmed the link to the destination file works, there's no network issue, and I've kept Joplin open the entire time. Any ideas where I can start to look to debug this?

I'm using:
Ubuntu 22.04
Joplin 2.10.19
Remote-note-pull 1.0.2

github-actions commented

Hi, thank for reporting an issue, we will check it out very soon

hegerdes commented

Hi @blokpardi, thanks for using my plugin and I'm happy that you like it.

To narrow down the Problem can you confirm that the Enable periodic sync setting in the settings is checked and that you have a reasonable sync Intervall.
What happens if you press the upper right downarrow button:

image

If you are willing, can you please share how many site do you sync and provide an example site where it stopped working?

Little Disclaimer: I did not test it on Ubuntu and I'm not primarily a JS developer. Also found using the Joplin API a little off form time to time. But I will do my best to fix any problem that occur with this Problem when time allows me to

blokpardi commented

Hi @hegerdes. Yes, I have Enable periodic sync checked, and the interval is the default 30 min. RIght now I just have one document syncing.

I did some ad-hoc tests and here's what I noticed:

  • After setting up the sync, I can make changes clicking the arrow in the upper right syncs the document. I can keep doing that and it seems to work. Each time I get a confirmation dialog that the sync completed.
  • After the sync interval expires, the auto-sync doesn't work (changes to the source are not updated in Joplin) and the arrow button has no effect. It doesn't sync the document and I don't get a confirmation dialog.
  • I can reset everything by killing and restarting Joplin.

Just a guess but it seems to have something to do with the auto-sync feature. My tests haven't been very rigorous, but it looks like something is going wrong when it hits the auto-sync interval.

How does the timer work? Could it be getting stuck when that fires? Happy to do more testing if we think that might be a place to investigate.

UPDATE: I tried this on my Windows machine as well (latest Windows 11 and Joplin 2.10.19). I set the auto-update interval to 10 min and that has not worked at all. It never automatically updates. However, I can consistently manually update and so far that has not failed.

hegerdes commented

I've looked into this and did not find a actual error but there are some things to improve as well a limitation form the Joplin App/API.

This is how it works:

  • On startup the plugin registers itself by
    • Setting UI elements
    • Register Settings/Load them
    • Register the functions
    • Registers a setInterval(updateNotes, intervall)
  • Every time the interval hits it gets all notes and searches for the footer paddern
  • If it exists it updates the note with the url metadata prop from each note. This can be found by hitting the i symbol next to the downarrow for updating the note. The URL in the footer is not used.
  • If the note is a sync note it gets entirely replaced

I found the following problems:

  • Updating one note and every note use the same function. I just use a for loop for every note. In the current version I do a double regex reach wich is expensive. -> Will be fixed in the new version
  • The default interval my be a little short since, depending on the note count this is a lot of work to do. -> will be updated to 120mins. Existing settings/installations will not be affected. The will also be a sleep between each note update for a more controlled loop and to avoid hitting eventual API limits.
  • When the note updates and new content is set, it is not directly visible in Joplin since a Note update does not trigger a UI refresh. This is related to laurent22/joplin#5955. There is not much I can do about it now. As a work-around you can select a different note and then the original one again. Now Joplin should display the up-to-date state.

I thought about adding a timestamp to every sync note which shows the last time it was updated but I'm not sure if this is smart since it would create a new revision of every sync note every time the update command is invoked.

Also I'm not a 100% sure how the setInterval(updateNotes, intervall) is handled by Joplin/NodeJS since every plugin runs in its won worker-process. There even might be differences on Linux/Win/Mac

  • 👍1
blokpardi commented

On my linux machine I uninstalled and reinstalled Joplin. Now the plug-in is behaving just like the Windows machine:

  • It will never auto-update
  • I can manually update anytime

So that's progress I guess :) I have quite a few notes (nearly 2000) so perhaps that's the issue. I'll look forward to your next update and see if that helps.

hegerdes commented

I've just published a new release with the before mentioned fixes. I don't know when Joplin will pick up the update since this is the first fix since the original release, but I hope it will not be too long. If you cant wait, the package is also available as a GH-Release.

I hope this fixes your problem - if not just post an update here.

blokpardi commented

Hi @hegerdes. I downloaded the zip, removed the old version (restarted Joplin), installed it from file (restarted Joplin), but no change. I can manually update but it will not automatically update, and I waited well past the sync interval and clicked away from the note and back to make sure. Perhaps you can confirm I have the right code. I got it from here https://github.com/hegerdes/joplin-plugin-remote-note-pull/releases/tag/v1.1.0, but the add-in is still showing as version 1.0.2 in Joplin and the json file in the zip is also showing "version": "1.0.2".

hegerdes commented

Sorry, work is time consuming right now 😑

Turns out Juplin build & publish does ignore the version set in the package version. Build process will be fixed in the future.
But you have already the newest version (even if the version bump is not seen).

Joplin also has quite a bad design for logging of plugins. (No central logger and plugins run as a own worker porzess that is not shown in the console) In the normal mode I only get Plugin found, Plugin load, Plugin started. All the logging I do in the plugin is only seen in the dev mode :/

If you feel comfortable, can you please start joplin from the terminal with the --env dev flag with my downloaded plugin in the settings section plugins -> Development plugins. This is a total separate, isolated instance of Joplin, so you will not see your existing notes.

Then you should see every log I produce and see if the periodic sync works. Feel free to post thees here.

hegerdes commented

@blokpardi I think i found the bug. Before I update all sync-notes I generate a list of all notes.

This function is flawed right now and creates an infinite loop:
image

This is also the reason my fans spin up to a 100 if I have joplin running long enough because the infinit loops steck up with each sync intervall. Never hapend in testing because I had to few notes, this only manifests if you have a higher number of notes. I will fix it as fast as possible and update hear.

Thank you for pointing me to this bug

  • 👍1
blokpardi commented

just catching up... I installed the new code, and it works great! In fact in repeated tests it updates almost instantly. Is that by design? I mean, it's great, but then what does the update interval do?

Thanks so much for getting this fixed!