extremely slow: xlog takes 30+ seconds to render, save and even "star" a note

[LordPraslea]

xlog is extremely slow and taking up to 30 seconds after saving a note, adding a "star"and oftentimes even loading new pages

Currently this is too slow as saving a page takes 30 seconds before it reloads
I've imported around 600 notes. Granted, some are text heavy and have a some tags in them
My original knowledge management sysstem has more than 2000 notes of varying sizes and most software I've used seems to work properly.

I'm trying to figure out why it is so slow.. Maybe the existing tags and/or the fact that it scans every word for every save?

If for example I rename a page It sometimes fails to rename it and locks indefinetely. Other times it finishes the renaming but when i click the new link it again takes 30 seconds+ to process.

What's the best way you envisioned that xlog would work in a large notes system?

Could the large number of notes be the issue?
Do you use subfolders or is everything in one big folder?

I did notice one thing on a specific page with a very well used keyword.. that it has arround 100 backlinks

Compared to hugo with markdown and autoreload this is a huge drawback

[emad-elsaid]

One or more of the extensions can be slow for your usecase. I deliberately didn't optimize the performance until I need to. One way to check which extension is the reason behind this is to build a custom xlog without any extensions. Check if the issue still exists. And if not then add one extension at a time until you pin it down. The way to build a custom xlog is described here: https://xlog.emadelsaid.com/Custom%20installation/

…

On Sat, Dec 9, 2023, 12:56 PM Andrei ***@***.***> wrote: xlog is extremely slow and taking up to *30 seconds after saving a note*, adding a "star"and oftentimes even loading new pages Currently this is too slow as saving a page takes 30 seconds before it reloads I've imported around 600 notes. Granted, some are text heavy and have a some tags in them My original knowledge management sysstem has more than 2000 notes of varying sizes and most software I've used seems to work properly. I'm trying to figure out why it is so slow.. Maybe the existing tags and/or the fact that it scans every word for every save? If for example I rename a page It sometimes fails to rename it and locks indefinetely. Other times it finishes the renaming but when i click the new link it again takes 30 seconds+ to process. What's the best way you envisioned that xlog would work in a large notes system? Could the large number of notes be the issue? Do you use subfolders or is everything in one big folder? I did notice one thing on a specific page with a very well used keyword.. that it has arround 100 backlinks Compared to hugo with markdown and autoreload this is a huge drawback — Reply to this email directly, view it on GitHub <#45>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAANJA4PUKVXS4NXJJPAV2TYIRGV7AVCNFSM6AAAAABANXW6Q6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTGOBSGAZDSMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[LordPraslea]

Here are my findings:

It seems that "github.com/emad-elsaid/xlog/extensions/autolink_pages" is the principal major culprit. At the same time it seems to be the basic magic behind the whole xlog system, isn't it? Without it it seems that many features no longer work.

However, with all other extensions enabled and autolink_pages disabled, it sometimes still manages to get to 8-9 seconds (even on load, todo and other small things) -

• hashtags is probably the other culprit - Since disabling it loads much faster
• recent page is also slow (yeah, 500+ pages..) -

So 2 of the most usefull (i'd reckon,probably most used) extensions are causing the massive slowdown. From your experience, what could be a quick fix or solution?

Without delving into the code, I'm thinking some usage of a in memory DB or even something such as sqlite could maybe help with indexing in development. For the build phase, that would probably not help at all. What would be your take on that?

[LordPraslea]

One other thing I noticed is that with those extensions enabled all other access requests (ex opening 3 links in new tabs) have to wait. This makes me think that everything is waiting untill that specific request is done processing. I'll take a look next week to see how this (if possible) could be improved

[emad-elsaid]

I think profiling should show us which functions takes more time. Optimizing without knowing which function may not solve the problem. One quick thought is that autolinking are trying to find any pages name in any part of the text. So the longer the names and more pages you haveand the longer the text the slower it is. One way could be doing it in the background instead of waiting for its result. That makes it faster but also makes the feature result not work right away

…

On Sat, Dec 9, 2023, 1:56 PM Andrei ***@***.***> wrote: One other thing I noticed is that with those extensions enabled all other access requests (ex opening 3 links in new tabs) have to wait. This makes me think that everything is waiting untill that specific request is done processing. I'll take a look next week to see how this (if possible) could be improved — Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAANJA65MNYY27HJTACK2ILYIRNXZAVCNFSM6AAAAABANXW6Q6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBYGQYDEMRXGI> . You are receiving this because you commented.Message ID: ***@***.***>

[emad-elsaid]

I would like to be able to regenerate this issue on my machine, is there a way to get the same files ? is it public site or is it private?

[emad-elsaid]

I just remembered autolink extension does 2 things:

• search current page for any occurrence of any other page name to convert them to links
• search all other pages for current page name to print the list of related pages at the bottom

and Hashtag extension does no 2 too to link related pages.

so removing no 2 should cut the time significantly in exchange for removing the related pages at the bottom.
no 2 is defined in: https://github.com/emad-elsaid/xlog/blob/master/extensions/autolink_pages/autolink_pages.go#L29
and for hashtag : https://github.com/emad-elsaid/xlog/blob/master/extensions/hashtags/hashtags.go#L28

there is no way to unregister a widget that's registered. so one way to test the hypothesis is to fork xlog, remove the lines and run it on your directory.

[LordPraslea]

Hi

Most if not all notes are private (intellectual property) so I cannot share them. However one way you could probably achieve this is to take 500~700 books from project gutenberg for exampe (or even generate 100 kb lorem ipsum notes).
Almost all have 1-3 tags and every now and then a link. Not much complexity. However by looking at the code it does indeed seem to reprocess everything after each save, which is quite inefficient.

I'm currently exploring other alternatives as I need to keep going.

[emad-elsaid]

I made some performance improvement the past couple days, it's not there yet but I noticed a good improvement in my usecases at least. I' scratching my own itch as you see 😃

[ckyb]

I ran into this issue with one of my pages.
Debugging this by simply removing lines from the markdown file, I found out that a link to an external web page was being loaded, and that website (opsist) is (currently) down.

After disabling the link_prefix extension, this timeout no longer affected me.

I'm not a fan of the link preview extension processing each url (and social media) - for my use cases I'm making a white list of links to generate previews for (and maybe youtube -> youtube-nocookie, etc)

edit:
Just saw that youtube-nocookie is used already :)

[emad-elsaid]

I have made a set of optimizations recently that reduced build times. I'm not sure if we should keep this issue.

As it's slightly generic I'll close it, feel free to ask to reopen it if you still think it's needed.