Timeout on page with large srcsets

[mrflix]

Thank you Jonathan for all the work you put into this plugin and thanks for making it public :)

I have the following problem when using it:
When running it on a page with lots of large images and image versions the generator is very slow, shows no progress and eventually dies.

Expected Behavior

The static files should get generated.
A progress indicator would show the progress of the media generation, like imagekit does:

Current Behavior

The generator stops unexpectedly after some time:

One weird thing I noticed: the size of the media directory goes up and down:

I expect it to go up because missing thumbs are being generated. But since it's going down I suspect that it's deleting and re-creating already created thumbs. This would explain why the generator doesn't eventually process the pages faster and come to an end without a timeout.

Possible Solution

• maybe there is a way to prevent the process from deleting already created thumbs
• since the total amount of pages and media is deterministic there is nice potential for a progress bar (x pages of x, y media of y) to better understand whats happening

Steps to Reproduce

This is my srcsets config:

'srcsets' => [
  'default' => [640, 1280, 2560, 3840, 5120],
  'rondel-product' => [320, 640, 1280],
  'gallery' => [640, 1024, 1440, 2560],
  'slideshow' => [1280, 2560, 3840, 5120],
  'slideshow-portrait' => [640, 1024, 1280, 2560],
  'product-menu' => [128, 256, 512, 1024]
]

When I remove all use of srcsets and just use one resized version per image the generator works as expected and doesn't timeout.

Context

Additional plugins: builder, color, focus, matomo. Of all of them focus might be the only one that interferes.

[jonathan-reisdorf]

Thank you, I'll see what I can do here. The way it's structured is to render all these pages in one row. Restructuring this is some work but not impossible.
Not quite sure though why the MB in your folder are decreased from time to time - no files are deleted by the plugin except in the beginning, so I'm guessing it's either Kirby or a bug in your file manager.

[scsskid]

increase php max execution time should solve the problem, the default in most envs is 30s

[cvholten]

Hello,

I have got 1.6 GB of media files. I am interrested in any solution as well. After some minutes of generating static pages I get the info "The JSON response from the API could not be parsed. Please check your API connection."

[mrflix]

The latest Kirby 3.2.3 changelog says:

Media files are now correctly generated again in multi-site setups

Maybe that fixes it.

[jonathan-reisdorf]

Hi @rs-vonholten, that is a lot of media files. Can you tell me what the request responds with when you get this error? To do so, go into your browser's developer tools and open the network tab, then once you see the error, click on the corresponding request and open the response tab.

[cvholten]

Hi @rs-vonholten, that is a lot of media files. Can you tell me what the request responds with when you get this error? To do so, go into your browser's developer tools and open the network tab, then once you see the error, click on the corresponding request and open the response tab.

I get an error page by my host as response:

<title>Sehr geehrter Besucher</title>

_________________________________________________________ 

Sehr geehrter Besucher, leider ist ein Problem aufgetreten. Die angeforderte Seite hat einen Script-Fehler verursacht. Haben Sie sich vielleicht vertippt oder eine alte URL aufgerufen? Wenn nicht, informieren Sie bitte den Webmaster dieser Homepage per Email. Um zu der vorherigen Seite zurückzukehren, verwenden Sie bitte einfach die "Zurück" - Taste Ihres Browsers.

_________________________________________________________

[jonathan-reisdorf]

Then it's probably either a script timeout or some other resource-related problem. Hard to tell if the actual PHP error is not displayed. (Can you enable the displaying of PHP errors?)
Resolving this issue will take a while because it is a complex problem (because there are three ways to use the plugin, and any change in one of these should not break the others).
Also, so far I've only used the plugin on sites with a negligible amount of media assets.

[cvholten]

Then it's probably either a script timeout or some other resource-related problem. Hard to tell if the actual PHP error is not displayed. (Can you enable the displaying of PHP errors?)
Resolving this issue will take a while because it is a complex problem (because there are three ways to use the plugin, and any change in one of these should not break the others).
Also, so far I've only used the plugin on sites with a negligible amount of media assets.

Is it possible to exclude the entire media folder from being copied, so I have to copy it manually?

[jonathan-reisdorf]

Not yet, but adding that would be an option (also particularly handy if you have your own implementation, e.g. where you upload new images to a CDN like cloudinary and the images in kirby are not relevant). I'll create a PR for that later today.

[jonathan-reisdorf]

The option will be added with #19

To skip copying the files, add 'skip_media' => true to your config.
In this case however, any media folder present in the output directory will always be erased.
To prevent that, combine it with preserve:

'preserve' => ['media'],
'skip_media' => true

Also, when making any change to media files, you'll now need to manually take care of regenerating and copying them (maybe a kirby file hook would work?)

[cvholten]

The option will be added with #19

To skip copying the files, add 'skip_media' => true to your config.
In this case however, any media folder present in the output directory will always be erased.
To prevent that, combine it with preserve:

'preserve' => ['media'],
'skip_media' => true

Also, when making any change to media files, you'll now need to manually take care of regenerating and copying them (maybe a kirby file hook would work?)

Thank you. This worked.

When I do not skip media, the media folder content will be deleted everytime I use your plugin. Is it necessary to delete all files over and over again?

[jonathan-reisdorf]

@rs-vonholten yes, because we do not know / keep track if images have been changed. See also last comment of #3
(when the mentioned issue is solved in Kirby, changing this might be easier)

[jonathan-reisdorf]

Hi everyone 👋 Sorry for the belated response, but I've now released a version https://github.com/d4l-data4life/kirby3-static-site-generator/releases/tag/1.0.10 which would drastically improve rebuild times for sites with lots of media, because the media cache is now finally taken into account. Please let me know if that works and helps to reduce this issue. The initial build might still be slow if the media cache is not populated, yet, but follow-up builds should be way faster.

[jonathan-reisdorf]

Closing this because with the latest update the issue seems fixed on subsequent runs. Feel free to reopen if there's an issue.