[FEATURE]
Closed this issue · 1 comments
Is your feature request related to a problem?
While a warmup session is running on my site of about 650 pages the server loadav jumps high and the frontend responds slowly or may return 500 error
Describe the solution you'd like
A configurable option for rate-limiting the warmup would be useful. e.g.
- maximum pages to be warmed per second
- or, wait-time between consecutive pages (milliseconds)
Describe alternatives you've considered
No response
Additional context
For example, on my server I would set a rate of one or two pages per second, giving me an acceptable run-time of 5-10 minutes with lower load instead of the 90-100 seconds it is taking at present which cause a high loadavg
Or I could set a wait time of 500 or 1000 milliseconds to achieve the same result
Code of Conduct
- I agree to follow this project's Code of Conduct.
Hi @srchild, thanks for your issue. This is already possible, see #199 (comment) for an example. In general, you can use all Guzzle request options within crawler options {"client_config": {/* ... */}}. We also discussed this topic on Slack, see https://typo3.slack.com/archives/C0400CSGWAY/p1673427975963929.