limit for preload config projects?

Question

limit for preload config projects?

lassitanskanen opened this issue 2 years ago · 8 comments

Hi,
Is there limit for preload config projects? How about the code behind this setting? The server seems to get stuck with ~20 qgis projects because it loads all projects in same time while starting.

Answer 1 · 2022-08-17T18:09:12.000Z

Hi,
There is no limit for preload projects since you define explicitely the list of preloaded projects.

The server seems to get stuck with ~20 qgis

Is it really stuck or this is just because projects tooks very long time to load ?.

The purpose of preload projects is to load projects that could take very long time at startup time to prevent blocking your workers if let the default lazy 'on-demand' loading process doing the job: the results is that you must expect a longer startup time.

There is no fit them all strategy, depending on your configuration, you have to mitigate between preloading, asynchronous loading, timeouts settings and eventually sharding using multiple pool of workers with a reverse proxy.
.

Answer 2 · 2022-08-17T18:12:53.000Z

Thanks for quickly response!

If the startup take a long time, is there way to preserve static cache? I'm working with docker environment.

Answer 3 · 2022-08-17T18:16:35.000Z

is there way to preserve static cache

Unfortunately no, Qgis does not handle shared memory between processes and a project is a complex structure is the Qgis codebase.

Answer 4 · 2022-08-17T18:38:51.000Z

Okey, how about update static cache from management api without restart server? If use POST /pool/restart, does it update project cache also?

Answer 5 · 2022-08-18T13:17:03.000Z

how about update static cache from management api without restart server

You have two options:

From managment: POST /pool/restart will restart the workers gracefully and thus the static cache
By using a restart monitoring file: see https://docs.3liz.org/py-qgis-server/configuration.html#server-restartmon

Answer 6 · 2022-08-19T05:38:11.000Z

Thanks! I will test these options.

Answer 7 · 2022-08-25T18:38:06.000Z

Related this case, I tested little bit HEALTCHECK CMD for Dockerfile. It looks work well but my implementation was quite poor.

Is it difficult to add "static cache ready" or "workers ready" endpoint to management api? I tested with this shell script:

#!/bin/sh
response=$(curl --silent http://localhost:19876/pool)
if [ "$response" = '{"num_workers": 4, "workers": []}' ]; then
exit 1
else
exit 0
fi

and put this to Dockerfile
HEALTHCHECK CMD /bin/sh /healthcheck.sh
COPY ./healthcheck.sh /

Above could be simplier:
HEALTHCHECK CMD curl --fail http://localhost:19876/pool/ready || exit 1

If apply healthchecks, it could be easier roll-update docker service containers without cut-offs.

Answer 8 · 2022-10-14T18:55:30.000Z

Healthcheck can be specifiied at runtime (you may refer to docker compose documentation).
Since how your may consider your container ready may depends on many parameters and execution contexh, I would advise you to not hard code HEALTCHECK in the image.

Is it difficult to add "static cache ready" or "workers ready" endpoint to management api? I tested with this shell script

It should not be very hard, please consider opening an enhancement issue for tracking this feature request.