implement multiprocessing by adding php proxy/wrapper
r9zzai opened this issue · 4 comments
not starting with topic because i had some troubles with installing py-qgis-server by pip. my colleguae had same problem. maybe of non supported qgis(server) version 3.16. succeeded with docker but in the original image from https://hub.docker.com/r/kartoza/qgis-server/ was a reference to an non existing file. so i had to replace
"py-qgis-server==${BUILD_VERSION}" py-amqp-client
to
"py-qgis-server" "git+https://github.com/3liz/py-amqp-client.git"
and then py-qgis-server ran flawlessly. Very nice is the project caching.
now to my question. your are using qgis python server from
https://qgis.org/pyqgis/3.16/server/QgsServer.html
and do requests with
https://qgis.org/pyqgis/3.22/server/QgsServerRequest.html
in my thoughts that should do the multiprocessing. Because of the behaviour that if i add a project to lizmap and then add layers from that project to a new project via the cgi with WMS in QGIS Desktop (https://url/to/qgis_mapserv.fcgi?map=path/to/my/map.qgs&service=WMS). Now with that project every layer which is embedded from that "internal/external" WMS source starts a new thread even on another core if that layers gets loaded on lizmap.
one bad thing about that is that if we add a layer with from our serverside qgis_mapserv.fcgi is that the user can see that internal url and (s)he cann add the project to their projects because there is no password etc. that does lizmap user interface make useless. so the scope is to use a php proxy which forwards the requested wms url to the "internal/external" qgis_mapserv.fcgi process. now we request only xyz.php and the url to the qgis_mapserv.fcgi remains unknown (attached php as log file).
the question is: can that behaviour be reproduced in py-qgis-server e.g. using subprocess and return a temporary php with nearly the same content as the attached file to achieve multiprocessing like in the aforementioned behaviour?
"py-qgis-server==${BUILD_VERSION}" py-amqp-client
This is really specific to docker images build for https://hub.docker.com/r/3liz/qgis-map-server . You don't need py-amqp-client unless yout intend to export servermetrics to rabbitmq. Just install py-qgis-server with pip as any other python module - better to install in venv.
I don't understant: are you trying to install py-qgis-server in the kartoza image ?
In my thoughts that should do the multiprocessing
This is Qgis api, what this has to do with multiprocessing ?
Now with that project every layer which is embedded from that "internal/external" WMS source starts a new thread even on another core...
Which thread ? From what sofware ? py-qgis-server don't use threads, it uses parallel processes, so it is already multiprocessing.
the question is: can that behaviour be reproduced in py-qgis-server e.g. using subprocess and return a temporary php...
It's not clear to me what you want to achieve: py-qgis-server is a OWS/WFS3 server built on to of the Qgis API, not a reverse pro
y, there is no reason to serve php.
one bad thing about that is that if we add a layer with from our serverside qgis_mapserv.fcgi is that the user can see that internal url
I can't figure what kind of url could possibly link from your internal project in the OWS responses.
May be you should rephrase the issue by beeing more specific about what you expect from py-qgis-server: either from its functionality or from the configuration.
"py-qgis-server==${BUILD_VERSION}" py-amqp-client
This is really specific to docker images build for https://hub.docker.com/r/3liz/qgis-map-server . You don't need py-amqp-client unless yout intend to export servermetrics to rabbitmq. Just install py-qgis-server with pip as any other python module - better to install in venv.
i tried and actually try to install py-qgis-server in venv:
python3.9 -m venv --system-site-packages env
but local qgis modules are missing (error on import qgis and import qgis.PyQt). even with:
export PYTHONPATH="${PYTHONPATH}:/usr/lib:/usr/share/qgis/python"
if i create venv with system python3:
python3 -m venv --system-site-packages env
there is reference to qgis modules (no error on import qgis and import qgis.PyQt). But i cant install py-qgis-server with:
python3 -m pip install py-qgis-server
in the docker image there py-qgis-server is installed with the systems python3. But i cant reproduce that on Ubuntu 16.04.7 LTS.
I don't understant: are you trying to install py-qgis-server in the kartoza image ?
originally i used the image on https://hub.docker.com/layers/3liz/qgis-map-server/3.24/images/sha256-989fa68ac2f49c7e2c4d031cb844e7554834b6af0ce74396fba678cbb36b5afa?context=explore
which did not work for me. i used dockerfile from image https://hub.docker.com/r/dduvnjak/dockerfile-from-image/ to look where the error comes from the image. well i made my own dockerfile with that two dockerfiles then what was working for me when changing the mentioned lines. but it would be nice to install py-qgis-server in an venv.
In my thoughts that should do the multiprocessing
This is Qgis api, what this has to do with multiprocessing ?
nothing. i was just thinking while digging into https://github.com/3liz/py-qgis-server/blob/master/pyqgisserver/qgsworker.py.
Now with that project every layer which is embedded from that "internal/external" WMS source starts a new thread even on another core...
Which thread ? From what sofware ? py-qgis-server don't use threads, it uses parallel processes, so it is already multiprocessing.
that behaviour is not related to py-qgis-server.
assuming i have a project in lizmap i can access it in QGIS Desktop via add WMS Source. That is by adding the project via qgis_mapserv.fcgi like in the attached php file.
the question is: can that behaviour be reproduced in py-qgis-server e.g. using subprocess and return a temporary php...
It's not clear to me what you want to achieve: py-qgis-server is a OWS/WFS3 server built on to of the Qgis API, not a reverse pro y, there is no reason to serve php.
to reproduce what i am trying to achieve one could create a qgis project and upload it to lizmap. now we can access that project in lizmap but also from https://url/to/qgis_mapserv.fcgi?map=/var/www/to/lizmap/maps/project/project.qgs&Request=GetCapabilities. in a new qgis project we can add the layers from the old project in the new project if we add them as WMS Source from the mentioned link (qgis_mapserv.fcgi). now if we load the new project with the layers from the old project added as WMS Source in lizmap all cores are used to load these layers. in tests on 4 core cpu all cpus were used (new qgis project) while on py-qgis-server only 2 were used in mean (initial qgis project).
to do that programmatically one could write a python script to copy the .qgs file and replace the layers sources to:
crs=EPSG:3857&format=image/png&layers=LAYERNAME&styles&url=https://url/to/qgis_mapserv.fcgi?map=/var/www/to/lizmap/maps/project/project.qgs?
maybe some parameters are missing here. it should only demonstrate what i am trying to achieve. another way could be to do that in py-qgis-server by writing temporary copies of the current .qgs file and referencing to that file in the request. But as filesize can be few MB that isnt be useable in production.
do you undestand what i trying to achieve?
one bad thing about that is that if we add a layer with from our serverside qgis_mapserv.fcgi is that the user can see that internal url
I can't figure what kind of url could possibly link from your
internalproject in the OWS responses.
if in the project a WMS source is added from an internel project via https://url/to/qgis_mapserv.fcgi?map=/var/www/to/lizmap/maps/project/project.qgs&Request=GetCapabilities then the url to https://url/to/qgis_mapserv.fcgi?map=/var/www/to/lizmap/maps/project/project.qgs is published in the network log in the browser. so one see that url the layers of that project can be added to their project without password etc. normally that does not happen because the request for a layer looks like this
https://url/to/index.php/lizmap/service/?repository=id&project=name&LAYERS=..
May be you should rephrase the issue by beeing more specific about what you expect from py-qgis-server: either from its functionality or from the configuration.
it is more functionality related. that is just a thought if my achieving could be implemented.
Sorry but I do not understand what your issue is about.
Ok thank you anyway