Model output format

[marioba]

Hi,
thanks a lot for this interesting project.

We are testing some QGIS models. In QGIS I can choose as output whether to create a temporary layer, generate a gpkg or save with another format (e.g. csv).

With py-qgis-wps, I always get the result as gpkg. If I define the parameter of the output file such as file.csv, a gpkg with a layer called "file.csv" is created instead of a csv file.

Is it possible to specify somehow what the desired output format should be?

[dmarteau]

Thank you for your interest,

What Qgis version are using ?

Can you give the response of a DescribeProcess request along with an Execute request response ? - it gives some informations about the underlying Qgis parameters used.

Note that destination parameters (as feature-sink) are stored server-side as layers in a temporary Qgis project from the generated output - the idea is that they may be served by a qgis server easily, so the WPS returned document should be a ComplexOutput with a link/href to a WMS requests and not file document whatever the format of the output is.

If you want the WPS to return a csv document then Destination Parameters are probably not the way to go, because it does not translate actually to a WPS file reference - something that we have to change in some way and we are working on it.

[m-kuhn]

Thank you for the response.

The Qgis version is 3.16.

The underlying parameter is QgsProcessingOutputLayerDefinition.

I am evaluating to add a condition in layersio.get_processing_value() and check if the input data starts with file:// and in this case use this as filename instead of the generated one, would that be acceptable?

[dmarteau]

I am evaluating to add a condition in layersio.get_processing_value() and check if the input data starts with file:// and in this case use this as filename instead of the generated one, would that be acceptable?

It is a bit more complicated, in this case the Parameter is QgsProcessingParameterFeatureSink and the value is a QgsProcessingOutputLayerDefinition holding the output format specification.

The input parameter value is the destinationName: i.e the name of the layer that appear in the result project, while the file is the output parameter name defined + the file extension.

The rational about this it is:

• For security reason your cannot specify a path from the client side, in the case of the models, the output path is hard encoded in the model definition and not a parameter for the model execution, which is somewhat what we want - Except that you define a path relative to your local qgis environnment and this has to change when running on server side.
• We want exposing results as usable with WMS/WFS request, so the input value need to be related to the layer that will be visible with WMS/WFS.
• Figure how to get informations about QgsProcessingOutputLayerDefinition held in the model definition and py-qgis-wps is agnostic toward models: this is only a provider as another (like scripts or R- scripts) - so maybe it is encoded in the defaultValue (this has to be checked)
• Convert the path to something safe server-side, for file this is easy, in case of database, this is much more complicated: remember that the process may be run concurrently, so outputs of each runs must be isolated from each, other. This cannot be guaranted if you take the path definition 'as is'

So we may have to select another approach: I think that we still need to overcome the QgsProcessingOutputLayerDefinition in order take into accounts the points above but we may provide an options for returning a file content or référence instead a WMS link and maybe some more server options for controlling how output is managed server-side.

[m-kuhn]

The scenario we are having here is, that we are using it as a "pure" algorithm processing service. It's internal only and results will be further processed by different parts of the application, ideally not consumed through a service but file based.

For input data it's already now possible to provide absolute paths and db/postgres connection strings (since you have been talking about security, I guess this would also have to be taken care of).

I think that we still need to overcome the QgsProcessingOutputLayerDefinition

I am not sure if you mean "replace it with something else" or "make better use of it". The advantage of QgsProcessingOutputLayerDefinition is that you can develop an algorithm on qgis desktop and reuse as-is, so sticking to it sounds great.

Being able to specify relative paths instead of absolute ones for better isolation would also be an option. But in the end an application can also ensure on another level if conflicts are possible (e.g. here jobs are triggered every 24 hours and can overwrite previous results). I think both, relative (to the current job folder) and absolute paths have their advantages.

In short, an option to release the constraint about passing paths, as you proposed in #15 would make a lot of sense to me. Should the file:// prefix be kept in this case? With an explicit parameter we could also say that <wps:LiteralData>result.csv</wps:LiteralData> goes to the job folder, <wps:LiteralData>/io/result.csv</wps:LiteralData> goes to an absolute path (consequences to be taken care of by the developer) and <wps:LiteralData></wps:LiteralData> an empty string (or a special command?) will generate the output as it currently is.

[dmarteau]

I am not sure if you mean "replace it with something else" or "make better use of it".

Sorry, I used the wrong word. I mean sticking to it but without taking into account the parameters given by the model.

is that you can develop an algorithm on qgis desktop and reuse as-is

That is the whole idea.

In short, an option to release the constraint about passing paths, as you proposed in #15 would make a lot of sense to me.

I'm going to work on it unless you want to amend #15

Should the file:// prefix be kept in this case?

I think we should support it but not make it mandatory (I think of a larger use of protocols)

[m-kuhn]

I am not sure if you mean "replace it with something else" or "make better use of it".

Sorry, I used the wrong word. I mean sticking to it but without taking into account the parameters given by the model.

is that you can develop an algorithm on qgis desktop and reuse as-is

That is the whole idea.

Perfect

In short, an option to release the constraint about passing paths, as you proposed in #15 would make a lot of sense to me.

I'm going to work on it unless you want to amend #15

If I have confirmation (and additional requirements if any), I can do it, but I'm sure you are more efficient, so I don't insist ;-)

Should the file:// prefix be kept in this case?

I think we should support it but not make it mandatory (I think of a larger use of protocols)

postgres:// is already supported (for input parameters), but handled in qgis/processing. I think handling file:// on python level is not the preferred way, more an attempt to make it explicit that we want to bypass the default handling.