GIScience/ohsome-api

Accept application/json as data input

Opened this issue · 10 comments

SlowMo24 commented

JSON is a widely used format that can be processed in many applications. Accepting JSON as input would increase the usability of the API.

For example https://github.com/GIScience/ohsome-py is logging the data body input it sends to the API in JSON format if the query fails. If the API accepted JSON the logged files could be directly used to reproduce the failed query using curl:

curl --data-urlencode "bpolys@the_bpolys.geojson" \
    --data-urlencode "@the_body.json" \
    -X POST https://api.ohsome.org/v1/elements/count/groupBy/boundary \
    -H "Content-Type: application/json"

Currently manual interaction is necessary to rewrite the query as follows:

curl -d "time=2007-10-08T00:00:00Z,2019-05-04" \
    -d "clipGeometry=false" \
    ...
    --data-urlencode "bpolys@the_bpolys.geojson" \
    -X POST https://api.ohsome.org/v1/elements/count/groupBy/boundary

temp_body.json.txt

temp_bpolys.geojson.txt

joker234 commented

Thanks for your issue. I would agree that accepting JSON instead of parameters is something the ohsome API should be able to do. But I'm not quite sure if you are able to use multiple -d (or --data-urlencode) in a cURL request. Maybe you still have to merge the two JSON files together.

tyrasd commented

But I'm not quite sure if you are able to use multiple -d (or --data-urlencode) in a cURL request.

it does work with the (default) form url-encoded content, but not with JSON as curl wouldn't know how to combine these into a single valid body.

so, the first request would rather look like this:

// manually combine "the_bpolys.geojson" and "the_body.json" into "combined__the_bpolys__and__the_body.json"
curl --data "combined__the_bpolys__and__the_body.json" \
    -X POST https://api.ohsome.org/v1/elements/count/groupBy/boundary \
    -H "Content-Type: application/json"

because of the needed manual intervention (which even requires the creation of an additional file), I don't see any particular benefit for the mentioned use-case of using the logged json content directly for a curl command for debugging. Maybe it would be better for suggesting a feature in ohsome-py to output debug curl commands for issuing curl requests using url-encoded data?

SlowMo24 commented

Just FYI: The reason for ohsome-py to log into seperate files is that

  • the produced bpolys.geojson can be directly loaded into GIS for analyses
  • the bpolys.geojson may eventually get very large and hinder the extraction/analyses of the other parameters in body.json
tyrasd commented

the produced bpolys.geojson can be directly loaded into GIS for analyses

Yes, this makes sense 👍

body.json

Maybe a small addition to my previous proposal: ohsome-py could just output a body.txt (or params.txt) instead of (or in addition to) the body.json, which could then be used in a oneliner curl statement. Like in this example: https://gitlab.gistools.geog.uni-heidelberg.de/snippets/7 – I think it should not be hard to create this format in python (somethink like this should do the trick: https://stackoverflow.com/a/1233551/1627467).

tyrasd commented

JSON is a widely used format that can be processed in many applications.

This is true.

Accepting JSON as input would increase the usability of the API.

Would it really? I would assume that every tool which can perform POST requests with JSON payloads would also be able to perform the same request with a url-encoded (x-www-form-urlencoded) payload (if anything, tools like curl sometimes have built-in support for x-www-form-urlencoded data, but don't offer anything special for JSON data). So there would not be any added benefit from that side. Am I overlooking something?

SlowMo24 commented

Am I overlooking something?

I don't think so, I just felt the urge to justify my issue ;-) The increased usability may though result from having more than one option for the user, or does accept/json mean not-accept/form-urlencode?

SlowMo24 commented

@tyrasd and @joker234 could you discuss this internally (e.g. with @FabiKo117 ) and then notify and support @redfrexx about what would be the best solution (e.g. here: GIScience/ohsome-py#42)?

  • 👍2
tyrasd commented

or does accept/json mean not-accept/form-urlencode?

no of course not 😉 however, implementing your suggestion does not come without cost, since there's always added code complexity involved, extra tests need to be maintained, builds would run slower, etc.

I would be a bit less hesitant to give this higher priority if the JSON format would give us something extra, which is not possible without it. Thinking of it… maybe it could: Since JSON is more flexible than the simple url-encoding of a list of parameters, it would open some options. Just scratching the surface, this would come to my mind:

  1. use a unified data structure for the bboxes/bpolys/bcircles, perhaps like this: 
      "boundaries": [{
    "id": "bbox 1",
    "type": "bbox",
    "coordinates": [0,0,1,1]
  }, {
    "id": "bcircle 2",
    "type": "circle",
    "center": [47, 11],
    "radius": 100
  }, {
    "id": "bpoly 3",
    "type": "polygon",
    "geometry": {…}
  }]
  2. less custom ohsome-api specific encoding magic, e.g. properties as boolean flags, instead of comma separated list 
        "showMetadata": true
  3. use proper json data types instead of all strings, e.g. true/false instead of `"true"/"false" 
        "properties": { "metadata": true, "tags": false }
  4. time parameter as object with dedicated from, to and period; or an array of timestamps.
  5. or maybe even allow some enhanced version of GeoJSON data to be used as the body which can hold all required parameters: 
    {
  "type": "FeatureCollection",
  "features": […],
  "ohsome-query": {
    "time": …,
    "filter": "…",
    "properties": …,
    …
  }
joker234 commented

@tyrasd and @joker234 could you discuss this internally (e.g. with @FabiKo117 ) and then notify and support @redfrexx about what would be the best solution (e.g. here: GIScience/ohsome-py#42)?

This enhancement will be brainstormed in discussions about the next major ohsome API release (2.0).

  • 👍1
FabiKo117 commented

Additionally to points 1 and 5 of Martin's comment above, further ideas regarding parameters: Endpoints like /elements/count/density could also be restructured to using /elements/count or just /elements and having e.g. density = true as additional parameter. This could also enable to have a more flexible grouping structure, so that any supported grouping structure would be possible, e.g. groupByTagGroupByBoundary or also the other way around groupByBoundaryGroupByTag.

We would have to further think then though how or if we want to continue to support the current structure as it is for the GET requests, if it should also e.g. be possible to give just one JSON as single parameter for GET requests.