GIScience/ohsome-py

Improve logging

SlowMo24 opened this issue · 6 comments

SlowMo24 commented

Logging errors works. Splitting logs into separate files would though improve readability.

I would suggest one file for each of the following

  • general logging (currently done to the console)
  • ohsome response
  • ohsome query parameters
  • ohsome bpolys parameter (if used)

the bboxes parameter is geojson but gets invalid when duped to json due to escaping. Also the geojson may get quite large even with a small amount of geometries making the log hard to read (see ohsome_exception_20210312T165323.json.txt)

This is some code I use for logging but that may not be good code!

#!/usr/bin/python

import os
global filepath
filepath=os.path.dirname(os.path.realpath(__file__))
import logging
logging.basicConfig(filename=os.path.join(filepath,'temp_automatic_workflow.log'),filemode='w',level=logging.DEBUG,format='%(asctime)s %(levelname)s %(message)s',force=True)
redfrexx commented

Good idea! I agree that the bpolys parameter should be written to a separate file. Why would you split the ohsome response which contains timestamp, error code, message, requestURL and the query parameters in two different files?

SlowMo24 commented

I thought of the response and the request as two different things. query parameters + (if applicable) bpolys.geojson represent what I want and can be used to reproduce (or not, if temporary or not due to request) the error.

The response on the other hand is was annoyed the API and how it tries to help me solve the problem.

The Idea was to create a file (e.g. query_parameters.json) that I can present to e.g. curl without further formatting and that will reproduce the same query.

redfrexx commented

The bpolys parameter is now written to a separate file. I agree that it would also be nice to have an additional file using which the query can be easily reproduced. I don't have the time right now to look into it though, so if you have an idea of how this file should be formatted, please post it here.

SlowMo24 commented

I don't think you have to change a lot just enter another line where you copy the 'bpoly' code to adapt it for the body-input. Unfortunately I did not yet have a close look at your code but here is mine:

So I do general logging as stated in the issue above and then do body and bpoly logging as follows:

#body is the requests post body without the bpolys parameter
#aoi is the bpolys geoemtry in pyhton-dict format

#save body for debugging
with open(os.path.join(filepath,'temp_body.json'),'w') as outbody, open(os.path.join(filepath,'temp_bpolys.geojson'),'w') as bpolyout:
    json.dump(body,outbody)
    json.dump(aoi,bpolyout)

body["bpolys"] = json.dumps(aoi)

logging.debug("commit ohsome query")
response=requests.post(URL, data=body)

You can then continue to write the response (if any) to the fourth file as you already do:

try:
    response.raise_for_status()
except HTTPError as e:
    # write response to file as usual
    if response.text:
        raise HTTPError('{} Error Message: {}'.format(str(e), response.text))
    else:
       raise e
SlowMo24 commented

PS: I'm sure you already do that but I always forget so: Please ensure formatting of the json e.g. with 4-space-indent