Tool to help reverse-engineer legacy HTTP APIs.
This package was born of a need to be able to use multiple different kinds of existing, undocumented legacy HTTP APIs not following any kind of consistent, well planned design. It was also thought helpful to be able to add a little bit of documentation about what those legacy APIs do.
This package is NOT meant for defining new APIs. Use e.g. Swagger for that.
Why then not just use Swagger or some other such tool? They are really meant for creating new APIs from scratch and as such cater to a bit different use case. For example they tend to be geared toward the verbose. When reverse-engineering and documenting existing APIs, all the details are not that important. We just need to make it easy to use the APIs and be able to add an explanation of what they do, rather than documenting everything.
The examples hopefully clarify the difference and some of the benefits of this package.
Note that this package does NOT make HTTP requests using some client library. That is up to you; use something from the Python standard library, or the 'requests' package, or something asynchronous, whatever.
Note: these examples are illustrative. For working examples, see the tests.
Simple example
An example API definition in YAML that specifies two operations for querying single and double rooms reservations, respectively:
label: Hotel API
description: An API to check room reservations
operations:
list-singlerooms:
label: List single room reservations
description: List all reserved single rooms
request:
method: GET
path: /hotel/reservations
params:
size: single
response:
type: application/json
parser: hotelapi.util:parseresponse
list-doublerooms:
label: List double room reservations
description: List all reserved double rooms
request:
method: GET
path: /hotel/reservations
params:
size: double
response:
type: application/json
parser: hotelapi.parseresponse
This is similar to how many specification syntaxes express HTTP APIs. Clear, but often lots of boilerplate and repetition. Parses into a plain dict using PyYaml as-is. Now let's see how to save some effort.
Using Jinja templating for API spec expansion
The API document can be expanded using Jinja2 templating. Using our room reservation example, we could generate an API operation for each room size variation:
operations:
{% for size in sizes %}
list-{{size}}-rooms:
label: List {{size}} room reservations
description: List all reserved {{size}} rooms
request:
method: GET
path: /hotel/reservations
params:
size: {{size}}
{% endfor %}
Two different API operations would be generated, such as with this code, assuming the api spec has been read into a string variable called 'yamlsource':
>>> from httpreverse import expand_jinja
>>> expanded = expand_jinja(yamlsource, context={"sizes":["single", "double"]})
>>>For blunt copying of parts of the YAML document to another place, the standard YAML anchor/alias mechanism can of course be used as well.
Templated request specifications
Besides Jinja templating, a custom templating mechanism is provided for request
and response specification convenience. Here's an example with a roomapi
request/response template that is used to move repetitive request and response
specifications into a common template, referred to from the actual specs:
label: Hotel API
description: An API to check room reservations
templates:
roomapi:
request:
method: GET
path: /hotel/reservations
response:
type: application/json
parser: hotelapi.parseresponse
operations:
list-singlerooms:
label: List single room reservations
description: List all reserved single rooms
template: roomapi
request:
params:
size: single
list-doublerooms:
label: List double room reservations
description: List all reserved double rooms
template: roomapi
request:
params:
size: double
Here's how to apply the request/response template in Python:
>>> from httpreverse import apply_template
>>> api = yaml.load(yamlsource)
>>> templates = api["templates"]
>>> operation = api["operations"]["list-doublerooms"]
>>> applied = apply_template(operation, templates)
>>>Simple parametrization
The API definitions can also be parametrized for convenient run-time use. The parametrization function accepts an optional context argument that is simply a dictionary that is used to assign values to all the named parameters found in the operations. Parameters are prefixed with the dollar sign ('$'). So it would be possible to also specify a single dynamically invoked operation for listing the rooms:
operations:
list-rooms:
label: List room reservations
description: List reserved rooms
template: roomapi
request:
params:
size: $size
By passing either {"size":"single"} or {"size": "double"} as context,
room size values would then be assigned:
>>> from httpreverse import parametrize
>>> api = yaml.load(yamlsource)
>>> operation = api["operations"]["list-rooms"]
>>> parametrized = parametrize(operation, context={"size":single})
>>>More complex parametrizations are possible using the same simple mechanism:
operations:
add-reservation:
label: Add reservation
description: Add a room reservation
template: roomapi
request:
method: POST
body:
value: {"size": $roomsize, "customers": $customers}
type: application/json
The context would then have to include both the room size and occupants, ie.
{"roomsize":"double", "customers":["John Doe", "Jane Doe"]}.
Consult the YAML documentation for more on what kind of data structures are possible to express.
When a type + value is given for a parameter or body (as above), the value is automatically marshaled to the given type (json in above example). If a parameter or body is given directly (no type+value syntax), a default must be given thus:
defaults:
structured_param_type: json structured_body_type: json
The above API snippet would specify that whenever a structured parameter or body value is encountered (such as a container or mapping), it will be marshalled to json. Simple values (strings, numbers etc) are used as-is.
Request generator and response parser loading
There are two convenience functions, _load_generator for loading the
request generator and _load_parser for loading the response parser:
>>> from httpreverse import _load_parser
>>> api = yaml.load(yamlsource)
>>> parser = _load_parser(api["list-rooms"])
>>>Recommended API operations spec generation and use
Typically, when using httpreverse to e.g. make http requests using whatever http client you have, you might want to first run just the Jinja expansion first and parse the resulting YAML string. Then, apply the request/response templates for the operations you expect to be using (or maybe all of them). Keep a copy of the the result. Finally, for each HTTP request, just parametrize the API operation being used, marshal parameters and body and fire away!