OpenAPI Testing Framework

A framework for testing an API based on its OpenAPI schema.

This framework allows to create anything between a manual test and a fully automated test.

Installation

Install Python (versions 3.6-3.9 where tested to work)

Clone this repository

git clone https://github.com/specify/open_api_tools
cd open_api_tools

Configure a virtual environment

python -m venv venv

Install the dependencies

./venv/bin/pip install -r requirements.txt

Install this package locally

pip install -e .

Usage

There are three main use cases, each going from more automated to more manual.

The most automated is the full_test, which, by default, generates test URLs with some parameters based on OpenAPI schema, then sends those requests and makes sure that responses match the schema definition. By default, this method only tests GET endpoints and it does not generate request body object. However, this can be changed by providing additional parameters. Also, you can define parameter constraints (e.x if 'a' is set to True, then response must contain 'b') to further improve the quality of this test.

Next, there is a Chain test that allows to test a chain of requests and make sure that each request correctly influenced the response of the next request.

Finally, for those that need complete control, there is a make_request method which facilitates validating the request parameters, sending a single request, validating the response parameters and returning the result.

Full test

The handler function should return a boolean value saying validating whether the response object is as expected

Basic usage

Run the test

from open_api_tools.test.full_test import full_test
from open_api_tools.common.load_schema import load_schema

schema = load_schema('open_api.yaml')

# Error message schema is defined in open_api_tools.validate.index
def after_error_occurred(*error_message):
    print(error_message)

full_test(
    schema=schema,
    max_urls_per_endpoint=50,
    failed_request_limit=10,
    after_error_occurred=after_error_occurred,
)

This script would automatically generate test URLs based on your API schema.

All requests would be sent to the first server specified in the servers part of the API schema.

max_urls_per_endpoint parameter defines the limit of queries to send to a single endpoint.

failed_request_limit makes sure that the full_test function quits with an exception if a certain number of requests failed validation. This is useful for preventing needless server load when all requests are failing for the same reason.

Supplying test values for parameters

By default, the test reads the examples object in the schema to generate request parameters. If examples weren't provided, it would try to create some test values based on the parameter type.

If you would like more customization, an optional after_examples_generated hook can be provided to the full_test method.

after_examples_generated must be a function that accepts an endpoint name as the first parameter and the parameter object as the second parameter (the parameter object would vary depending on how it is defined in your schema). In turn, the function must return a list of valid examples.

Example usage:

from open_api_tools.test.full_test import full_test
from open_api_tools.common.load_schema import load_schema

schema = load_schema('open_api.yaml')

def after_error_occurred(*error_message):
    print(error_message)

def after_examples_generated(
    endpoint_name,
    parameter,
    autogenerated_examples
):
    if endpoint_name == '/api/posts/' and parameter.name == 'post_id':
        return [1, 2, 3, 4, 5]
    elif parameter.schema.type == 'string':
        # Some naughty strings
        return ["ÅÍÎÏ˝ÓÔÒÚÆ☃", "Ω≈ç√∫˜µ≤≥÷", "⅛⅜⅝⅞"]
    else:
        return autogenerated_examples

full_test(
    schema=schema,
    max_urls_per_endpoint=50,
    failed_request_limit=10,
    after_error_occurred=after_error_occurred,
    after_examples_generated=after_examples_generated
)

The third parameter in after_examples_generated is the list of examples that were generated by this framework. If you don't want to change the generated examples, the function can return back this value.

Note that after_examples_generated would also get called with requestBody as a parameter.name. This allows you to to provide a list of request objects that would be used in testing. Each request object should be of type (str, str), where the first string is the MIME type and the second one is the serialized payload that would be send with the request.

Handling authentication and amending the request object

full_test also supports a before_request_send hook that allows you to modify the request object before a request is sent. This is useful if you want to edit the headers or add authentication cookies.

Example usage:

from open_api_tools.test.full_test import full_test
from open_api_tools.common.load_schema import load_schema

schema = load_schema('open_api.yaml')

def before_request_send(endpoint_name, request_object):
    if endpoint_name == '/api/main/{id}/':
        if request_object.headers is None:
            request_object.headers = {}
        request_object.headers['Authorization']='Basic YWxhZGRpbjpvcGVuc2VzYW1l'
    return request_object

full_test(
    schema=schema,
    max_urls_per_endpoint=50,
    failed_request_limit=10,
    before_request_send=before_request_send
)

The schema for the request object is defined here.

Defining parameter constrains

If response object depends on the query parameters, you can test for these relationships by adding your parameter names and handler functions to the parameter_constraints dictionary and passing it to full_test.

Each handler function would receive the following arguments:

parameter_value (any): the value of the parameter this handler works with
path (str): name of the current endpoint (useful if the same parameter is shared between multiple endpoints)
response (any): response object.

Example usage:

from open_api_tools.test.full_test import full_test
from open_api_tools.common.load_schema import load_schema

schema = load_schema('open_api.yaml')

def get_popular_posts(
    parameter_value: int,
    endpoint: str,
    response: object
):
    for post in response.json():
        if post.popularity < parameter_value:
            raise Exception(
                f'{endpoint} failed to filter the posts by popularity'
            )

full_test(
    schema=schema,
    max_urls_per_endpoint=50,
    failed_request_limit=10,
    parameter_constraints={

    },
)

Chain test

For more fine-grained testing, there is a chain method that allows to test a chain of request URLs with request/response object validation and assurance that each request produced expected results.

Example usage:

from open_api_tools.test.chain import chain, Request, Validate
from open_api_tools.common.load_schema import load_schema
import json

schema = load_schema('open_api.yaml')

post_id = 345

def create_post(_arguments, _response, _previous_values):
    return {
        "requestBody": [
            'application/json',
            json.dumps(dict(
                id=post_id,
                name='Post Name',
                body='Post content'
            ))
        ]
    }

chain(
    schema=schema,
    definition=[
        Request(method='GET', endpoint='/api/posts/'),
        Validate(
            validate=\
                lambda response: post_id not in response.json().posts
        ),
        Request(
            method='POST',
            endpoint='/api/posts/',
            parameters=create_post
        ),
        Validate(
            validate=lambda response: post_id in response.json().posts
        ),
        Request(
            method='DELETE',
            endpoint='/api/posts/',
            parameters={'id': post_id}
        ),
        Validate(
            validate= \
                lambda response: post_id not in response.json().posts
        ),
    ],
)

The response object for the validation function is described here.

Keep in mind that the endpoint string in the Request class should be identical to one of the endpoints in your OpenAPI schema. For example, you should specify /api/user/{user_id}/ instead of /api/user/1/. Both path parameters and query parameters should be supplied in the parameters dictionary or returned by the parameters function. If parameters is a function, it would get called with these arguments: (list_of_parameter_objects, previous_response, previous_parameter_values). Alternatively, you can omit the parameters key altogether if the endpoint doesn't expect any.

Also, Request can omit parameters if there aren't any to define. Alternatively, you can supply a dictionary, or a function that would get called with three arguments: (list_of_parameter_objects, previous_response, previous_parameter_values).

Additionally, you can supply a requestBody parameter. Unlike most parameters, requestBody must be a Tuple[str,str] where the first string is a MIME type and the second string is a serialized version of the request body.

The Validate class expects a function that takes a response object and returns a boolean saying whether a value is valid. On false, the chain stops. Note, if Validate returned false, an exception is not thrown, but you can throw one on your own if you need to.

The chain method also accepts a before_request_send parameter, which is described in detail in the previous section

Manual test

make_request method is most useful when you need complete control over the requests that get send, but still need the assurance that request/response objects confirm to schema.

Example usage:

from open_api_tools.validate.index import make_request
from open_api_tools.common.load_schema import load_schema
import json

schema = load_schema('open_api.yaml')

response = make_request(
  schema=schema,
  request_url='http://localhost/api/posts/1/?update_indexes=true',
  endpoint_name='/api/posts/<post_id>',
  method='POST',
  body=('application/json', json.dumps({"name": 'New post name'})),
)

For additional control, the make_request method also accepts a before_request_send parameter, which is described in detail in the previous section.

If you want to only verify the request object, or want to execute some additional code before executing the request, the make_request can be broken down into prepare_request and file_request methods. They are defined in open_api_tools/validate/index.py.