/benchmark-mock-generator

Primary LanguageShell

Benchmark Mock Generator

Goal

Benchmark a bunch of tools that, given an OpenAPI produce a mock. Evaluate the following requirements:

  1. Let you override specific endpoints.
  2. Let you keep some state between successive API calls.
  3. Validate the request's payload.
  4. Let you write some assertions.
  5. Capable of recording network calls.
  6. Capable of reproducing recorded calls.
  7. Require the knowledge of a particular language.

Summary

Tool Let you override endpoints Let you keep state Validate requests Let you write assertion Capable of recording Capable of reproducing Require the language
mockoon no* [1] no no no yes no custom* [2]
ts-openapi-generator yes [3] yes* [4] no no yes* [5] no custom* [6]
prism yes* [9] no* [7] yes no no* [8] no no
imposter yes* [11] yes* [12] yes* [10] no no* [13] no custom* [14]
open-api-mocker yes* [15] no yes no no no custom* [15]
mock-server yes no no yes* [17] yes [16] no* [16] custom* [18]
specmatic yes no yes [19] no yes [20] yes custom [21]
wiremock yes no no no yes [22] no custom [23]
  1. It is possible to override an endpoint given a Mockoon's file format; it is not possible to override an endpoint given an open-api (more details here).
  2. Mockoon implements Handlebars, Faker.js v5.5.3, and a set of custom helpers to create dynamic responses. (more details here).
  3. Yep, updating the routes.json file (more details here).
  4. Not tested, but maybe it could. Documentation says it uses lowdb to store data.
  5. Not tested, but it could be possible writing a custom middleware (more details here).
  6. Endpoint's responses are retrieved from the db.json file generated. Another way is to generate data using JS (more details here).
  7. Based on prism roadmap, at some point the data persistence feature will be available, allowing Prism to act like a sandbox, no ETA is provided.
  8. Based on prism roadmap, at some point the recording/learning mode feature will be available, no ETA is provided.
  9. Prism allows you to override the return response through open-api examples.
  10. Request validation must be enabled (more details here).
  11. Based on documentation it allows to control the response in many ways, via templates, examples.
  12. Based on documentation it allows to keep a state (store, and capture).
  13. Based on documentation it can log everything so somehow can be stored on a file.
  14. Based on documentation imposter use a custom notation and accept different scripting language.
  15. Open-api-mocker allows you to override the return response through adding x-faker to the OpenAPI.
  16. Based on documentation should be easy to reply the recorded requests.
  17. Based on documentation we can write assertions but there is no way to check them.
  18. Support different languages (details here), and different template engine (details here).
  19. More details here.
  20. It could be possible using the logs.
  21. It is possible to use Gherkin to write the assertions. More details here
  22. Details here.
  23. It uses Handlebars.

Details

Mockoon

To run it execute the following command:

npm run mockoon:start

Nebula ts-openapi-generator

To run it, execute the following command:

npm run nebula:start

This command will generate the models and fix some issues explained in the next section.

Issues

In general, the tool looks not enough mature and used.

It uses json-server under the hood, but the quality of the generated files doesn't look good, especially on endpoint's path with path parameters (take a look at the routes.json file generated).

It doesn't generate all the elements present in the open-api file (e.g.

Processing models from 'components'
  Processing model PreLoadBulkRequest
    ERROR: PreLoadBulkRequest not a correct Schema
    ERROR: ARRAY not supported on COMPONENTS SCHEMA

so the mock-server doesn't serve all the routes (e.g. the POST /delivery/requests doesn't work).

In general, a lot of customization described in the table above, ara available thanks to the json-server library.

Yaml vs JSON

When generating models from OpenAPI, if we are using the .yaml file, we gets an error.

FATAL ERROR
Error: Function yaml.safeLoad is removed in js-yaml 4. Use yaml.load instead, which is now safe by default.
    at Object.safeLoad ([...]/node_modules/@protocolnebula/ts-openapi-generator/node_modules/js-yaml/index.js:10:11)
    at parseYAML ([...]/node_modules/@protocolnebula/ts-openapi-generator/build/utils/files.util.js:123:17)
    at FileReaderService.parseFile ([...]/node_modules/@protocolnebula/ts-openapi-generator/build/services/parsers/file-reader.service.js:121:51)
    at FileReaderService.<anonymous> ([...]/node_modules/@protocolnebula/ts-openapi-generator/build/services/parsers/file-reader.service.js:36:41)
    at Generator.next (<anonymous>)
    at fulfilled ([...]/node_modules/@protocolnebula/ts-openapi-generator/build/services/parsers/file-reader.service.js:5:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

To solve this issue, we should use the .json file available in docs/openapi/pn.json. Note that this file is the very same as the pn.yaml file, but in a different format.

Code generator

In order to run the mock server, we need to update the generated code, or to move a file in a proper location. This second option is the simplest one, and it has been encoded in a script available in the package.json file.

Postinstall

Postinstall script is required with this version more details here

Prism

To run it execute the following command:

npm run prism:start

open-api-mocker

To run it execute the following command:

npm run mocker:start

Note that the run command executes a shell script which downloads and runs the Docker image. As it is, the container will accept requests on port 8080.

Issues

It has some issues when running directly using Node.js; using Docker it works fine. The Dockerfile runs an old version of Node: version 10.

Imposter

To run it execute the following command:

npm run imposter:start

Visit http://localhost:8080/_spec to get the API specifications. Visit http://localhost:8080/system/store/preload-request/preload to lookup the key preload of the perload-request database.

Mock-Server

To run it execute the following command:

npm run mockserver:start

Visit http://localhost:8080/mockserver/dashboard to get the dashboard.

Mock-server is really powerful, it provides internal open-api.

To get the recorder request execute the following command:

curl -v -X PUT "http://localhost:8080/mockserver/retrieve?type=REQUESTS"

Specmatic

To run it execute the following command:

npm run specmatic:start

Issues

It doesn't work with the pn.yaml file, because specmatic has some issues with the parsing of the file. In order to try the tool, there is the employee.yaml file, which is a very simple OpenAPI file.

Wiremock

To run it execute the following command:

npm run wiremock:start

Issues

Wiremock has various tools we can use to stub, but none of them is complete. Wiremock doesn't allow to load an OpenAPI, and the stubs are passed through a configuration or an external file (details here). Wiremock studio, instead, allows to load an OpenAPI file, but it's not simple to use without a GUI (nor is well documented to use in that way).

We can take a look on how it implements some feature, but it's not a solution suited for our needs.