grafana/postman-to-k6

Improved compatibility with Postman collections [bounty: $1500]

robingustafsson opened this issue · 30 comments

robingustafsson commented

We would like to improve the workflow for users of Postman collections (v2.x) that want to load test their APIs using k6, provide an even better onboarding experience to k6 (and in extension Load Impact), by extending the postman-to-k6 converter to take the following Postman collection features into account in the conversion process. A completed solution will also include tests and docs (updated README).

Related issues:

  • #7: Convert collection that uses Environment Variables
  • #12: Add support for variables in body

Cookies

Setting cookies properly if they’re explicitly defined in the collection, see docs.

Relevant links:

TLS Client Certificates

Setting up use of client certificates in k6 if defined in the collection, see docs.

Relevant links:

Authentication

Support the most common authentication options supported by Postman, see docs.

The authentications we want to support:

This should also include the case when the authentication is set to “Inherit auth from parent”.

Relevant links:

Environment/global variables

Support environment and global variables defined in the collection, see docs. The conversion to k6 JS must make sure to adhere to the scoping rules defined in https://learning.getpostman.com/docs/postman/environments_and_globals/variables/#variable-scopes.

In general variables in the request URL, params, headers and body will have the form {{someVariable}}. These needs to be set by properly resolving the variable.

Relevant links:

Dynamic variables

The conversion needs to support Postman’s dynamic variables that can be present in the request URL, params, headers and body:

  • {{$guid}}: Adds a v4 style GUID
  • {{$timestamp}}: Adds the current timestamp
  • {{$randomInt}}: Adds a random integer between 0 and 1000

Data variables

The conversion needs to support what’s called “data variables” in Postman, see docs. This is parameterization data loaded from a CSV or JSON file and then used in request URL, params, headers, body or pre-/post-request scripts.

In scripts

In scripts you get variables (global, environment, dynamic or data) by using one of the APIs, see Scripting below.

Scripting

Postman has support for JS scripting in the form of pre-request scripts and post-response test scripts. By providing a shim layer in the converted script to support the Postman Sandbox execution environment, where pre-/post-requests scripts are executed, we should be able to provide a much more useful conversion tool from Postman to k6 for doing load testing.

Shimming

The shim layer could either be embedded in the converted script (or as a module that the converter spits out and the main file loads) or fetched from k6 github perhaps as a remote import, something like:

import postman from “github.com/loadimpact/k6/master/shims/postman-sandbox.js”;

Pre-/Post-request scripts/tests

These are currently appended as comments in the k6 script, but will now need to be inserted into k6 verbatim, with the necessary Postman APIs made available by the shim layer. See pre-request scripts docs and post-request test scripts for more info.

Support both pre-request scripts specified for a collection, a folder and an individual request (respecting the execution order of scripts).

Postman Sandbox APIs

The following Postman Sandbox APIs needs to be supported/shimmed:

  • It will include making some third-party libs available by default (see list in API docs). This could probably be handled by inserting them as imports to be loaded by the supported CDN/Github loaders in k6.
  • General
    • postman.getResponseHeader(headerName)
    • xml2Json(xmlString)
  • Environment and global variables
    • postman.setEnvironmentVariable(variableName, variableValue)
    • postman.getEnvironmentVariable(variableName)
    • postman.setGlobalVariable(variableName, variableValue)
    • postman.getGlobalVariable(variableName)
    • postman.clearEnvironmentVariable(variableName)
    • postman.clearGlobalVariable(variableName)
    • postman.clearEnvironmentVariables()
    • environment (a global object variable used by the functions above)
    • globals (a global object variable used by the functions above)
  • Dynamic variables
    • {% raw %}{{$guid}}{% endraw %}: Adds a v4 style GUID
    • {% raw %}{{$timestamp}}{% endraw %}: Adds the current timestamp.
    • {% raw %}{{$randomInt}}{% endraw %}: Adds a random integer between 0 and 1000
  • Cookies
    • responseCookies (array variable with response cookies for current response)
    • postman.getResponseCookie(cookieName)
  • Request/response related properties
    • request (object with request attributes)
      • data (key/value form data object)
      • headers (key/value object with request headers)
      • method
      • url
    • responseHeaders (key/value object with response headers)
    • responseBody (string, use JSON.parse or xml2json to parse it)
    • responseTime (number, total response time in milliseconds)
    • responseCode (object)
      • code
      • name
      • detail
    • tests (object with the users “checks”/”asserts”, boolean results of logical tests)
    • iteration (number, the current test run index)
  • Data files
    • data (object with parsed CSV or JSON data, each VU and iteration in k6 would need to fetch a new row of data, see this blog post)
  • All APIs in the Postman Sandbox API reference (the pm object, which is only available in Postman, not Newman): https://learning.getpostman.com/docs/postman/scripts/postman_sandbox_api_reference/
    • Except the require()’ing of Nodejs modules, that should result in an error message saying “Can’t load module X, Node.js modules aren’t supported in k6”
    • pm.sendRequest(url|requstObject, callback) this call would need to be synchronous in k6, so it wouldn’t return until the callback function is done executing.
mul1sh commented

@robingustafsson this looks interesting, i'll take a look at the bounty 🙂

robingustafsson commented

@mul1sh Great, go ahead 🙂Let me know if you have any questions, if anything needs to be clarified.

mul1sh commented

@robingustafsson thanks will do

mul1sh commented

@robingustafsson is it ok if i split the functionality into seperate files i.e. for the authentication i use auth.js e.t.c

Edit

Also do ou have an im i.e. gitter, slack e.t.c where I can quickly reach out in case I have more questions because I'm actively working on this and so i'll definitely have a lot of questions.

robingustafsson commented

@mul1sh Yes, I'd expect the code to be modular to make it easy to follow and extend, so not have everything being in the same file 🙂

You can find me in k6 Slack (https://k6.io/slack/) as "robin".

  • 🎉1
mul1sh commented

@robingustafsson awesome thanks sounds great, I'll reach out on slack when I have some more questions 🙂

  • 👍1
robingustafsson commented

@mul1sh Do you push your WIP code to github? As this task is quite big, it'd be great if I could follow the progress in a branch of your fork (https://github.com/mul1sh/postman-to-k6) to avoid having a huge PR to review in the end. It would also help avoid potential frustration as part of the review process if I have to request time consuming refactorings 🙂

mul1sh commented

@robingustafsson i'll push today. Sorry for the delay, I was first going through the k6 documentation and the existing converter code to make I fully understood the current functionality, which I think i do .

So anyway working on the WIP PR now as indicated above and then i push the required functionality in chunks till the issue is closed.

robingustafsson commented

@mul1sh Any update on the WIP code? I'm not expecting anything to be functional, it's more to get a feel for how you're approaching the project and the code quality so we can course correct early if necessary 🙂

  • 👍1
mul1sh commented

@robingustafsson yes its coming in less than 6 hours, sorry I was still going back and forth in the k6 docs there are some things i didn't fully grasp but now I do. So actually working on the PR now

  • 👍1
mul1sh commented

@robingustafsson finally a PR 😄 even though it didn't exactly take 6 hours as promised 😢 because I still had some more learning to do for K6 through its docs.

Nonetheless expect more commits till we're done, I'm planning to wrap this up by Wednesday because i'm actively working on it and I fully understand how k6 works now.

robingustafsson commented

@mul1sh Great, I'll have a look.

  • 👍1
bookmoons commented

bookmoons/improve has something toward this. All the auth methods except OAuth are implemented.

I've reorganized the code a little. It was kind of difficult to work with.

  • There seemed to be no consistent style. There was an obsolete linter installed, but it didn't even seem to be used. Ran it and it found hundreds of issue. I've normalized instead to StandardJS, which has good tool support, and corrected a some issues it found (including some crashing errors). Is that an acceptable style?
  • Removed callbacks from the conversion logic. They were completely unused. Everything is happening synchronously.
  • Enabled CircleCI.

Re the AWS auth method: The signature module, awkwardly, requires a decomposed URL. Since the entire URL could come from a variable at runtime, it has to be decomposed at runtime. So I've had to add urijs as another dependency. At the moment I have these as a manual build requirement.

  • 🎉1
bookmoons commented

I'm not sure there's a way to get access to the cookies and certificates. They seem to be treated as a local workspace thing and don't get included in exports. I asked these questions about it on the forum but so far no love.

How to export cookies?
How to export certificates?

  • 👍1
robingustafsson commented

@bookmoons Nice, great progress and I'm glad to see you cleaned up the code, super!

  • Regarding the styling, StandardJS looks fine to me. Talked with our JS devs as well and we'd prefer to always use curly-braces for statements where StandardJS allows to remove them, eg.:
if (options.quiet !== true) {
  console.log('done')
}

instead of

if (options.quiet !== true) console.log('done')

Makes the code a bit less compact, but a bit more readable.

  • AWS v4 auth: yeah not ideal with external deps but as we don't have all the necessary APIs in k6 itself yet it will have to do for now.
  • Cookies: ok, if it turns out cookies are not included in the collection there's not much we can do. I misunderstood the docs then regarding the "cookie manager" available in the Postman app.
  • TLS client certs: ok, maybe I misunderstood the docs here as well. I guess it makes sense that you'd not include cert keys in the collection.
bookmoons commented

Talked with our JS devs as well and we'd prefer to always use curly-braces for statements where StandardJS allows to remove them

Will update this. How do you guys feel about indenting in switches?

switch (variable.type) {
  case 'boolean': return VariableType.Boolean
  case 'json': return VariableType.Json
  case 'number': return VariableType.Number
  case 'string': return VariableType.String
}
switch (variable.type) {
  case 'boolean':
    return VariableType.Boolean
  case 'json':
    return VariableType.Json
  case 'number':
    return VariableType.Number
  case 'string':
    return VariableType.String
}
robingustafsson commented

@bookmoons We'd prefer the second one 🙂

bookmoons commented

Alright, updated all of that indenting. Also documented these things in STYLE.

  • 👍1
bookmoons commented

OAuth and auth inheritance are in. Everything in the Authentication section is done.

  • 👍1
bookmoons commented

I want to propose dropping the default for the -j version option. Currently v2 is default. If you run on a v1 file it fails with some obscure error.

I have version detection in. So if we just drop the default and let it detect v1, users will get a more meaningful error.

robingustafsson commented

Yeah, that sounds like a better UX to me as well, go ahead and drop -j.

  • 👍1
bookmoons commented

Yeah, that sounds like a better UX to me as well, go ahead and drop -j.

Done.

  • 👍1
bookmoons commented

All the nonlocal variables are implemented, including dynamic.

Environments and global variables have to be exported separately from Postman, so I've added CLI options to read them. Data files don't seem to be in any exports so I've just created options for them where you provide a path.

  -g --global <path>       JSON export of global variables.
  -e --environment <path>  JSON export of environment.
  -c --csv <path>          CSV data file. Used to fill data variables.
  -j --json <path>         JSON data file. Used to fill data variables.
robingustafsson commented

Nice, thanks for sharing your progress, looks great! I'll test the converter a bit in the next few days.

  • 👍1
bookmoons commented

The shim is not deploying at the moment, so if you go to do tests lib/shim.js has to be copied to ./postman-shim.js.

  • 👍1
bookmoons commented

Have just restructured this, lib/shim/core.js is the one to copy now.

  • 👍1
bookmoons commented

I think everything possible is implemented. Have just submitted #16 to start it off. I'm ready to jump on bug fixes.

  • 🎉1
robingustafsson commented

Fantastic, will start reviewing the PR tomorrow.

  • 👍1
robingustafsson commented

Closed by #16.