Improved compatibility with Postman collections [bounty: $1500]
robingustafsson opened this issue · 30 comments
We would like to improve the workflow for users of Postman collections (v2.x) that want to load test their APIs using k6, provide an even better onboarding experience to k6 (and in extension Load Impact), by extending the postman-to-k6 converter to take the following Postman collection features into account in the conversion process. A completed solution will also include tests and docs (updated README).
Related issues:
Cookies
Setting cookies properly if they’re explicitly defined in the collection, see docs.
Relevant links:
- https://docs.k6.io/docs/cookies
- https://www.getpostman.com/docs/v6/postman/sending_api_requests/cookies
TLS Client Certificates
Setting up use of client certificates in k6 if defined in the collection, see docs.
Relevant links:
- https://docs.k6.io/docs/ssl-tls-client-certificates
- https://www.getpostman.com/docs/v6/postman/sending_api_requests/certificates
Authentication
Support the most common authentication options supported by Postman, see docs.
The authentications we want to support:
- No auth (already supported in the converter)
- Bearer token
- HTTP Basic (already supported in the converter)
- HTTP Digest
- OAuth 1 / 2
- See: TODO KB example script
- AWS Signature v4
- See https://support.loadimpact.com/4.0/test-scripting/examples/#aws-signature-v4-authentication (will be more performant once we have support for binary output for the hashing algorithms in the crypto package: grafana/k6#856)
- NTLM Authentication
This should also include the case when the authentication is set to “Inherit auth from parent”.
Relevant links:
- https://github.com/loadimpact/postman-to-k6/tree/master/lib/postman/auth
- https://www.getpostman.com/docs/v6/postman/sending_api_requests/authorization
Environment/global variables
Support environment and global variables defined in the collection, see docs. The conversion to k6 JS must make sure to adhere to the scoping rules defined in https://learning.getpostman.com/docs/postman/environments_and_globals/variables/#variable-scopes.
In general variables in the request URL, params, headers and body will have the form {{someVariable}}. These needs to be set by properly resolving the variable.
Relevant links:
- https://docs.k6.io/docs/environment-variables
- https://www.getpostman.com/docs/v6/postman/environments_and_globals/variables
Dynamic variables
The conversion needs to support Postman’s dynamic variables that can be present in the request URL, params, headers and body:
{{$guid}}
: Adds a v4 style GUID{{$timestamp}}
: Adds the current timestamp{{$randomInt}}
: Adds a random integer between 0 and 1000
Data variables
The conversion needs to support what’s called “data variables” in Postman, see docs. This is parameterization data loaded from a CSV or JSON file and then used in request URL, params, headers, body or pre-/post-request scripts.
In scripts
In scripts you get variables (global, environment, dynamic or data) by using one of the APIs, see Scripting below.
Scripting
Postman has support for JS scripting in the form of pre-request scripts and post-response test scripts. By providing a shim layer in the converted script to support the Postman Sandbox execution environment, where pre-/post-requests scripts are executed, we should be able to provide a much more useful conversion tool from Postman to k6 for doing load testing.
Shimming
The shim layer could either be embedded in the converted script (or as a module that the converter spits out and the main file loads) or fetched from k6 github perhaps as a remote import, something like:
import postman from “github.com/loadimpact/k6/master/shims/postman-sandbox.js”;
Pre-/Post-request scripts/tests
These are currently appended as comments in the k6 script, but will now need to be inserted into k6 verbatim, with the necessary Postman APIs made available by the shim layer. See pre-request scripts docs and post-request test scripts for more info.
Support both pre-request scripts specified for a collection, a folder and an individual request (respecting the execution order of scripts).
Postman Sandbox APIs
The following Postman Sandbox APIs needs to be supported/shimmed:
- It will include making some third-party libs available by default (see list in API docs). This could probably be handled by inserting them as imports to be loaded by the supported CDN/Github loaders in k6.
- General
postman.getResponseHeader(headerName)
xml2Json(xmlString)
- Environment and global variables
postman.setEnvironmentVariable(variableName, variableValue)
postman.getEnvironmentVariable(variableName)
postman.setGlobalVariable(variableName, variableValue)
postman.getGlobalVariable(variableName)
postman.clearEnvironmentVariable(variableName)
postman.clearGlobalVariable(variableName)
postman.clearEnvironmentVariables()
environment
(a global object variable used by the functions above)globals
(a global object variable used by the functions above)
- Dynamic variables
{% raw %}{{$guid}}{% endraw %}
: Adds a v4 style GUID{% raw %}{{$timestamp}}{% endraw %}
: Adds the current timestamp.{% raw %}{{$randomInt}}{% endraw %}
: Adds a random integer between 0 and 1000
- Cookies
responseCookies
(array variable with response cookies for current response)postman.getResponseCookie(cookieName)
- Request/response related properties
request
(object with request attributes)data
(key/value form data object)headers
(key/value object with request headers)method
url
responseHeaders
(key/value object with response headers)responseBody
(string, use JSON.parse or xml2json to parse it)responseTime
(number, total response time in milliseconds)responseCode
(object)code
name
detail
tests
(object with the users “checks”/”asserts”, boolean results of logical tests)iteration
(number, the current test run index)
- Data files
data
(object with parsed CSV or JSON data, each VU and iteration in k6 would need to fetch a new row of data, see this blog post)
- All APIs in the Postman Sandbox API reference (the
pm
object, which is only available in Postman, not Newman): https://learning.getpostman.com/docs/postman/scripts/postman_sandbox_api_reference/- Except the
require()
’ing of Nodejs modules, that should result in an error message saying “Can’t load module X, Node.js modules aren’t supported in k6” pm.sendRequest(url|requstObject, callback)
this call would need to be synchronous in k6, so it wouldn’t return until the callback function is done executing.
- Except the
@robingustafsson this looks interesting, i'll take a look at the bounty
@mul1sh Great, go ahead
@robingustafsson thanks will do
@robingustafsson is it ok if i split the functionality into seperate files i.e. for the authentication i use auth.js e.t.c
Edit
Also do ou have an im i.e. gitter, slack e.t.c where I can quickly reach out in case I have more questions because I'm actively working on this and so i'll definitely have a lot of questions.
@mul1sh Yes, I'd expect the code to be modular to make it easy to follow and extend, so not have everything being in the same file
You can find me in k6 Slack (https://k6.io/slack/) as "robin".
@robingustafsson awesome thanks sounds great, I'll reach out on slack when I have some more questions
@mul1sh Do you push your WIP code to github? As this task is quite big, it'd be great if I could follow the progress in a branch of your fork (https://github.com/mul1sh/postman-to-k6) to avoid having a huge PR to review in the end. It would also help avoid potential frustration as part of the review process if I have to request time consuming refactorings
@robingustafsson i'll push today. Sorry for the delay, I was first going through the k6
documentation and the existing converter code to make I fully understood the current functionality, which I think i do .
So anyway working on the WIP PR now as indicated above and then i push the required functionality in chunks till the issue is closed.
@mul1sh Any update on the WIP code? I'm not expecting anything to be functional, it's more to get a feel for how you're approaching the project and the code quality so we can course correct early if necessary
@robingustafsson yes its coming in less than 6 hours, sorry I was still going back and forth in the k6 docs there are some things i didn't fully grasp but now I do. So actually working on the PR now
@robingustafsson finally a PR
Nonetheless expect more commits till we're done, I'm planning to wrap this up by Wednesday because i'm actively working on it and I fully understand how k6 works now.
@mul1sh Great, I'll have a look.
bookmoons/improve
has something toward this. All the auth methods except OAuth are implemented.
I've reorganized the code a little. It was kind of difficult to work with.
- There seemed to be no consistent style. There was an obsolete linter installed, but it didn't even seem to be used. Ran it and it found hundreds of issue. I've normalized instead to StandardJS, which has good tool support, and corrected a some issues it found (including some crashing errors). Is that an acceptable style?
- Removed callbacks from the conversion logic. They were completely unused. Everything is happening synchronously.
- Enabled CircleCI.
Re the AWS auth method: The signature module, awkwardly, requires a decomposed URL. Since the entire URL could come from a variable at runtime, it has to be decomposed at runtime. So I've had to add urijs
as another dependency. At the moment I have these as a manual build requirement.
I'm not sure there's a way to get access to the cookies and certificates. They seem to be treated as a local workspace thing and don't get included in exports. I asked these questions about it on the forum but so far no love.
@bookmoons Nice, great progress and I'm glad to see you cleaned up the code, super!
- Regarding the styling, StandardJS looks fine to me. Talked with our JS devs as well and we'd prefer to always use curly-braces for statements where StandardJS allows to remove them, eg.:
if (options.quiet !== true) {
console.log('done')
}
instead of
if (options.quiet !== true) console.log('done')
Makes the code a bit less compact, but a bit more readable.
- AWS v4 auth: yeah not ideal with external deps but as we don't have all the necessary APIs in k6 itself yet it will have to do for now.
- Cookies: ok, if it turns out cookies are not included in the collection there's not much we can do. I misunderstood the docs then regarding the "cookie manager" available in the Postman app.
- TLS client certs: ok, maybe I misunderstood the docs here as well. I guess it makes sense that you'd not include cert keys in the collection.
Talked with our JS devs as well and we'd prefer to always use curly-braces for statements where StandardJS allows to remove them
Will update this. How do you guys feel about indenting in switches?
switch (variable.type) {
case 'boolean': return VariableType.Boolean
case 'json': return VariableType.Json
case 'number': return VariableType.Number
case 'string': return VariableType.String
}
switch (variable.type) {
case 'boolean':
return VariableType.Boolean
case 'json':
return VariableType.Json
case 'number':
return VariableType.Number
case 'string':
return VariableType.String
}
@bookmoons We'd prefer the second one
OAuth and auth inheritance are in. Everything in the Authentication section is done.
I want to propose dropping the default for the -j
version option. Currently v2 is default. If you run on a v1 file it fails with some obscure error.
I have version detection in. So if we just drop the default and let it detect v1, users will get a more meaningful error.
Yeah, that sounds like a better UX to me as well, go ahead and drop -j
.
Yeah, that sounds like a better UX to me as well, go ahead and drop
-j
.
Done.
All the nonlocal variables are implemented, including dynamic.
Environments and global variables have to be exported separately from Postman, so I've added CLI options to read them. Data files don't seem to be in any exports so I've just created options for them where you provide a path.
-g --global <path> JSON export of global variables.
-e --environment <path> JSON export of environment.
-c --csv <path> CSV data file. Used to fill data variables.
-j --json <path> JSON data file. Used to fill data variables.
Nice, thanks for sharing your progress, looks great! I'll test the converter a bit in the next few days.
The shim is not deploying at the moment, so if you go to do tests lib/shim.js
has to be copied to ./postman-shim.js
.
Have just restructured this, lib/shim/core.js
is the one to copy now.
I think everything possible is implemented. Have just submitted #16 to start it off. I'm ready to jump on bug fixes.
Fantastic, will start reviewing the PR tomorrow.
Closed by #16.