benchttp/runner

Extend runner capabilities as a CI tool

Closed this issue · 0 comments

Description

Benchttp runner can be used as a CI tool, executing it via a script and reading its stdout/json output.
This is facilitated by the option template that allows to display any Report data in the desired format, using Go's templating engine.

But it has its limit:

  • While the default summary provides basic stats information (min ,max, mean), there is currently no way to get them via template. It's obviously a major use case we're missing here, because a benchttp user is very likely to base its CI check on these kind of stats
  • Accessing a specific request event via template is tedious, e.g. event "GotFirstResponseByte" of the first record:
    {{ range (index .Benchmark.Records 0) }}
      {{ if eq .Name "GotFirstResponseByte" }}{{ .Time }}{{ end }}
    {{ end }}
  • There is currently no way to actually fail the test. It has to be done by the user's script reading the output and deciding whther it should fail. We want to improve that by providing a way to fail the test depending on results.

Tasks

Suggestions

Notes