Extend runner capabilities as a CI tool
GregoryAlbouy opened this issue · 0 comments
GregoryAlbouy commented
Description
Benchttp runner can be used as a CI tool, executing it via a script and reading its stdout/json output.
This is facilitated by the option template
that allows to display any Report data in the desired format, using Go's templating engine.
But it has its limit:
- While the default summary provides basic stats information (
min
,max
,mean
), there is currently no way to get them viatemplate
. It's obviously a major use case we're missing here, because a benchttp user is very likely to base its CI check on these kind of stats - Accessing a specific request event via
template
is tedious, e.g. event "GotFirstResponseByte" of the first record:{{ range (index .Benchmark.Records 0) }} {{ if eq .Name "GotFirstResponseByte" }}{{ .Time }}{{ end }} {{ end }}
- There is currently no way to actually fail the test. It has to be done by the user's script reading the output and deciding whther it should fail. We want to improve that by providing a way to fail the test depending on results.