proposal: cmd/go: -benchformat option in go test
nodirt opened this issue · 10 comments
Background
Currently go test -bench . prints out benchmark results in a certain format, but there is no guarantee that this format will not change. Thus a tool that parses go test output may break if an incompatible change to the output format is made.
Also currently there is no good way to distinguish benchmark results from the rest of go test output, other than checking that a line starts with "Benchmark".
Proposal
- Add
-benchformatflag togo test. Its value is a text/template template withBenchmarkResultinput, similar to-fflag ingo list. - Add
ProctoBenchmarkResult. It will have the value ofruntime.GOMAXPROCSat the beginning ofB.launchcall.
Example:
$ go test -run=^$ -bench . \
-benchformat "BENCHMARK STARTS HERE:{{.Name}}:{{.Proc}}:{{.T}}:{{.N}}:{{.Bytes}}"
PASS
BENCHMARK STARTS HERE:BenchmarkFoo:8:1s:10000:0
BENCHMARK STARTS HERE:BenchmarkBar:8:1s:10000:0One may be concerned that adding -benchformat for benchmarks without adding -format for tests would be inconsistent. We can add -format for tests too.
Currently there is no struct for a test result template. It would need to be added. I don't see a strong reason to add testing.TestResult to that API, though, as we can just document the test result struct in go test -help.
Considering that go list accepts a text/template format, this seems consistent. I think it's probably worth exploring and writing up a more formal proposal, unless somebody disagrees.
/cc @adg
Thanks for mentioning #2981.
Assuming implementation would use encoding/json package without manual concatenation of json chunks, one advantage of -format over -json is that go test can stream results to stdout, rather than waiting for completion of all tests/benchmarks and then dump a big json blob. This is important for tools that present test results to the user. In my case, I need to process test output and print it out (annotated) as soon as a benchmark completes. Without streaming UX would be worse.
Even if we concatenate json chunks, parsing them with encoding/json would not be convenient either.
Also from reading #2981 it seems that adding -json is hard, whereas adding -format is trivial, more consistent with go list and provides same, if not more, functionality.
Currently
go test -bench .prints out benchmark results in a certain format, but there is no guarantee that this format will not change.
OK. Counter-proposal: let's guarantee that this format will not change.
Specifically, I propose that the output format is exactly a sequence of lines containing space-separated fields:
<name> <iterations> (<value> <unit>)+
The number of on a particular line is not specified, nor is the space of possible s.
This describes all the current output and blesses (instead of forcing rewrites of) tools that parse the output already, like benchcmp and benchstat.
/cc @aclements
I don't have a strong opinion about this, but I'm inclined to think it's unnecessary as long as we simply bless the current format as suggested by @rsc and especially unnecessary if we add -json (and if I had to choose between the two, I would definitely choose -json).
Suppose we were to simply bless the current format. What future changes to go test -bench might that cause friction with? I have a few things on my list. I'd like an external tool (e.g., benchstat) to be able to do linearity testing. We could support that by outputting lines for each run-up step. I'd like go test -bench to be able to output many more instances of shorter runs so external tools can compute more robust statistics. Again, go test -bench can just output more instances of the same benchmark line. I've thought about adding custom outputs (# of GCs, max pause, etc). We could support that by adding more (value unit) pairs.