benchmarks appear to not work on the main branch

[mvdan]

$ git rev-parse HEAD
a8d880238c2679e6191d67d26dd859896c32237f
$ cd benchmarks
$ go version
go version go1.17.3 linux/amd64
$ go test -run=- -bench=. -benchtime=1x
goos: linux
goarch: amd64
pkg: github.com/ipfs/go-graphsync/benchmarks
cpu: AMD Ryzen 7 PRO 5850U with Radeon Graphics     
BenchmarkRoundtripSuccess/test-20-10000-16         	      1	  4261905 ns/op	3863928 B/op	  66003 allocs/op
BenchmarkRoundtripSuccess/test-20-128MB-16         	      1	10000215922 ns/op	5697168952 B/op	73886804 allocs/op
--- FAIL: BenchmarkRoundtripSuccess/test-p2p-stress-10-128MB
    benchmark_test.go:185: received error on request: request cancelled by client
2021-12-02T16:46:13.394Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreihiod5chhe4bwv7wximrv7k5uyfqny7so2c2jmbiomrocmpzte5pe, nBlocksRead=10, err=context cancelled
2021-12-02T16:46:13.394Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreibr6ksfhbkw4q66qu3po42w6aqntcu3kgcxgoslx4dg67vol7wcaq, nBlocksRead=9, err=context cancelled
2021-12-02T16:46:13.394Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreiaj4gsp5yqjiahjhzqloqc23oeuoa2z56xikeetzaze7a3desxn7e, nBlocksRead=9, err=context cancelled
2021-12-02T16:46:13.394Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreiaa7pq6tb6gdls6pabqvrdo7ohun34tx5a7lldsqs2nkfpsrv77nm, nBlocksRead=8, err=context cancelled
2021-12-02T16:46:13.394Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:194	traversal completion check failed, nBlocksRead=11, err=context cancelled
2021-12-02T16:46:13.394Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreicuezn6dedejb723chqajf6x6xdj2mm3z4caxzw5wvcundfyfalr4, nBlocksRead=12, err=context cancelled
--- FAIL: BenchmarkRoundtripSuccess/test-p2p-stress-10-128MB-1KB-chunks
    benchmark_test.go:185: received error on request: request cancelled by client
2021-12-02T16:46:30.387Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreihvacc5k7wctlzutkf2424vpqebpkbdw75xt4km3ui6och46vnbyq, nBlocksRead=7232, err=context cancelled
2021-12-02T16:46:30.387Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:194	traversal completion check failed, nBlocksRead=7291, err=context cancelled
2021-12-02T16:46:30.387Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:194	traversal completion check failed, nBlocksRead=7168, err=context cancelled
2021-12-02T16:46:30.387Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreibg7g6c4kv4vp5hfenrxvc6db2xwe7p7vbiu4zgupyvtbok74guty, nBlocksRead=7441, err=context cancelled
2021-12-02T16:46:30.387Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreic2ad7kko2vnpzl4pexyxbvkltrwo4gndibrcxltibabqxubkro6q, nBlocksRead=7335, err=context cancelled
2021-12-02T16:46:30.387Z	ERROR	gs-queryexecutor	queryexecutor/queryexecutor.go:238	failed to advance traversal, link=bafkreiac7kn5yktamaffojvkna6xngxgkc5lnuqaiw3gugsrgnc4nao3aq, nBlocksRead=7356, err=context cancelled
^Csignal: interrupt
FAIL	github.com/ipfs/go-graphsync/benchmarks	78.581s

Are the defaults too aggressive with timeouts? Am I missing something obvious? I imagine one benchmark iteration shouldn't take tens of seconds, because the entire point of Go benchmarks is to run many iterations (at least hundreds or thousands) so the result can avoid noise or skew from setup/teardown work.

I noticed because I attempted to do a minor benchmark code cleanup, so re-running all benchmarks with go test -bench=. -benchtime=1x ./... is a quick way to see if I can run all benchmarks once and they still work. That basically hanged for a while and ended up failing :)

[hannahhoward]

@mvdan we should chat. honestly I tend to use the testground plan much more often for benchmarking cause it's much more robust. I should probably remove those regular go benchmark.... except of course testground itself has reliability problems.