run zBenchmark as part of tests ?

[FObersteiner]

This is more of a suggestion, not an "issue" ;-) I just started learning Zig, coming from a go background. So naturally, I was looking for benchmarking options - and found your blog-post + library. Very nice work!

What confused me was that the readme suggests to run benchmarks as part of main(). Naturally, I would expect benchmarks to be part of tests (spoiled by go maybe). As a Zig beginner, I'm still fighting the build-system (also spoiled by go ^^ - but hey, at least the compiler is nicer to me than the one for Rust). I finally managed to get your library to work in tests by adding it to the compile step as returned from addTest - not too difficult after all, but maybe a useful addition to the readme?

However (here comes the issue..), when running zbench as part of tests (zig build test, with "test" being the option to run my test-step), the result is shown but the test fails:

zig build test     1798ms < Sat 28 Oct 2023 04:17:00 PM CEST
run test: error: Total operations: 1590
benchmark            time (avg)   (min ... max)        p75        p99        p995
--------------------------------------------------------------------------------------
My Benchmark         1.0ns        (3.0ns ... 7.0ns) 4.0ns      5.0ns      6.0ns

The test case only contains your example code

test "bench test" {
    var customAllocator = CustomAllocator.create();
    var resultsAlloc = std.ArrayList(zbench.BenchmarkResult).init(customAllocator.allocator);
    var bench = try zbench.Benchmark.init("My Benchmark", customAllocator.as_mut());
    var benchmarkResults = zbench.BenchmarkResults{
        .results = resultsAlloc,
    };
    try zbench.run(myBenchmark, &bench, &benchmarkResults);
}

There are no things that actually fail here, no? Not sure what's going on...

[hendriknielaender]

Hey @FObersteiner,

Thank you for bringing this up and for your kind words on the library!

You've hit the nail on the head regarding the placement of benchmarks within the main() function. It does make more sense to align this with common practice and have benchmarks as part of tests (will be added into the readme).

Regarding the error you encounter when running zbench as part of tests, I will have a look into this in the next few days to determine the cause and work on a solution.

Thank you once again for your input and for taking the time to provide a description of your experience.

[FObersteiner]

By the way, I noticed that the tests containing the zBench don't "fail" if I build a binary, via a std.Build InstallArtifact in the build.zig. If it's a RunArtifact, it produces the above mentioned "error". However, this only works the first time the code is compiled (and run). If you compile it again, nothing happens since the result is cached - unless you change the code of course (same as with "normal" tests). The open question here is what makes a test "fail" if there is no assert or expect that could fail?

Anyways, the bottom line here is that it makes most sense to "build" the benchmark test, as one builds examples. Ok this is mostly me still learning to use the build-system ^^ Nevertheless, hope this is useful in some way ;-)