Indicate which run fails when exiting with nonzero exit code

[Julian]

When running hyperfine -m on a command which exits with non-zero exit status (or even with the default -m), it seems like it would be nice to know which run number was reached. Specifically here's an intentionally broken program

⊙  cat fail_after_one_second.py                                                                                                                                                                                                                                                                                     julian@Airm
import sys
from pathlib import Path
from time import time

if time() - Path(__file__).stat().st_mtime > 1:
    sys.exit(2)

and if I run:

⊙  touch fail_after_one_second.py; hyperfine -m37 'python3 fail_after_one_second.py'                                                                                                                                                                                                                                     julian@Airm
Benchmark 1: python3 fail_after_one_second.py
Error: Command terminated with non-zero exit code: 2. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.

I don't see any information about what run count it was that failed.

(In particular in my case this is nice as a crude way of knowing just how flaky a flaky command is -- I want to know if it was the first execution that failed, or the 7th, or...)

[sharkdp]

Thank you for your request. I agree this can be improved.

Note that you can get the information today by using -i/--ignore-failure and --export-json:

▶ touch fail_after_one_second.py; hyperfine -m37 'python3 fail_after_one_second.py' --export-json results.json -i
▶ jq -c '.results[0].exit_codes' results.json
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
 0,0,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,
 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]

[Julian]

Aha! That's quite useful to know, I should have thought of looking at the JSON. Thanks for the temporary workaround! (And for the great tool of course.)