mlcommons/logging

RCP checker finds benchmark name in surprising, undocumented way

Closed this issue · 0 comments

The rcp checker needs to know which benchmark it should be checking the RCPs for. It finds the benchmark name in a surprising and undocumented way: by parsing it out of the FOLDER command line argument. But this means that if you run the RCP checker to check some logs in the current working directory (or in a directory that isn't named exactly correctly), it will fail with the confusing log message:

python3 -m mlperf_logging.rcp_checker --rcp_usage=<usage> --rcp_version=<version> .
...
ERROR - Cannot find any RCPs, RCP test FAILED`

My recommendation would be to find the benchmark name from the submission_benchmark log lines (:::MLLOG ... "key": "submission_benchmark", "value": "<benchmark_name>).

Another possible solution would be to add a new command line option (--benchmark <benchmark_name>, or something like that).

In any case, if the benchmark_name can't be deduced, the error message should be that the benchmark_name can't be deduced, rather than Cannot find any RCPs.