RCP checker finds benchmark name in surprising, undocumented way
Closed this issue · 0 comments
The rcp checker needs to know which benchmark it should be checking the RCPs for. It finds the benchmark name in a surprising and undocumented way: by parsing it out of the FOLDER command line argument. But this means that if you run the RCP checker to check some logs in the current working directory (or in a directory that isn't named exactly correctly), it will fail with the confusing log message:
python3 -m mlperf_logging.rcp_checker --rcp_usage=<usage> --rcp_version=<version> .
...
ERROR - Cannot find any RCPs, RCP test FAILED`
My recommendation would be to find the benchmark name from the submission_benchmark
log lines (:::MLLOG ... "key": "submission_benchmark", "value": "<benchmark_name>
).
Another possible solution would be to add a new command line option (--benchmark <benchmark_name>
, or something like that).
In any case, if the benchmark_name can't be deduced, the error message should be that the benchmark_name can't be deduced, rather than Cannot find any RCPs
.