sdv-dev/SDGym

The returned `Evaluate_Time` does not include results from all metrics

npatki opened this issue · 0 comments

Environment Details

  • SDGym version: 0.8.0 (latest)

Error Description

According to the SDGym docs, the final results include a column called Evaluate_Time. This column is supposed to include the total time it took to evaluate the synthetic data, based on the metrics provided.

But in practice, we see that the Evaluate_Time computation is hardcoded such that it only ever includes the time to run the quality report, and the NewRowSynthesis metric (if provided):

SDGym/sdgym/benchmark.py

Lines 329 to 335 in 50eac8e

evaluate_time = None
if 'scores' in output or 'quality_score_time' in output:
evaluate_time = output.get('quality_score_time', 0)
for score in output.get('scores', []):
if score['metric'] == 'NewRowSynthesis':
evaluate_time += score['metric_time']

Expected Behavior

When running a benchmark, it is possible to input additional SDMetrics to be used for evaluation. If I add new metrics, I expect that the time it takes to compute them should be included in the final Evaluate_Time that is reported.