cta-observatory/ctapipe

Write cross-validation results immediately after cross validation

Closed this issue · 0 comments

maxnoe commented

Please describe the use case that requires this feature.
Currently, the CrossValidator keeps the results of the cross-validation, including the full result table in memory and writes it in finish method of the tools.

We should write the results immediately in the cross validation loop using write_table(..., append=True) to avoid keeping this information in memory unnecessarily.

Describe the solution you'd like

Write cross validation results out immediately after each step, not in finish.