openml/OpenML

Check storage of user-defined measures

joaquinvanschoren opened this issue · 8 comments

As noted in #409, user-uploaded evaluation measures are not returned by the API.
It seems that for all runs since 595627, this is not being stored in the database.

Probably a bug. Needs to be fixed, and runs re-evaluated.

Yes, it is a bug. Based on a legacy naming convention in the XML.

As a consequence, currently user defined measures are not properly stored in the database and user uploaded 'check variables' are not used (i.e., there will probably be no api warnings in the database too).

The information is still in the xml's and can be retrieved though. need some time to fix this properly.

This is fixed but some rerunning is needed, see #426

Would be nice if this could be fixed, as this is blocking me a bit. ;)

Ping ;)

All runs should be re-evaluated by now.
@PhilippPro can you check and close?

This code still give different results without and with runtime...

listOMLRunEvaluations(run.id = 2253351)
listOMLRunEvaluations(run.id = 2306596)

You're right, we missed a few. Processing now...

Thanks, solved, can be closed.