Allow only output variables as reference results
klausschuch opened this issue · 8 comments
The current cross-check rules document does not specify which variables can be part of the reference results. Currently, all variables can be used as reference results.
If the exporting tool adds parameters (even fixed or constant ones) to the reference results and the importing tool cannot record parameters during simulation, such an exporting tool cannot publish valid cross-check results for such an FMU.
Example:
The fmusdk/bouncingBall FMU writes the parameters g and e as reference results (csv-header: "time","h","der(h)","v","der(v)","g","e").
All cross check results that contain only all other variables like (csv-header: "Time","h","der(h)","v","der(v)") will be dropped even though the actual simulation results are perfectly fine.
Are internal variables allowed as reference results or only outputs? In some importing tools I would not know how to get internal varibles (if the importing toolbox would not have the option to expose internal variables as additional outputs)
If we cannot make parameters as reference, handling parameters can only be measured indirectly.
Not very nice, I think.
On the other hand if the parameters have no significant influence on the output it's probably not a good test model...
These are test FMUs. We are trying to spot errors easily. By excluding entire classes of variables, we make problem detection harder. If you now declare that parameters are suddenly not allowed as reference variables that a number of FMUs and their references might have to be changed again.
Exporters could add additional outputs to their test models if they think these variables should be part of the reference results.
Discussion at the Design Meeting Regensburg: No objections. Let's do this.
Let's do this
What exactly did we decide in Regensburg?
Should we add a sentence to the rules that we allow all kinds of variables (if that was decided?)
How do we treat tools that can only record output values?
A note on synonymous variables: some FMUs define several real variables with different names but same value reference and variable type -> hence, results will be the same (I call these synonymous variables). It would suffice to only have one of these variables in the reference results - otherwise it would be only a waste of hard drive space (and the git-repo may grow beyond meaningful limits).
With respect of logging internal variables, the cross-check rules should state that the submitted xxx_out.csv file shall only contain variables that are requested as reference results (again: save hard disk space and bandwidth).