Test against original Matlab code
Opened this issue · 5 comments
@bmurphy-usgs, @erigler-usgs
Starting a new issue spawned from the discussion in #1.
We need to validate this code against the original Matlab code for some explicit geometries. I'm not sure if we validate against the geomag-imp code base, or actually go back to the original Matlab for this. I'll leave that up to you all. But, it would be great to get some actual values for specified cases and directly test against those in the test suite.
#2 laid the foundation for the test-suite with some initial basic tests for the code. It should be easy to expand and put the new test cases in there with references to where original values are taken from when needed in comments.
I was just going through this and wondering if anything has transpired with the validation, or if I needed to do anything for it on my end?
Sorry, haven't worked on this at all, got distracted by other things... If you can make some progress on this that would be great; I can still try to help, although not sure when I'll have a chance to get to this...
I don't have any original Matlab code, so I am unable to do the direct comparison... I did incorporate some basic tests for sanity though when you found the previous error. So, everything is passing the tests now, but that could still benefit from a little more rigor.
Hi guys, sorry I'm only chiming in now. I'm happy to help with this, although I probably can't do much for at least a week or two. If @bmurphy-usgs wanted to take a stab, he should be able to find the original Matlab code here: https://code.chs.usgs.gov/erigler/geomag-conus-secs. @greglucas , if you want to take a stab, let me know, and I'll zip up the files and email to you.
To be clear, we cannot make this Matlab code accessible to the public, given that it does not belong to us. Therefore, I'm not sure how best to incorporate it into a validation suite. We might document a comparison of the original Geomag-IMP and the ML code better than I did, then set aside the original Geomag-IMP code for future comparisons. Or maybe we just use the original ML code to generate a suite of results files against which we validate moving forward.
I'm not sure which approach is best.
No problem at all, I haven't exactly been keeping up on this over the past several months either. I'm actually just beginning to pick this up again to try and use it for some other work which is why I brought this up again.
My real desire is not necessarily to make it into a full-blown validation to be run every time. I'm really more interested in a sanity check because Ben already found one error :) This could involve a set of points that we run through each routine to test. A mix of SECs below/above, near/far from observations. We could easily drop the expected output arrays into the test suite too without needing to run the Matlab code every time.