Consider moving validate_observation to .judge() to accommodate observation-free tests
JustasB opened this issue · 16 comments
Currently, the observation passed into a test constructor is validated in Test.init()
https://github.com/scidash/sciunit/blob/dev/sciunit/tests.py#L34
However, in cases where a model does not have observations to compare against (e.g. many in the NML-DB), the validation fails if no observation is included in the constructor (I still use the test to generate the prediction, I just don't judge() it).
I work around this by passing in a dummy observation, which satisfies the validator.
It seems that a more general approach would be to validate the observation inside the .judge() method, instead of the constructor. If it were validated inside judge(), a test of a model with an observation could be performed, and the observation would be validated.
The validation could be done on this line:
https://github.com/scidash/sciunit/blob/dev/sciunit/tests.py#L179
As far as I can tell, this would not break any depending code, and would allow observation-free test creation.
@rgerkin Thoughts?
This would also allow chaining of tests, where the results of a simpler test(s) could be used in the prediction of a more complex test.
I.e. an observation would be available for the more complex test, but not necessarily for the simpler one(s).
@JustasB
So basically you want to be able to call generate_prediction only?
Have you seen the TestM2M class? This allow for test with no observations. Maybe there would be some way to combine these ideas.
So basically you want to be able to call generate_prediction only?
Yes
Have you seen the TestM2M class?
I see what you mean. I'm not sure that the tests I'm building will never have observations though. I think I'm just looking for a way to make the observations optional in general up until they're absolutely needed, e.g. by judge()
OK, I'll make a branch and test this possibility against a bunch of other stuff in NeuronUnit.
Actually, maybe you can make it @JustasB and forge ahead using that and let me know how it goes. I'll test it later.
ok sounds good. what do you use to run the unit test suite?
The test.sh script in the root directory, which basically wraps python -m unittest.
Same thing for neuronunit.
got it. will use that for test for breakage
Does neuronunit/test.sh work for you now? I get this:
======================================================================
ERROR: test_aibs (neuronunit.unit_test.resource_tests.AIBSTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "neuronunit/unit_test/resource_tests.py", line 45, in test_aibs
observation = aibs.get_observation(dataset_id,'rheobase')
File "neuronunit/aibs.py", line 78, in get_observation
value = cmd[kind]
KeyError: 'ef__threshold_i_long_square'
Stdout:
Getting Rheobase data value for from AIBS dataset 354190013
sciunit/test.sh works
I think AIBS changed the API. The threshold is now under: cmd[u'ephys_features'][0][u'threshold_i_long_square']
That was it. Test.sh runs fine with the above fix.
This has been done. Sci- and neuronunit test.sh pass. Tests can now be initialized without observations.
Just in dev