Convert all functional tests into bona fida tests
Closed this issue · 10 comments
I noticed that in the pay-related branches (adc, dac, etc) there's a functional test directory func_tests/ containing functional tests. I think these should all be moved into the tests directory, under a common theme. E.g. adc functional tests should all be kept in a adc/ subdirectory of tests/.
@brunofalmeida thoughts? This would help us track testing more effectively, and enable us to test multiboard functionality using the harness.
Wherever possible we should use the test harness, but there are many cases (especially in PAY and EPS) where it's impossible to use assertions to verify that something works. Examples include motor control, battery charging, and heater control where we need to verify things work using hardware tools and observation.
I suggest we keep separate folders for tests using the test harness and tests that must be run individually. @siddMahen
Maybe the naming scheme isn't obvious, but that was the logic for separating the folders
Ahh I see, that makes sense. Thanks for the clarification.
Would it be possible to incorporate them into the regular testing framework if I added a new command, called say, PROMPT which required a positive input to register as a successful test? Example usage includes
.. start motor testing
PROMPT WITH MESSAGE (Success/Fail)
You input success
.. testing continues
We could add an additional option that disables these kinds of assertions when we want to run automated tests, and then enable them for day-to-day stuff.
I'm thinking of PROMPT as an opportunity for the testing framework to wait for some external input @brunofalmeida
@siddMahen I think for some cases that would work, but there are many cases where we want to run a test for an indefinite amount of time and it's not a one-time program run (usually to see values changing over some time). Examples I can think of include heater temperature control in PAY and voltage/current measurements in EPS.
I see, ok. In that case, independent functional testing makes sense. I'll close this issue.
@siddMahen does the naming scheme tests for harness-based tests and func_tests for individual functional tests make sense, or do you recommend naming the folders differently?
I don't have much preference; I'd be ok with any naming scheme. The harness program accepts any directory so it'll require relatively little modification if we change tests to something else.