Test failures are reported in reverse order in version 1.7
Closed this issue · 6 comments
Before version 1.7 test failures were reported in the order they appeared in the test sketch. This changed in 1.7 where they are reported in reverse order.
I don't know that you can guarantee that order --- C++ instantiation rules
say nothing about this order AFIK --- so the order they get in the list is
(potentially) random. That's why I went for alphabetical, since at least
it is deterministic.
Factoring into a setup_all() and loop_all() does make for less code in the
main loop --- this is probably pretty minor.
Why not a Test::setup()? It's not completely necessary, but there is
additional code that must be executed before looping.
On Sun, Feb 24, 2013 at 3:15 AM, Matthew Murdoch
notifications@github.comwrote:
Before version 1.7 test failures were reported in the order they appeared
in the test sketch. This changed in 1.7 where they are reported in reverse
order.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/17.
Initialization order of globals (which I guess the generated test structs are) isn't guaranteed in C++, so you are right that relying on this behavior is a bad idea. For the time being, putting back the execution order based on the (undefined in theory but defined by avr-gcc in practice) will be backward compatible with previous versions (at least until avr-gcc changes).
Let's go for your more robust, deterministic approach in 2.0.
Actually, the initialization order is guaranteed to be top-to-bottom within a single translation unit (e.g. a .cpp file, or all .ino files combined). See point 3 at http://en.cppreference.com/w/cpp/language/initialization#Dynamic_initialization or https://stackoverflow.com/a/7222755/740048
Reverting to the previous behaviour might not be so bad, or perhaps adding a "sort by name" flag could give the best of both worlds?
2.0 has been out long enough that this would be a breaking change. The list of tests is generated before the setup function is called, and made alphabetical. Changing this leads to a chicken-egg problem of which setup would actually sort them, and still leads to non-deterministic problems for projects spread among multiple files. If you want to control the order, you can prefix the tests with a numbering scheme: t01_config, t02_io, etc.
Oh, right, setting a "do not sort" config in setup() is too late, so that won't work. Oh well, sorting isn't really bad anyway.
1.7 is no longer supported