Wish: AssertNotImplemented() should return True

[RickBorup]

AssertNotImplemented() should return True, not False. Here's why:

Reason 1 (tautological): If a test is not implemented, then by definition it's always true that it is not implemented.

Reason 2 (practical): By returning False, AssertNotImplemented() trains us to ignore failing tests.

The general rule of unit testing is that all test results must be green before code is committed or shipped ("all green or no go").

When tests that are not implemented return False, we train ourselves to ignore those red results because we know they're expected. This is a problem because if we get used to seeing and ignoring any red results, even if only those from tests that are not implemented, we might miss a red result from a newly failing test that is implemented.

This can be particularly problematic in larger test suites.

Example: A test suite has 100 tests. The 76 that are implemented return True (green). The 24 that are not implemented are there as placeholders for future tests, and they return False (red). We become accustomed to seeing and ignoring the 24 red rows every time we run the whole test suite. If we then modify or refactor a method and its test subsequently fails, we might not notice there are now 25 failing tests instead of only 24. We might go ahead and commit or ship the code even though not all test results are green, and if we do that we have just released a bug.

On the other hand, if tests that are not implemented return True (green), we know all red results require action and we know we can't commit and ship the code until all tests are green.

[eselje]

Perhaps a 3rd color would be merited for tests not implemented. Some use the tests as a "to-do" list, with every red being an item unimplemented or failing.

…

On Tue, Mar 20, 2018 at 4:18 PM, Rick Borup ***@***.***> wrote: AssertNotImplemented() should return True, not False. Here's why: Reason 1 (tautological): If a test is not implemented, then by definition it's always true that it is not implemented. Reason 2 (practical): By returning False, AssertNotImplemented() trains us to ignore failing tests. The general rule of unit testing is that all test results must be green before code is committed or shipped ("all green or no go"). When tests that are not implemented return False, we train ourselves to ignore those red results because we know they're expected. This is a problem because if we get used to seeing and ignoring *any* red results, even if only those from tests that are not implemented, we might miss a red result from a newly failing test that *is* implemented. This can be particularly problematic in larger test suites. Example: A test suite has 100 tests. The 76 that are implemented return True (green). The 24 that are not implemented are there as placeholders for future tests, and they return False (red). We become accustomed to seeing and ignoring the 24 red rows every time we run the whole test suite. If we then modify or refactor a method and its test subsequently fails, we might not notice there are now 25 failing tests instead of only 24. We might go ahead and commit or ship the code even though not all test results are green, and if we do that we have just released a bug. On the other hand, if tests that are not implemented return True (green), we know *all* red results require action and we know we can't commit and ship the code until all tests are green. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#13>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHy8TPCVyh4bC17MEjrmTjC19RZkUv71ks5tgXIvgaJpZM4SyqyB> .

[RickBorup]

Good suggestion. I'd be OK with a third color, maybe orange to signify 'attention needed' rather than a 'stop' condition like red.

[eselje]

All the colors are user-configurable on the Options page. I'm thinking maybe default to yellow to go along w/the red/green stoplight theme?

…

On Wed, Mar 21, 2018 at 12:31 PM, Rick Borup ***@***.***> wrote: Good suggestion. I'd be OK with a third color, maybe orange to signify 'attention needed' rather than a 'stop' condition like red. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHy8TCQU4bhlJjeH-jNCSBNKyxUP5HvFks5tgo5lgaJpZM4SyqyB> .

[RickBorup]

/stoplight theme/ That's what I was thinking, too. Yellow would be good for not-implemented.

[cwollenhaupt]

If a test shouldn't run, because it's not done yet, you should mark the test procedure as "HIDDEN" and FoxUnit won't attempt to execute it. Once you start working on a test case remove the "HIDDEN" keyword to mark the test as active. HIDDEN is equivalent to the "Ignore" attribute you can assign to a test in other languages.

Unit tests are really a binary thing. They pass or they fail, there's no try, to quote Yoda.