Add TEST_RUNNER method to whitebox/integration tests and use it in test code

[mulkieran]

TEST_RUNNER's job is to call RUNNER, catch any StratisCLI exception that might be raised, and then reraise the exception, chained with a subtype of AssertionError.

Since there is no handy AssertionError, our test framework should define one. We can call it something like StratisCliFailureException and it should have a message like "Unexpected failure of CLI command during test". Since this is Python, the rest should be handled by our exception mechanism.

Then TEST_RUNNER should be called in place of RUNNER wherever RUNNER is being called within a test in order to test something.

The purpose of this is to make sure that if there is a test failure, then the unittest mechanism identifies it as a test failure, not as a test error which is what it does now.

[mulkieran]

The order of steps might be:

• Define TEST_RUNNER exactly the same as RUNNER, and then make the change from RUNNER to TEST_RUNNER everywhere that is appropriate in the tests. Note that this requires judgement, because even individual test methods may include some setup actions that use RUNNER, and these should not be changed.
• Now, redefine TEST_RUNNER to be different from RUNNER
  • Define the new exception
  • Write TEST_RUNNER so it catches one exception and then re-raises the newly defined exception

[bgurney-rh]

First, I have to find RUNNER. Is this it?

./tests/whitebox/_misc.py:172:RUNNER = run()

[mulkieran]

First, I have to find RUNNER. Is this it?

./tests/whitebox/_misc.py:172:RUNNER = run()

That's the one.

[bgurney-rh]

I made a quick list of the whitebox/integration tests that run RUNNER:

key:
test_unset
test_list
test_reset
test_set
test_destroy

logical:
test_destroy
test_create
test_snapshot
test_list
test_rename

_misc

physical:
test_list

pool:
test_destroy
test_create
test_init_cache
test_add
test_list
test_rename

[mulkieran]

@bgurney-rh : This code here bgurney-rh@e458059 is the right first step. Later you will want to do something like this:

def test_runner(command_line_args):
    try:
        RUNNER(commad_line_args)
    except StratisCliActionError as err:
        raise <the new error, not yet defined> from err

TEST_RUNNER = test_runner

and you will be done. But for now, just figuring out where TEST_RUNNER should be substituted for RUNNER is the right thing to do.

[bgurney-rh]

Now that pylint works on my Fedora 31 test machine without modification, I can create a new branch without my temporary modification to check.py, and create a PR from that one.