onflow/cadence-tools

[Test] Allowing/expecting a test case to fail/panic

m-Peter opened this issue · 3 comments

m-Peter commented

Issue To Be Solved

Currently, there is no way to specify that a test is expected to fail/panic. For example:

flow test --cover MerkleProof_test.cdc
Running tests...

Test results: "MerkleProof_test.cdc"
- PASS: testVerifyValidProof
- FAIL: testVerifyInvalidProof
		Execution failed:
			error: panic: invalid proof
			  --> ./contracts/MerkleProof.cdc:12:16
			
- PASS: testVerifyProofWithLowerLeaf
- PASS: testVerifyWithHigherLeaf
Coverage: 100.0% of statements

For the testVerifyInvalidProof test case, the error: panic: invalid proof is the expected program behavior, however the test case is marked with a FAIL status.

Suggest A Solution

Add a built-in method/matcher, which wraps a method call execution and expects a panic with a certain error message.

Test.assertFailsWith(fun(): Void {
    merkleProof.verifyProof(
        proof: [proof.decodeHex()],
        root: root.decodeHex(),
        leaf: leaf.decodeHex(),
        hasherRawValue: 1
    )
}, "invalid proof")

Context

onflow/developer-grants#148

m-Peter commented

cc @SupunS @turbolent
Correct me if I'm wrong, but other than running this through a script and verifying that the ResultStatus is failed, I didn't find any other way. (the script workaround applies only to integration tests, for unit tests this remains impossible)

m-Peter commented

I have come up with a draft for the following behavior:

➜  flow-merkle-proof git:(main) ✗ ~/Dev/forks/flow-cli/cmd/flow/flow-x86_64-linux- test --cover MerkleProof_test.cdc
Running tests...

Test results: "MerkleProof_test.cdc"
- PASS: testVerifyValidProof
- FAIL: testVerifyInvalidProof
		Execution failed:
			error: Expected error message to include: "not what I expected!". Found: Execution failed:
			error: panic: invalid proof
			  --> ./contracts/MerkleProof.cdc:12:16
			
			  --> 7465737400000000000000000000000000000000000000000000000000000000:21:8
			
- PASS: testVerifyProofWithLowerLeaf
- PASS: testVerifyWithHigherLeaf
Coverage: 100.0% of statements

➜  flow-merkle-proof git:(main) ✗ ~/Dev/forks/flow-cli/cmd/flow/flow-x86_64-linux- test --cover MerkleProof_test.cdc
Running tests...

Test results: "MerkleProof_test.cdc"
- PASS: testVerifyValidProof
- PASS: testVerifyInvalidProof
- PASS: testVerifyProofWithLowerLeaf
- PASS: testVerifyWithHigherLeaf
Coverage: 100.0% of statements
SupunS commented

Add a built-in method/matcher, which wraps a method call execution and expects a panic with a certain error message.

Test.assertFailsWith(fun(): Void {
    merkleProof.verifyProof(
        proof: [proof.decodeHex()],
        root: root.decodeHex(),
        leaf: leaf.decodeHex(),
        hasherRawValue: 1
    )
}, "invalid proof")

This sounds like a nice solution! 👌

Maybe the name of the function could be something like expectFailure().

Going one step further, we could probably make the second parameter accept another string-matcher (rather than a string), so we could also support asserts like contains/startsWith/etc., for the error message. But of course, doesn't need to be in the first iteration, maybe a future improvement.

  • 🚀2