fukamachi/prove

Planning the number of tests?

eudoxia0 opened this issue · 5 comments

eudoxia0 commented

Prove is probably the most modern and extensible test framework for Common Lisp. But there's just one thing that's keeping me from using it: The requirement that you specify the number of tests to run. I know this is optional and the idea is that you set it to a fixed number after you've written the tests, but it still feels superfluous.

Is there some way -- maybe through subtest, or a define-suite macro -- to just skip the planning stage altogether?

Maybe have two global variables: The number of tests registered into a suite, and the number of tests run, and each test suite macro mutates these and afterwards compares them to see if any tests were skipped.

proofit404 commented

Maybe we can use cl-annot library to implement @skip helper similar to python unittest library. Or just use macro like (skip (is 4 4)).

Still new to Common Lisp so maybe I've missing some points.

fukamachi commented

Sorry, I don't get what is an actual problem since it's optional.
Put (plan nil) before tests if you want to suppress a warning "△ Tests were run but no plan was declared."

proofit404 commented

I doesn't understand why you make test plan such explicit thing? Why user need to care about number of tests at all? Non of test framework I used before do this in other languages.

  • 👍1
inaimathi commented

👍 to this issue. Any chance of getting something like

(defmacro prove:suite (&rest forms)
  `(progn
     (prove:plan ,(length forms))
     ,@forms
     (prove:finalize)))

into the repo? Seems like it would do away with the manual book-keeping, while still respecting whatever internal mechanisms might need that test count/finalize call.

eudoxia0 commented

I was reminded of this issue by this post, and I thought, since the suite macro is more complicated than the trivial implementation (not all forms might be tests), why not do it this: when plan or finalize aren't detected, simply don't say anything about the number of tests run or to be run. That way it can be elided and no-one notices.

Also, I've been testing some JavaScript recently, and I rather like Mocha's recursive suites feature, where you create nested test suites with associated descriptions, and get output like this:

 Number
    Integer
      Constructor & type predicate
        ✓ 1 is an integer
        ✓ 10 is an integer
        ✓ -3 is an integer
        ✓ 1234567 is an integer
        ✓ 1 is not a float
        ✓ 10 is not a float
        ✓ -3 is not a float
        ✓ 1234567 is not a float
    Float
      Constructor & type predicate
        ✓ 1 is a float
        ✓ 10 is a float
        ✓ -3 is a float
        ✓ 1234567 is a float
        ✓ 1 is not an integer
        ✓ 10 is not an integer
        ✓ -3 is not an integer
        ✓ 1234567 is not an integer
    Complex
      Constructor
        ✓ type predicate
        ✓ real part
        ✓ imaginary part
      If Im(z) = 0, simplify to a real
        ✓ type predicate

Could we get implement something like this for prove? Maybe as a plugin?

  • 👍5