Track WIP tests with test tags

[zachallaun]

A funny side effect of testing (and updating tests) being so easy is that I want to write tests that I know are wrong. Not tests that are failing, mind you, but tests that are passing — I just know they’re passing on the “wrong” value.

For instance, for the semantic diff stuff, I have a diff_test file of formatted diffs, and sometimes I know that the diff could be better for a specific case. I don’t want to write a failing test, because it’s not obvious yet how I’m going to fix it, but I want the example in there because I want to know when it changes. It’s a work in progress.

The problem, though, is that it feels bad to commit a test that I know isn’t right. It just is what it is — it’s the current state of the system — but I’m still working on it or am coming back to it later. These WIP tests are kind of like a persistent REPL, a reminder of what’s happening right now. But I often don’t commit them because I’m afraid I’ll forget about them. After all, the tests pass!

—

I propose adding a new feature to Mneme based on a conventional (and configurable) :wip tag.

This would use ExUnit tags to mark a test as a work in progress. The Mneme runner would pick up on these and report them as warnings at the end of your test run. Your tests pass, but you’ll see a warning with a summary and list of any files containing WIP tests at the end. Getting rid of the warning is as easy as deleting the tag.

@tag :wip
test "parses the thing correctly" do
  auto_assert {:ok, …} <- parse(…)

  # should be handling this case
  auto_assert :error <- parse(…)
end

Not sure yet whether this should use ExUnit tags, @tag :wip, or Mneme options, @mneme wip: true. One benefit of ExUnit tags is that I could make the test run return an error code if any WIP tests run, even if they pass, and you could get around that by doing mix test --exclude wip. Error code could be useful for CI stuff.

[mindreframer]

Interesting idea! I am myself fine committing work-in-progress code on a feature branch and keeping things simple. I might squash some commits before merging, if I really want to the progress to be scrutinized during the code review, but for private things I usually do not care much, as long the final result looks OK.

Just so you know: features do not come for free :) Implementation, documentation, regressions and maintenance. If there is a way to get a similar workflow with non-code approaches, maybe it should be preferred.

[zachallaun]

@mindreframer Thanks for sharing your thoughts!

Just so you know: features do not come for free :) Implementation, documentation, regressions and maintenance. If there is a way to get a similar workflow with non-code approaches, maybe it should be preferred.

Agreed, and I don’t think the feature set is going to grow much at this point. I think this is similar to ExUnit’s @tag :skip, however, which you might use to to skip a known failing test that you plan to address later. That prevents the test from running, of course, which means Mneme can’t update. I think this might enable a similar (and I think better) workflow.

[zachallaun]

I've added a :wip flag for now as a private option, which means it doesn't appear in the docs, while I try it out for a little while and see whether it's actually a worthwhile addition.

@mneme :wip
test "something" do
  ...
end

# after tests run, prints "[Mneme] 1 wip" in the after-run summary

[zachallaun]

This didn't prove very useful in practice, removed.