Property tests during compilation are getting in the way...

[asmodehn]

Hi everyone, here is some feedback...

After using type_class for a while, I reached the conclusion that running property tests during compilation doesn't match the "usual elixir developer" expectations. Well, at least mine, I'll let everyone else make their own mind about it while I attempt to make my case.

I see mainly two reasons:

• False positives: if mix compile succeed, it might just be because the values generated didn't break the property, but the property may still be unsatisfied for values that weren't generated. Further mix test runs will not trigger the property tests again, and therefore will not expose the problem. We now have "broken code" compiled and tests passing...

• Breaking the usual code / test cycle: the developer is writing tests & modifying code & writing tests while running mix test very frequently to see what tests are impacted by the code changes. I see this situation being more common with dynamic languages, where "exploratory coding" is easier, compared to compiled languages, where compile time make that somewhat less fluid...
  In this usecase, a developer can modify code and then running mix test will recompile, and maybe break on compilation (property test failed). Where is the code triggering the problem ? The property might be very remotely related with the changed code, and tracking it down maybe difficult, because now the developer cannot run tests anymore, until the cause for the property failing is found... Usually having multiple tests failing will indicate the code path that was affected by a change, but by running type_class property tests on their own, in the compilation phase, it has become impossible to run any other test to help track down the problem.

What does everyone else think ?

Overall I find type_class very useful for structuring code, but I would prefer typeclass property tests to run with mix test.
Especially because a codepath can jump through various protocol implementation in various files, having tests failing along the path would help... Maybe via a classtest macro, accepting the typeclass atom to test as parameter, that could be used with ExUnit just like doctest ?

[asmodehn]

I had a go at adding a macro to run property test for typeclass during mix test, you can find it there : https://github.com/asmodehn/type_class/pull/2/files

[cognivore]

In our fork we disabled proptests by default, but can run them on CI with an env flag!