How to re-run failed tests

[alanvardy]

Hello!

I've been really enjoying this tool and using everywhere I can, and have one question:

I am working on a codebase with intermittently failing tests, causing me to have to frequently re-run the whole suite. I'm leading a push to fix the intermittent cases, but it is not going to be quick.

In the meantime I am hoping to speed up my local development by being able to retry mix test with just failed cases in the case of failure. Something like:

    {:ex_unit, umbrella: [parallel: false], env: %{"MIX_ENV" => "test"}, command: "mix test || mix test --failed"},

(I am running this on an umbrella app, and a lot of the issues are the result of a large global cache causing tests to share state)

But I have not been able to find a way to pass anything beyond a single command into :command.

I understand that what I am asking for is an anti-pattern and you may not wish the add the option of doing this, if that is the case would you be able to point me to where I could make the change in a fork of my own?

Thank you 🙂

[karolsluszniak]

Hello Alan! Great to hear ex_check is useful to you!

Well, anti-pattern or not, it's an interesting problem to tackle and one that's not so uncommon :)

Fortunately, it's possible with existing feature set of ex_check. Elixir's ports can't run such shell command directly, but you may create a proper shell script, say test/test_and_retry.sh, with content like the following:

#!/bin/sh
mix test || mix test --failed

Be sure to make it executable with chmod +x test/test_and_retry.sh and then override the default ex_unit tool configuration in .check.exs (run mix check.gen.config if you don't have one yet):

[
  tools: [
    {:ex_unit, "test/test_and_retry.sh"}
  ]
]

That's it. I've just tested it with a sample project with randomly failing test case and it works like a charm.

---

Btw ex_check also allows to run a tool as a dependency of other failing tool with config like below:

[
  tools: [
    {:ex_unit_retry, "mix test --failed", deps: [{:ex_unit, status: :error, else: :disable}]}
  ]
]

It works as expected except it doesn't mute the error from the original failing ex_unit call so the exit status from ex_check will also be non-zero. This would be an useful solution if you'd like to alarm yourself and the CI upon such failure but see if the retry helped the suite to pass - something that you could go for if you'd declare an open war to your failing tests and would like to hunt them down without mercy. You could even declare a chain of such retries to if a single retry is not enough. Otherwise, if you want the successful retry to make things green, then you're better off with the script.

Hope that helps.

[karolsluszniak]

Oh, there's also a way to do this without an extra script file, with mix cmd. Plus, added the --color flag to enforce colors in mix test called indirectly (as direct calls have colors enabled autmatically by ex_check hacking the mix command).

[
  tools: [
    {:ex_unit, ["mix", "cmd", "mix test --color || mix test --color --failed"]}
  ]
]

[alanvardy]

YES! This is perfect. This massively cut down my run times AND justified my recent ram upgrade! Thanks @karolsluszniak !