Tags in tests ignored when executing specific `describe` blocks

[tugayac]

Elixir and Erlang/OTP versions

Environment

Elixir & Erlang/OTP versions

Erlang/OTP 27 [erts-15.2.5] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

Elixir 1.18.3 (compiled with Erlang/OTP 27)

Operating system

Ubuntu on WSL, result of uname -a:

Linux <NODENAME> 6.6.87.2-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

and result of cat /etc/lsb-release:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

Operating system

Ubuntu 22.04.3 on WSL

Current behavior

When running mix test against a describe block, the tags configured to be excluded in the ExUnit test configuration are ignored.

Example: See the files here (the line numbers will be useful in describing the issue).

Command:

mix test test/list_ops_test.exs:6

(Line 6 is targeting the describe "count" do block) the exclude tag :pending is ignored.

Expected behavior

When targeting lines that contain test blocks instead of a describe block (e.g. mix test test/list_ops_test.exs:8:13:18, lines only contain test blocks in the gist), the tests tagged with :pending are included. This makes sense, as I'm specifically asking for those tests to be included.

However, when targeting a describe block (e.g. mix test test/list_ops_test.exs:6), I would expect tests tagged as :pending (per the ExUnit config in test_helper.exs) to be excluded.

Going through the mix test task code, I found it overrides the ExUnit configuration from test_helper.exs, specifically at this line, which explains the behavior for describe blocks above. However, the documentation for mix test does not indicate your ExUnit configuration will be overridden.

I'd expect one of the following:

1. mix test documentation states prior ExUnit.configure/1 calls will be overridden if any args are passed to mix test.
2. mix test only overrides arguments passed to mix test, instead of all the ExUnit configuration previously defined.

Solution 1 would make more sense, as solution 2 could become overly complex (how do you determine which args should override config and which shouldn't?). I only expect this to be a problem in local dev, can't think of a reason to run specific describe/test blocks in CI.

[tugayac]

FYI @joshprice, since we encountered this together ☝️

[josevalim]

Thank you for the issue.

This is by design. Tests are always excluded first and then they included. I can see why you would prefer the behavior described, but if we changed the behavior as you propose, we can also imagine someone opening the opposite issue, where they tag the tests as integration, excluding by default, and then they want to run them all at once in the describe, and they cannot.

So instead of supporting complex rules, which may or may not work, depending on the use case, we always exclude first, then we include

[tugayac]

Thank you for your response @josevalim 🙏. When arguments are provided to mix test, it makes sense for tests to be excluded first, and then included.

Then for test driven development, if I want to progress one test at a time, instead of running specific describe blocks, I should be running the entire test file (mix test test/list_ops_test.ex). Since I'm not overriding what to exclude/include via arguments, my ExUnit configuration in test_helper.exs is used as expected.