Temp file not equal to input file in `test/support/integration.ex`

[pharesdiego]

System

• Windows' WSL Ubuntu 22.04.2 ETS

Description

When running mix test, the run_test:56 function in test/support/integration.ex sometimes does not generate equal input and temp files thus making the test Mneme.Integration.RaiseTest fail:

Input file test_integration/raise_test.exs:

defmodule Mneme.Integration.RaiseTest do
  use ExUnit.Case
  use Mneme

  describe "auto_assert_raise" do
    test "defaults to no message" do
      # y
      auto_assert_raise ArgumentError, fn ->
        error!()
      end

      # y
      auto_assert_raise ArgumentError, &error!/0
    end

    test "can also match a message" do
      # k y
      auto_assert_raise ArgumentError, "message", fn ->
        error!("message")
      end
    end
  end

  defp error!(s \\ "message") do
    raise ArgumentError, s
  end
end

Temp file tmp/Mneme.Integration.RaiseTest/test-Mneme.Integration.RaiseTest-9b9ed38c/integration_test.exs:

defmodule Mneme.Integration.RaiseTest do
  use ExUnit.Case
  use Mneme

  describe "auto_assert_raise" do
    test "defaults to no message" do
      # y
      auto_assert_raise ArgumentError,
                        fn ->
                          error!()
                        end

      # y
      auto_assert_raise ArgumentError,
                        &error!/0
    end

    test "can also match a message" do
      # k y
      auto_assert_raise ArgumentError, "message", fn ->
        error!("message")
      end
    end
  end

  defp error!(s \\ "message") do
    raise ArgumentError, s
  end
end

Notice the difference in indentation:

# Input
      auto_assert_raise ArgumentError, fn ->
        error!()
      end
---
# Temp
      auto_assert_raise ArgumentError,
                        fn ->
                          error!()
                        end
---

What I have tried

Looking at the run_test function:

 def run_test(test) do
    ...
    file_path = Path.join([test.tmp_dir, "integration_test.exs"])
    File.write!(file_path, test.test_code)

   ...
    code_after_test =
      file_path
      |> File.read!()
      # This allows us to modify the test file in-place without the
      # `code_after_test == test.expected_code` check failing
      |> String.trim_trailing("#")

I think test.test_code is not an exact representation of the input file contents due to:

defmacro build_tests!(wildcard) when is_binary(wildcard) do
...
            data = %{
              ...
              test_code: unquote(test_code(source)),
              ...
            }

...
---
defp test_code(source), do: source.private[:mneme][:test_code] |> eof_newline()

I don't know exactly what source.private[:mneme][:test_code] |> eof_newline() does so I can't say for sure that there is where the issue is, but maybe String.trim_trailing and other String-related methods (maybe even File.read/write?) works differently in Ubuntu

[zachallaun]

Thanks for the report!

I encountered this myself and believe it is a change in the formatter between Elixir v1.14.3 and v1.14.4. Since we use the Elixir formatter to format output, and the integration tests are doing a direct text comparison, changes in the formatter will cause the test to fail.

If you are able, can you please try upgrading to Elixir v1.14.4 and see if the problem persists?

[pharesdiego]

Yep! Upgrading to 1.14.4 fixed the issue, thanks!

[zachallaun]

Thank you for confirming!

I'm doing some refactoring now of the integration tests that will allow us to mark certain modules to only run if Elixir matches a given version spec:

# version: >= 1.14.4
defmodule Mneme.Integration.SomeTest do
  ...
end