pg_notify not working when testing

[Deleted user]

Using notification when testing it seems like pg notifications are not triggered.
When using the dev env I have notifications working.
I guess this is related to the ecto Sandbox.

OTP 21
Elixir 1.8.1
ecto_job 2.0.0
ecto 3.0.0
postgrex 0.14.1

[mbuhot]

Yes, this is a consequence of ecto sandbox.

Notifications are transactional, and since the sandbox is implemented as a transaction that gets rolled back between tests, the notifications will never fire.

Not sure what the best work-around is - any suggestions?

[Deleted user]

That what my testing got me to think but that's not great news.
First, let's make it documented on the readme.
I would suggest adding an helper function to trigger the notification/processing of the queue from the tests.

[mbuhot]

Hi @steffenix, I may have a workaround for tests that rely on job processing to run:

1. Add a flag to the :test config to disable running the job queue on app startup:

if Mix.env() == :test do
  config :logger, :level, :warn
  config :ecto_job_demo, EctoJobDemo.Repo, pool: Ecto.Adapters.SQL.Sandbox
  config :ecto_job_demo, EctoJobDemo.JobQueue, nil
end

2. In the application start callback, only start the JobQueue if it is configured to run

  # application.ex
  def start(_type, _args) do
    children = [EctoJobDemo.Repo] ++ job_queue_children()
    opts = [strategy: :one_for_one, name: EctoJobDemo.Supervisor]
    Supervisor.start_link(children, opts)
  end

  def job_queue_children() do
    case Application.get_env(:ecto_job_demo, EctoJobDemo.JobQueue) do
      nil -> []
      config -> [{EctoJobDemo.JobQueue, config}]
    end
  end

3. In the test setup, checkout a connection, set the sandbox mode to shared and start the job queue:

 setup do
    :ok = Ecto.Adapters.SQL.Sandbox.checkout(EctoJobDemo.Repo)
    Ecto.Adapters.SQL.Sandbox.mode(EctoJobDemo.Repo, {:shared, self()})
    start_supervised!({EctoJobDemo.JobQueue, [repo: EctoJobDemo.Repo, max_demand: 100]})
    :ok
  end

4. In the tests, run the code that enqueues jobs, then trigger the JobQueue by sending a :poll message to the Producer process:

test "Run jobs from a test" do
    Enum.reduce(1..10, Ecto.Multi.new(), fn i, multi ->
      multi
      |> EctoJobDemo.JobQueue.enqueue(i, %{hello: i})
    end)
    |> EctoJobDemo.Repo.transaction()

    # Trigger job processing to run   
    send(EctoJobDemo.JobQueue.Producer, :poll)

    wait_for_jobs_to_complete()
  end

5. From the test, wait for jobs to complete by polling the job table:

def wait_for_jobs_to_complete() do
    if EctoJobDemo.Repo.exists?(EctoJobDemo.JobQueue) do
      Process.sleep(100)
      wait_for_jobs_to_complete()
    end
  end

Modified example code in the jobs-in-tests branch: https://github.com/mbuhot/ecto_job/blob/jobs-in-tests/examples/ecto_job_demo/test/ecto_job_demo_test.exs

If the above strategy works for your application, we can look at adding some helpers for triggering the jobs and waiting for completion to the library.

[Deleted user]

That seems good to me, on the waiting for completion side I would let people do whatever they want.
I am using it with phoenix sockets https://hexdocs.pm/phoenix/Phoenix.ChannelTest.html#functions and most of the test helpers are built in with timeout feature.

I also checked the popular email library https://github.com/thoughtbot/bamboo and the test helpers are also including a timeout mechanism.

What I think we can add is an helper on send(EctoJobDemo.JobQueue.Producer, :poll)