How to use Oban.Pro.Relay with Oban.Pro.Workers.Chunk

[AHBruns]

So, I have a chunk worker that calls a batch endpoint on a third party service. The idea being, I want to be able to enqueue a bunch of jobs from different places, then execute a single HTTP request on behalf of all those jobs. This works! It's a great way to expose a simple do X to Y API while using batching behind the scenes.

The natural continuation of this is to use Relay. That way callers can get results back + wait for their underlying batch call to complete. The goal is something like this:

def get_by_id(id) do
  {:ok, results} =
    id
    |> MyChunkWorkerThatDoesABatchCall.new()
    |> Relay.async()
    |> Relay.await()

  {:ok, results[id]}
end

// this should get batched along with calls from elsewhere in the app happening at the same time
the_record_i_wanted = get_by_id(the_id_i_of_some_record_i_want_to_get)

I'm not sure if this is bug report or a feature request, but right now it seems to half work. The jobs Relay.async enqueues are chunked together, but it seems relay doesn't map the jobs back correctly. Await seems doesn't seem to map the chunk rather than the underlying jobs back. So, if I Relay.async 100 jobs then Relay.await_many all those and they get execute in a single chunk, I'll get [{:ok, result}, {:error, :timeout}, ...]. If they get execute in 2 chunks, I get: [{:ok, result}, {:ok, result}, {:error, :timeout}, ...]. I would expect each await to return the result of the chunk it was a part of.

[AHBruns]

So, after looking into this more, I think this is due to only the lead job actually being executed by the executor which is where the job telemetry events are emitted, and it is those job telemetry events that make relay work (relay attaches to them an emits notifications based on them). To confirm this, I have a very bad work around that I've tested and works:

  @impl true
  def process([lead_job | non_lead_jobs] = jobs) do
    result = the_work_i_need_to_do(jobs)

    # work around here

    result
  end

By emitting relay notifications myself for all the non-lead jobs, I'm able to trick relay into working.

[sorentwo]

@AHBruns You've done some serious code diving. You did an excellent job connecting the dots! This is between a bug and a feature request, but it seems entirely reasonable 🙂. In the meantime, your workaround is just fine.

Note: I've edited your code sample to remove examples of inner Pro code