Access to "provider_job_id" in ActiveJob?

[slothbear]

ActiveJob has started to add a provider_job_id so that jobs can access the transaction in the provider. I was wondering if it is possible to do that now with queue_classic, or if it is planned? I'd really like for my jobs to be able to access the queue_classic_jobs row for the job. I've looked at the QC:Queue#enqueue method and tried to monkey patch it, but I'm not sure how to extract the ID. Thanks for making a great queue.

rails/rails#18821

[slothbear]

Following the examples already completed for other queue providers, I have explored some patches. They don't quite work. If anyone has guidance on where I've gone wrong, I'd be happy to do the work of creating the pull requests and tests for both Rails and queue_classic.

I added the code below to an initializer in my project. Here's how it works:

1. Add the provider_job_id accessor to Rails ActiveJob::Core.
2. Ask PostgreSQL to provide the ID of the row inserted into queue_classic_jobs by RETURNING ID.
3. Have the Rails QueueClassicAdapter set the provider_job_id with the return value from enqueue.

The ID is returned and successfully set via job.provider_job_id=...,
but when the job is run (via rake jobs:work), the value of provider_job_id is nil.

module ActiveJob
  module Core
    #.# Follow example of other queue providers that have been added.
    #.# https://github.com/rails/rails/commit/dd8e859829bfcfd8cb0287ce804055b827a35af1
    #.# In the other examples, this is included in the Concern `included do` block.
    attr_accessor :provider_job_id
  end
end

module QC
  class Queue
    def enqueue(method, *args)
      QC.log_yield(:measure => 'queue.enqueue') do
        #.# return the ID from the INSERT statement
        #.# http://stackoverflow.com/a/2944481/2464
        s="INSERT INTO #{TABLE_NAME} (q_name, method, args) VALUES ($1, $2, $3) RETURNING ID"
        res = conn_adapter.execute(s, name, method, JSON.dump(args))
      end
    end
  end
end

module ActiveJob
  module QueueAdapters
    class QueueClassicAdapter
      class << self
        def enqueue(job) #:nodoc
          #.# This call to set provider_job_id succeeds, but the value
          #.# is nil when provider_job_id is accessed when the job runs.
          job.provider_job_id = build_queue(job.queue_name).enqueue("#{JobWrapper.name}.perform", job.serialize)
        end
      end
    end
  end
end

[slothbear]

Reading my own comment, I noticed the call to job.serialize in the adapter's #enqueue method. ActiveJob::Core has methods for serialization, but none of the other queue adapter patches to add provider_job_id changed the serialization. Perhaps they don't need to... but I added it to my exploration anyway.

It makes sense that provider_job_id needs to be serialized when enqueued, but the ID itself is returned by the enqueue method. Chicken and egg? Or am I going about this all wrong?

[senny]

@slothbear Thanks for your reports and your time 💛. Although I'm not sure I follow all your discoveries 😊.

One thing to note is that the current approach with provider_job_id only grants you access to the row when enqueue time. During job execution provider_job_id wont be available. You can read more on the corresponding Rails issue:

The downside of all this is that you have access to the provider_job_id only at enqueue time. When your job is executed you cannot get from AJ. But I don't think is needed nor the purpose of this talk

Currently we haven't implemented provider_job_id but I'll have a look at it.

[senny]

For reference, here's the relevant part on the Rails end rails/rails@68e3279