Circular dependency detected while autoloading constant
diei opened this issue · 9 comments
Newly I have 12 static cron jobs scheduled with resque-scheduler starting at the same time. The scheduler always raises some exceptions during queuing the jobs the first time, at second time no errors are raised and the jobs are queued normally.
I use:
- resque 1.27.4
- resque-scheduler 4.3.1
- rails 5.1.6
- Ruby 2.5.1
I digged a little bit further and discovered that the issue is that the jobs started at the same time. And if I play with the Rails app config I get different errors.
With config.cache_classes = false I get only this error:
resque-scheduler: [ERROR] 2018-06-26T14:14:00+02:00: RuntimeError: Circular dependency detected while autoloading constant Core::Importer::Jobs::Job
["/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:509:in `load_missing_constant'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:202:in `const_missing'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler/util.rb:28:in `block in constantize'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler/util.rb:22:in `each'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler/util.rb:22:in `constantize'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:231:in `enqueue_from_config'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:214:in `enqueue'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:426:in `enqueue_recurring'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:144:in `block (2 levels) in load_schedule_job'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:210:in `do_call'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:254:in `trigger_now'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:296:in `block (3 levels) in start_work_thread'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:299:in `block (2 levels) in start_work_thread'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:285:in `loop'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:285:in `block in start_work_thread'"]
With config.cache_classes = true I get the error above and:
resque-scheduler: [ERROR] 2018-06-26T14:14:00+02:00: ArgumentError: A copy of Core::Importer::Jobs has been removed from the module tree but is still active!
["/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:496:in `load_missing_constant'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/activesupport-5.1.6/lib/active_support/dependencies.rb:202:in `const_missing'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler/util.rb:28:in `block in constantize'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler/util.rb:22:in `each'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler/util.rb:22:in `constantize'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:231:in `enqueue_from_config'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:214:in `enqueue'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:426:in `enqueue_recurring'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/resque-scheduler-4.3.1/lib/resque/scheduler.rb:144:in `block (2 levels) in load_schedule_job'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:210:in `do_call'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:254:in `trigger_now'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:296:in `block (3 levels) in start_work_thread'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:299:in `block (2 levels) in start_work_thread'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:285:in `loop'",
"/Users/test_user/.rvm/gems/ruby-2.5.1@core/gems/rufus-scheduler-3.5.0/lib/rufus/scheduler/jobs.rb:285:in `block in start_work_thread'"]
The config.eager_load does not matter in this case. The result is the same with true and false. Parallelly class loading in different threads seems to be the problem.
A workaround is to change the cron expression. If I let start the 12 jobs sequentially with 5 seconds delay, no error is raised.
Does anyone know a fix for this?
This seems to be a bug in the application. A dependency of a class is referencing the same target class before the dependency is fully loaded, this is causing a circular autoloading. I'll have to make sure that all classes can be loaded without this circular autoloading.
I'll have to make sure that all classes can be loaded without this circular autoloading.
Hello @rafaelfranca, have you created a new issue for this or why have you closed this one without doing anything?
Sorry, I made a typo in my message. I meant: You have to make sure all classes can be loaded without this circular autoloading. There is a bug in your application, it is not resque-schedule issue.
But the classes can be loaded without errors, if I change the cron expression so that they are started one after another. So basically there are no problems loading the classes.
The issue comes up if resque-scheduler loads the same class multiple times "simultaneously".
Is resque-scheduler starting the cron jobs in different threads or sequentially in one thread? The real cause need not to be resque-scheduler but perhaps how Ruby or Rails do class loading (maybe not thread safe).
The classes can load fine in some ordering, but in others not. This is why you are getting this problem only occasionally. Sometimes the order your classes are loaded are different, and in those cases you get an exception. You need to make sure your classes can be loaded correctly in any order.
➕
We just hit this problem this week after changing one of our items in scheduler.yml to be the same as one of the others.
You need to make sure your classes can be loaded correctly in any order.
I have no clue what this would mean in my case. I have a Rails app which is doing nothing out of the ordinary when it comes to loading the app.
Got the same issue out of the blue Yesterday. The bug was defining task 'resque:setup' => :environment twice in rake files. What I don't get, is why this started to reliably occur yesterday when it was working OK like this for 4 years and nothing related seems to be changed in the codebase.
The usage of the zeitwerk mode introduced with Rails 6 seems to fix this problem as well.
In case it helps anyone, I think this is actually a bug in ActiveSupport that can occur anytime you have multiple threads autoloading a nested constant at the same time. This happens in resque-scheduler when you have multiple jobs configured to start at the same time. A separate thread is created to load each job class and call #enqueue and these threads can run simultaneously.
Related Rails issue: rails/rails#33209