Possible deadlock when accessing constants in different threads

[alanhala]

Hello!

I'm running a Rails application, version 7.0.4. I'm using a custom background job processor built on top of ActiveJob. It uses Google Cloud PubSub as backend. The background job processor calls Rails.application.eager_load! manually before starting. After that a bunch of threads are spawned for listening to the topics and process messages/jobs.

After every deploy there is one instance in which jobs get stuck and don't finish. I've managed to add some debugging to the threads to print their backtrace and I found some pattern. They are stuck in one of these backtraces:

Thread: #<Thread:0x00007f700bcf2330@worker-2 /usr/src/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:332 sleep_forever> #214:
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/fields/content_type_field.rb:39:in `element'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/fields/content_type_field.rb:33:in `parse'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/fields/content_type_field.rb:25:in `initialize'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/field.rb:277:in `new'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/field.rb:277:in `new_field'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/field.rb:266:in `create_field'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/field.rb:187:in `field'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/field.rb:239:in `method_missing'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/header.rb:186:in `[]='
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/message.rb:1327:in `[]='
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/message.rb:2118:in `block in init_with_hash'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/message.rb:2111:in `each_pair'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/message.rb:2111:in `init_with_hash'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/message.rb:135:in `initialize'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:1043:in `new'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:1043:in `insert_part'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:1037:in `block in create_parts_from_responses'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:1037:in `each'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:1037:in `create_parts_from_responses'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:878:in `mail'

Thread: #<Thread:0x00007f700ae6e098@worker-8 /usr/src/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:332 sleep_forever> #170:
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:596:in `email_address_with_name'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:674:in `email_address_with_name'

Thread: #<Thread:0x00007f700bd0ca00@worker-2 /usr/src/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:332 sleep_forever> #156:
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/parsers/address_lists_parser.rb:6:in `<main>'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/mail-2.7.1/lib/mail/elements/address.rb:3:in `<main>'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:596:in `email_address_with_name'
/usr/src/app/vendor/bundle/ruby/3.1.0/gems/actionmailer-7.0.4/lib/action_mailer/base.rb:674:in `email_address_with_name'

There is only one Thread with the backtrace shown in 3. and that one should also belong to 2. as it is calling also the method email_address_with_name.

3. is stuck in the constant declaration Mail::Parsers::AddressListsParser, 2. is referencing at some point the same constant and 1. is referencing Mail::Parsers::ContentTypeParser.

Can this be some kind of deadlock in the gem? Is there any other information that might be useful to provide or any tool I can use to dig further?

Sorry in advance if this is not an issue related to the gem!

[fxn]

Pretty sure this is unrelated to Zeitwerk, the actionmailer and mail gems load their code without Zeitwerk. I think you could open a ticket in the Rails repo.