Problem with DelayedJob Psych gem when upgrading to Ruby 2.7.5
LucasCioffi opened this issue · 2 comments
LucasCioffi commented
Hi Everyone,
I ran into a parsing error after upgrading to Ruby 2.7.5.
This is due to a bug in the Psych gem which affects DelayedJob in an unexpected way. I explain my work-around (below), but I raise this issue, because I'm thinking that others might run into a problem like this, and that maybe the gem maintainers would like to prevent it in some way.
The error I saw was Last Error: invalid value for Integer(): "157653091864_1_" and happening during object.to_yaml in the DelayedJob gem (full stack trace is below): https://github.com/collectiveidea/delayed_job/blob/master/lib/delayed/backend/base.rb#L68
def payload_object=(object)
@payload_object = object
self.handler = object.to_yaml
end
In my gemfile I had this:
gem 'delayed_job_active_record', '~> 4.1.4'
I upgraded to 4.1.6 but it did not solve the problem.
After some investigation, I found that 157653091864_1_ referred to the name of a user's profile photo uploaded with the paperclip gem.
In the database, this user record has an attribute of photo_file_name with a value of "157653091864_1_" On that user record, when I change the file name to 157653091864_1_.jpg the error doesn't happen.
The error happens when I run my example my_method_goes_here for all users:
User.find_each do |user|
user.delay(priority: 3).my_method_goes_here
end
I see that I can prevent the bug from affecting my application by rewriting the method and making the class the payload object instead of the individual user record like this:
User.find_each do |user|
User.delay(priority: 3).my_method_goes_here(user_id)
end
This works because the user isn't passed in as an argument, so any JSON problems on the user record won't trigger any parsing errors.
What do you y'all think?
Full Stack Trace:
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/scalar_scanner.rb:104:in `Integer'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/scalar_scanner.rb:104:in `parse_int'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/scalar_scanner.rb:95:in `tokenize'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:279:in `visit_String'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:136:in `accept'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:523:in `block in emit_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:521:in `each'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:521:in `emit_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:506:in `dump_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:134:in `accept'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:359:in `block in visit_Enumerator'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:359:in `each'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:359:in `visit_Enumerator'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:351:in `visit_Array'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:136:in `accept'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:523:in `block in emit_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:521:in `each'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:521:in `emit_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:506:in `dump_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:134:in `accept'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:523:in `block in emit_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:521:in `each'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:521:in `emit_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:506:in `dump_coder'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:134:in `accept'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/visitors/yaml_tree.rb:118:in `push'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych.rb:513:in `dump'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/psych/core_ext.rb:13:in `to_yaml'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/backend/base.rb:68:in `payload_object='
/app/vendor/bundle/ruby/2.7.0/gems/activemodel-5.2.4.3/lib/active_model/attribute_assignment.rb:51:in `public_send'
/app/vendor/bundle/ruby/2.7.0/gems/activemodel-5.2.4.3/lib/active_model/attribute_assignment.rb:51:in `_assign_attribute'
/app/vendor/bundle/ruby/2.7.0/gems/activemodel-5.2.4.3/lib/active_model/attribute_assignment.rb:44:in `block in _assign_attributes'
/app/vendor/bundle/ruby/2.7.0/gems/activemodel-5.2.4.3/lib/active_model/attribute_assignment.rb:43:in `each'
/app/vendor/bundle/ruby/2.7.0/gems/activemodel-5.2.4.3/lib/active_model/attribute_assignment.rb:43:in `_assign_attributes'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/attribute_assignment.rb:23:in `_assign_attributes'
/app/vendor/bundle/ruby/2.7.0/gems/activemodel-5.2.4.3/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
/app/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment'
/app/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/core.rb:315:in `initialize'
/app/vendor/bundle/ruby/2.7.0/gems/oink-0.10.1/lib/oink/instrumentation/active_record.rb:60:in `initialize'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/inheritance.rb:66:in `new'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/inheritance.rb:66:in `new'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/backend/base.rb:16:in `enqueue_job'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/backend/base.rb:12:in `enqueue'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/message_sending.rb:11:in `method_missing'
/app/app/models/asynchronous_action.rb:581:in `block in send_weekly_announcements'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:70:in `each'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:70:in `block in find_each'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:222:in `loop'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:222:in `in_batches'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.3/lib/active_record/relation/batches.rb:69:in `find_each'
/app/app/models/asynchronous_action.rb:579:in `send_weekly_announcements'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/performable_method.rb:26:in `perform'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/backend/base.rb:81:in `block in invoke_job'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:66:in `execute'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/backend/base.rb:78:in `invoke_job'
/app/vendor/bundle/ruby/2.7.0/gems/newrelic_rpm-6.12.0.367/lib/new_relic/agent/instrumentation/delayed_job_instrumentation.rb:163:in `block in invoke_job'
/app/vendor/bundle/ruby/2.7.0/gems/newrelic_rpm-6.12.0.367/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:376:in `perform_action_with_newrelic_trace'
/app/vendor/bundle/ruby/2.7.0/gems/newrelic_rpm-6.12.0.367/lib/new_relic/agent/instrumentation/delayed_job_instrumentation.rb:162:in `invoke_job'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:230:in `block (2 levels) in run'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:230:in `block in run'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:229:in `run'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:312:in `block in reserve_and_run_one_job'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:66:in `execute'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:312:in `reserve_and_run_one_job'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:213:in `block in work_off'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:212:in `times'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:212:in `work_off'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:175:in `block (4 levels) in start'
/app/vendor/ruby-2.7.5/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:174:in `block (3 levels) in start'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:66:in `execute'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:173:in `block (2 levels) in start'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:172:in `loop'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:172:in `block in start'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:79:in `block in add'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:66:in `execute'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/worker.rb:171:in `start'
/app/vendor/bundle/ruby/2.7.0/gems/delayed_job-4.1.8/lib/delayed/tasks.rb:9:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/app/vendor/bundle/ruby/2.7.0/bin/rake:23:in `load'
/app/vendor/bundle/ruby/2.7.0/bin/rake:23:in `<top (required)>'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/cli/exec.rb:63:in `load'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/cli/exec.rb:63:in `kernel_load'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/cli/exec.rb:28:in `run'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/cli.rb:474:in `exec'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/cli.rb:30:in `dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/cli.rb:24:in `start'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/exe/bundle:49:in `block in <top (required)>'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/lib/bundler/friendly_errors.rb:128:in `with_friendly_errors'
/app/vendor/bundle/ruby/2.7.0/gems/bundler-2.2.21/exe/bundle:37:in `<top (required)>'
/app/bin/bundle:3:in `load'
/app/bin/bundle:3:in `<main>' 2021-12-27 11:27:24 -0500
houdl commented
this may solve your problem
paper-trail-gem/paper_trail#1267 (comment)
LucasCioffi commented
Thank you @houdl. I solved the problem with a refactor. I hope that your link solves the problem for other people.