Infinity restart when errors happens inside of hooks
Roriz opened this issue · 0 comments
Problem
Any raise on after_fork
hook will turn down the worker and start a new one. If the error persists, this process continues infinitely.
How we discovery
Here we work, this infinity error throws tons of logs, and we discover after the log platform reach the monthly limit.
What is the root cause?
We have some code on after_fork
to connect to the database and manage the database pool. With the upgrade of rails 6 to 7, some part of this code broke and was unseen on staging tests.
Workaround
For now, to make sure that we will stop the sneakers:run
and make sure the container is in an unhealthy state, we add:
Sneakers.configure(
...
hooks: {
after_fork: -> {
begin
{code_here}
rescue => e
Process.kill('TERM', Process.getpgid(Process.pid))
end
}
}
)
Before workaround (gif stoped, but will raise a log infinity)
After workaround
Minimal setup for reproduce
config/initializers/sneakers.rb
Sneakers.configure(
connection: Bunny.new(Rails.application.secrets[:RABBITMQ_URL]),
hooks: {
after_fork: -> {
raise 'some-error'
}
}
)
Opinion
In our opinion, any error inside of hooks should stop all sneakers process and not try again. Restart and try again should be responsible for the infrastructure, if necessary.