Infinity restart when errors happens inside of hooks

[Roriz]

Problem

Any raise on after_fork hook will turn down the worker and start a new one. If the error persists, this process continues infinitely.

How we discovery

Here we work, this infinity error throws tons of logs, and we discover after the log platform reach the monthly limit.

What is the root cause?

We have some code on after_fork to connect to the database and manage the database pool. With the upgrade of rails 6 to 7, some part of this code broke and was unseen on staging tests.

Workaround

For now, to make sure that we will stop the sneakers:run and make sure the container is in an unhealthy state, we add:

Sneakers.configure(
  ...
  hooks: {
    after_fork: -> {
      begin
        {code_here}
      rescue => e
        Process.kill('TERM', Process.getpgid(Process.pid))
      end 
    }
  } 
)

Before workaround (gif stoped, but will raise a log infinity)

After workaround

Minimal setup for reproduce

config/initializers/sneakers.rb

Sneakers.configure(
  connection: Bunny.new(Rails.application.secrets[:RABBITMQ_URL]),
  hooks: {
    after_fork: -> {
      raise 'some-error'
    }
  }
)

Opinion

In our opinion, any error inside of hooks should stop all sneakers process and not try again. Restart and try again should be responsible for the infrastructure, if necessary.