Feature Request: Integrate (or allow integration with) ActiveSupport::Notifications

[leehambley]

I'm working on an app where there's a massive amount of random test failures due to environmental issues (large SOA app, shared final integration test environment, and lots and lots of teams deploying their services frequently)

We're trying to gauge the problem by instrumenting retries with ActiveSupport::Notifications, we have our own retrying logic localized to individual cases (e.g async message handlers, eventually refreshing a UI) and also retry handlers within our SitePrism page objects, these are already instrumented. But lacking is the general "retried the whole spec" scope.

Thoughts?

[dwbutler]

It sounds like you would like to instrument each individual retry attempt. Currently there isn't any way to hook into the internals of the retry execution.

There has been some interest in implementing hooks. See #50

Ideally, this would look something like

config.around(:retry) do |example|
  ActiveSupport::Notifications.instrument("run", attempt: example.attempts) do
    example.run
  end
end

Does this look like it would do what you're looking for?

[leehambley]

It does indeed, I wonder if something like this could already work out of the box, around() all my tests, looking into the example.attempts (or similar) properties to make a report.

[dwbutler]

You could do it out of the box if all you care about is the number of attempts. Try this:

config.around(:each) do |example|
  ActiveSupport::Notifications.instrument("rspec.example") do |payload|
    example.run_with_retry
    payload[:attempts] = example.attempts
  end
end