Cleanup and visibility enhancement
sbrossie opened this issue · 5 comments
This task is to track some enhancements we could do on 0.24.x release
Can we simplify the reaper logic to only look at one date
NotificationQ
The current logic looks at both processing_available_date and effective_date. We expect the clause processing_available_date <= now to come prior effective_date <= reapingDate, therefore making the later condition the one that drives the reaper.
PersistentBus
The current logic is similar except the effective_date for the bus is the created_date.
Note that for inflight queue, processing_owner is null so processing_available_date <= now is never even considered.
=> Based on this, it seems we could remove processing_available_date altogether along with the claim time as this is only used for the reaper and we propose to simplify it.
Can we add some information about processing time?
One challenge here is to define what we want to measure:
- Time from when entry was fetched as
readyto when it moved tohistorytable - this will not represent the true processing time, the larger the batch we fetch the larger the time we would get - Real time it took to go through all handlers - harder to measure
Implementing this means we would need a new field in the history table.
Additionally it would be great to add a created_date for the history table to check when this entry was processed (and for instance track whether this was processed late - we would keep the created_date from when entry was inserted.
Can we link REAPED entries - right now, a new entry is inserted with the same event_json, so unless we look for entries with same event_json, we can't really figure out what happened?
Can we provide a mechanism to pause the reaper - e.g as we insert a lot of entries at once such as during a migration, and where default settings don't make sense?
Can we easily export the runtime config for the queue - bus + notification ?