Erroneous WARNING message after every schedule firing

[oliverlockwood]

See #2 (comment).

The log message "Couldn't cancel" appears every time a schedule is triggered, because we delete from the scheduler topic after.

What happens is we send the null to the scheduler topic, then we process that as a Cancel (to support explicit deletes, and to remove the schedule from the actors state) but the call to cancel fails since the event triggering has already fired

We still want some kind of warning if we fail to cancel something when it should have actually been cancellable, but we don't want false positive warnings in the mainline.

[oliverlockwood]

This effectively this is a state machine...

• SCHEDULED
• FIRED
• CANCELLED

and we need to differentiate between transitions direct from SCHEDULED to CANCELLED and those from FIRED to CANCELLED.

(At the moment, we just have two states: SCHEDULED and CANCELLED, denoted by presence or absence in the Akka scheduler. )

[paoloambrosio-skyuk]

The state machine would get reset at every restart, not being able to tell if the delete was from an explicit delete or the scheduler firing.

[oliverlockwood]

I guess that's a window condition where something has been deleted or fired, but Kafka compaction's not yet happened?

We probably want to configure (and document the need to configure!) a short compaction delay and a short delete retention period to minimise the impact of that?

[paoloambrosio-skyuk]

We can probably come up with a strategy that minimizes it.

I disagree that this is a bug though. It was designed that way knowing that it would happen... MVP