Task start time conditions
Opened this issue · 9 comments
Tasks should be able to have a specified time that they are not started until. Tasks will not be eligible to claim until the time has passed.
@aphyr: what format would you suggest specifying time in and what call should we use to get the current time?
Uh, this opens up a whole nest of issues, including that people will think the option actually means something. I'd avoid implementing this if at all possible.
I've ran into all sorts of reasons why I've wanted to put off running a task for some period of time.
What about something that says, "This can't be claimed until X seconds has passed"?
It would allow for doing backoff, but wouldn't be tied to a client clock.
Skuld's design assumptions allow for clock skew on the order of minutes to hours. Second-scale timeouts are almost definitely not feasible.
The problem I have that I was hoping to solve with this is cases where I look at a task and see that some other resource that this task depends on isn't available right now, so I don't need to try to run this task again for another X minutes.
As it is with a simple message queue (like what I use today), my only option is to throw it back into the end of the queue, which means that it will continue to see the same tasks over and over again while they're requeued until such a time that the other resource is ready again.
You can use linear-time as a realllly loose proxy for "global time", but it can and will drift by huge volumes, and has no relation to anything outside Skuld. Heck, it may not even be close to other nodes.
The correct solution is task dependencies. There's a writeup of this stuff in the Factual wiki; not sure if I ported it to the github repo.
Oh, or, if your consumer is the one that discovers the dependent service is down, just have the consumer stop processing records for a bit.