A scalable, distributed Kafka message scheduler with only a Kafka cluster as a dependency.
Some applications or microservices run without any database or external dependency only sending and receiving messages through Kafka.
If they have to schedule some tasks for example
- Send out a reminder email every day if a user has not verified their email
- Generate an export every 30 minutes
- ...
the application has to make sure that the task is executed exactly once. Even if the application moves to another node on a Kubernetes cluster.
The usual way to handle persistent jobs would be to use Quartz, Hangfire or Jobrunr. But all of them depend on a database.
Why should I have to introduce a database into my stack if I can "just" stay with Kafka?
Kafka-scheduler lets you schedule a Kafka message with a desired key, value and headers and when you want to receive them on which topic. This way your application receives the message as a trigger and can execute the business logic on the desired schedule.
A fire-and-forget command will be triggered at a certain point in the future.
sequenceDiagram
autonumber
note right of Application: Topic: one-time-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-when:2011-12-03T10:15:30+01:00
Application->>Kafka-Scheduler: key:value
Kafka-Scheduler->>Kafka-Scheduler: Schedule
note right of Application: Topic: mytopic
Kafka-Scheduler->>Application: mykey:value
| Header | Description |
|---|---|
| kafka-scheduler-key | The kafka message key used as a unique id so it can be cancelled |
| kafka-scheduler-value | The Kafka message value you want to receive back |
| kafka-scheduler-when | A ISO-6801 configured Timestamp |
| Field | Description |
|---|---|
| Key | The kafka message key used as a unique id so it can be cancelled |
| Value | The Kafka message value you want to receive back |
A command which will be triggered on a CRON schedule
sequenceDiagram
autonumber
note right of Application: Topic: recurring-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-cron: */5 * * * * *
Application->>Kafka-Scheduler: key:value
Kafka-Scheduler->>Kafka-Scheduler: Schedule
note right of Application: Topic: mytopic
Kafka-Scheduler->>Application: mykey:value
note right of Application: Topic: mytopic
Kafka-Scheduler->>Application: mykey:value
note right of Application: Topic: mytopic
Kafka-Scheduler->>Application: mykey:value
| Header | Description |
|---|---|
| kafka-scheduler-key | The kafka message key used as a unique id so it can be cancelled |
| kafka-scheduler-value | The Kafka message value you want to receive back |
| kafka-scheduler-cron | A cron expression with extended syntax support] |
| Field | Description |
|---|---|
| Key | The kafka message key used as a unique id so it can be cancelled |
| Value | The Kafka message value you want to receive back |
A command which a defined duration between runs
sequenceDiagram
autonumber
note right of Application: Topic: fixed-rate-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-period: PT5S
Application->>Kafka-Scheduler: key:value
Kafka-Scheduler->>Kafka-Scheduler: Schedule
note right of Application: Topic: mytopic
Kafka-Scheduler->>Application: mykey:value
note right of Application: Topic: mytopic
Kafka-Scheduler->>Application: mykey:value
note right of Application: Topic: mytopic
Kafka-Scheduler->>Application: mykey:value
| Header | Description |
|---|---|
| kafka-scheduler-key | The kafka message key used as a unique id so it can be cancelled |
| kafka-scheduler-value | The Kafka message value you want to receive back |
| kafka-scheduler-period | A ISO-8601 Duration |
| Field | Description |
|---|---|
| Key | The kafka message key used as a unique id so it can be cancelled |
| Value | The Kafka message value you want to receive back |
All commands can be cancelled by sending a Kafka tombstone message with the same key which was used to schedule the command.
sequenceDiagram
autonumber
note right of Application: Topic: one-time-commands<br>Headers<br>kafka-scheduler-key: myKey<br>kafka-scheduler-topic: mytopic<br>kafka-scheduler-when:2011-12-03T10:15:30+01:00
Application->>Kafka-Scheduler: key:value
Kafka-Scheduler->>Kafka-Scheduler: Schedule
note right of Application: Topic: one-time-commands<br>Headers
Application->>Kafka-Scheduler: key:null
Kafka-Scheduler->>Kafka-Scheduler: Cancel scheduled command
TBD
TBD
kafka-message-scheduler - Awesome implementation by @etf in Go. Only supports OneTimeCommands but has memory optimizations and an Admin UI.
high-available-task-scheduling - Implementation by @cbenaveen using Kafka Streams.
@rdehuyss for creating JobRunr the easiest to use scheduling library in the JVM space.