Add a "key" field to messages
Closed this issue · 2 comments
arnecls commented
Currently we have a lot of cases were messages are separated into key and value (e.g. after reading from kafka) or where a key has to be generated from a payload (routing, redis, kafka, etc.).
Along with that comes the need to modify keys separately that lead to different approaches and config + code bloat.
By adding keys to messages we will reduce processing overhead and can built much more streamlined pipelines that are easier to understand and debug.
See/discuss changes in comments
- Add a "key" to the Message struct ([]byte)
- Consumers reading from key/value stores should set the key
- Producers requiring a key should take the key field into account (can be empty)
- Modulators for keys must be possible (ideally by defining an ApplyTo field or sth. similar)
- Implement a "Copy" formatter that allows copying data between key and value fields
arnecls commented
After an internal discussion we agreed on changing this from a dedicated field to a metadata collection. Changed requirements:
- Add a "metadata" field to the message struct (map[string][]byte)
- Consumers can decided where to put the key by config value (default "key")
- Producers can read a required key from metadata by config value
- Formatters may store snippets in metadata fields
- Formatters may read/write metadata fields by config value (default: read/write payload)
- Filters may read from metadata fields
- Routers may react on metadata fields
msiebeneicher commented
This task is done for now. Open features will handles by new subtasks.