trivago/gollum

Add a "key" field to messages

Closed this issue · 2 comments

Currently we have a lot of cases were messages are separated into key and value (e.g. after reading from kafka) or where a key has to be generated from a payload (routing, redis, kafka, etc.).
Along with that comes the need to modify keys separately that lead to different approaches and config + code bloat.
By adding keys to messages we will reduce processing overhead and can built much more streamlined pipelines that are easier to understand and debug.

See/discuss changes in comments

  • Add a "key" to the Message struct ([]byte)
  • Consumers reading from key/value stores should set the key
  • Producers requiring a key should take the key field into account (can be empty)
  • Modulators for keys must be possible (ideally by defining an ApplyTo field or sth. similar)
  • Implement a "Copy" formatter that allows copying data between key and value fields

After an internal discussion we agreed on changing this from a dedicated field to a metadata collection. Changed requirements:

  • Add a "metadata" field to the message struct (map[string][]byte)
  • Consumers can decided where to put the key by config value (default "key")
  • Producers can read a required key from metadata by config value
  • Formatters may store snippets in metadata fields
  • Formatters may read/write metadata fields by config value (default: read/write payload)
  • Filters may read from metadata fields
  • Routers may react on metadata fields

This task is done for now. Open features will handles by new subtasks.