GreenmaskIO/greenmask

feat: Implement transformation conveyor

Opened this issue · 0 comments

We suspect that Greenmask's performance could be improved by implementing a transformation conveyor. The idea is to:

  • Run conveyor workers equal to the number of applied transformations.
  • Receive a record with a count matching the number of transformers.
  • Send the record to a channel.
  • Allow the next record into the channel based on the "head" length. For instance, if we have 3 conveyors and the last one is still processing, ensure that at least 2 records are pushed.

Note: This is a rough algorithm and may be adjusted during development.