feat: Implement transformation conveyor
Opened this issue · 0 comments
wwoytenko commented
We suspect that Greenmask's performance could be improved by implementing a transformation conveyor. The idea is to:
- Run conveyor workers equal to the number of applied transformations.
- Receive a record with a count matching the number of transformers.
- Send the record to a channel.
- Allow the next record into the channel based on the "head" length. For instance, if we have 3 conveyors and the last one is still processing, ensure that at least 2 records are pushed.
Note: This is a rough algorithm and may be adjusted during development.