Remove Redis as a dependency
marsupialtail opened this issue · 1 comments
marsupialtail commented
The current data plane uses Ray for same machine comms and Redis for inter-machine comms.
Both should be replaced by a custom Arrow Flight server, with two benefits:
- No more disk flushing and reading on demand on same machine
- No more pickle.loads or pickle.dumps
- Possibility for more custom behavior
- Better control from Python side
marsupialtail commented
moved data movement to flight server in PR #6