faust-streaming/faust

Migrate from python-rocksdb to rocksdict

wbarnha opened this issue · 0 comments

We should give users the option to install rockdict as an alternative to faust-streaming-rocksdb for a few reasons:

  1. I have profiled the performance of rocksdict personally and have noticed a 20x boost in performance. There's also an abundance of information on https://github.com/Congyuwang/RocksDict demonstrating its performant abilities
  2. rocksdict is actively maintained, I have not been able to actively support faust-streaming-rocksdb as much
  3. rocksdict wheels are only 4 MB, whereas faust-streaming-rocksdb wheels are 66 MB due to its intensive CI/CD build. This is undesirable because it can increase dependency installation time and Docker image size.

I think adding faust-streaming[rocksdict] may be the right way to approach this for now until we know beyond a reasonable doubt that we can phase out faust-streaming-rocksdb.