MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice; however, it relies on a pluggable replicated storage layer i.e. Object Store / DFS / NFS for storing data. The net result is a PubSub system that:
- Handles GB/s traffic
- Independently scales, writes, and reads
- Doesn’t require expensive rebalancing to handle traffic growth
- Is 90% more cost effective than our Kafka footprint
Checkout our Getting Started Guide
MemQ can be a valuable PubSub choice for the following use case families:
- Large scale data ingestion
- Bulk uploads
- Near Real Time Analytics
Detailed architecture docs can be found here
Please checkout the original MemQ blog for details and background.
See contributors
MemQ distributed under Apache License, Version 2.0