/memq

MemQ is a new PubSub system that augments Kafka

Primary LanguageJavaApache License 2.0Apache-2.0

MemQ: An efficient, scalable cloud native PubSub system

MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice; however, it relies on a pluggable replicated storage layer i.e. Object Store / DFS / NFS for storing data. The net result is a PubSub system that:

  • Handles GB/s traffic
  • Independently scales, writes, and reads
  • Doesn’t require expensive rebalancing to handle traffic growth
  • Is 90% more cost effective than our Kafka footprint

Design

Getting Started

Checkout our Getting Started Guide

Usage

MemQ can be a valuable PubSub choice for the following use case families:

  • Large scale data ingestion
  • Bulk uploads
  • Near Real Time Analytics

Design

Detailed architecture docs can be found here

Blog

Please checkout the original MemQ blog for details and background.

Authors

Ambud Sharma

Ping Min-Lin

See contributors

License

MemQ distributed under Apache License, Version 2.0