/QueueNado

Implementation of fast queue patterns (Push/Pull, request/reply, pub/sub, dealer/router) for C++

Primary LanguageC++MIT LicenseMIT

QueueNado

QueueNado is a collection of C++ implemntations of ZMQ fast queue patterns (Push/Pull, request/reply, pub/sub, dealer/router) for C++

Requirements

  1. g3log
  2. g3sinks
  3. FileIO
  4. ZMQ v3.2.5
  5. CZMQ v3.0.2
  6. ZLIB Most Linux distributions carry zlib. Example: In Ubuntu it can be installed with sudo apt-get install zlib1g-dev
  7. StopWatch
  8. DeathKnell
  9. boost For installing boost please follow their instructions.

Build and Install

Example of installation

cd Queuenado
cd 3rdparty
unzip gtest-1.7.0.zip
cd ..
mkdir build
cd build
cmake ..
make -j

Executing the unit tests

sudo ./UnitTestRunneer

Installing

sudo make install

Alternative on Debian systems

make package
sudo dpkg -i Queuenado-<package_version>Linux.deb

Rifle - Vampire

Rifle - Vampire implements the push / pull messaging pattern in zmq. The communication setup supports the following client/server communications

Client -to- Server

  • one-to-one
  • one-to-many
  • many-to-one

In a many scenario, it is the "one" side that does the bind ([Vampire.h] [Rifle.h]) on the queue.

Rifle - Vampire Limitation

It cannot do many-to-many

Use Cases for Rifle - Vampire

  • Reliable messaging without responses
  • High performance (500k msgs per second or higher) with zero_copy
  • Process to process communication

API

[Vampire.h] [Rifle.h]

Test usage

[RifleVampireTests.cpp]

Shotgun - Alien

Shotgun - Alien implements the pub / sub messaging pattern in zmq.

Use Cases for Shotgun - Alien

  • One to many: one sender communicating with many listeners.
  • Not high performance around 10k msgs a sec. This can be improved by batching many messages together.
  • Process to process communication
  • All listeners receive every message sent.

Known limitations and issues

  • Slow joiner issues don't matter or can be worked around

API

[Shotgun.h] [Alien.h]

Test usage

[ShotgunAlienTests.cpp]

Headcrab - Crowbar

Headcrab - Crowbar implements request / reply messaging pattern in zmq.

Use Cases for Headcrab - Crowbar

  • Need to receive acknowledgement from the receiver for each message sent to it
  • Know that their will always be a response (otherwise the socket will be broken)

Known limitations and issues

  • Lower performance (60k msgs a sec)

[Headcrab.h] [Crowbar.h]

Test usage

[CrowbarHeadcrabTests.cpp]

Harpoon - Kraken

Harpoon - Kraken implements a streaming version of pub / sub. It enables data streaming from a publisher to a subscriber.

Kraken Purpose:

The publisher sending streaming data.

Usage example calls from the API: [kraken.h]

  • SendTidalWave() : Send a data chunk to the subscriber ([harpoon]). The call blocks until there is space available in the queue. Returns TIMEOUT, INTERRUPT, CONTINUE status to indicate the status of the underlying queue.

  • FinalBreach() : Call to subscriber ([harpoon]) to indicate the end of a stream.

Harpoon: Subscriber that receives the data

Usage example calls from the API:

  • Aim() : Set location of the queue (tcp)
  • Heave() : Request data and wait for the data to be returned. Returns TIMEOUT, INTERRUPT, VICTORIOUS, CONTINUE to indicate status of the stream. VICTORIOUS means that the stream has completed.

API

[[Kraken.h]] (https://github.com/LogRhythm/QueueNado/blob/master/src/Kraken.h) [KrakenBattle.h] [Harpoon.h] [HarpoonBattle.h]

Test usage

Notifier - Listener

Notifier - Listener extends the Shotgun - Alien implementation of the pub / sub messaging pattern in zmq.

The Notifier - Listener classes are wrappers around the Shotgun - Alienqueueing framework. A Notifier is used to place a message onto a queue that is read from by multiple Listeners. It uses handshake communications, where each Listener must respond back to the Notifier that it received the message. A Notification is deemed successful only if every expected Listener responds to the Notifier.

Use cases for Notifier/Listener

  • One-to-many with handshake feedback
  • Alerting multiple processes of an event, or a call to action

Limitations for Notifier - Listener

The same as for Shotgun - Alien

Boomstick - Skeleton

The Boomstick - Skeleton is used for connecting to ElasticSearch over a wrapper. At the moment part of the pattern implementation is not open sourced and still proprietary. Until further notice it is not recommended to use the BoomStick - Skeleton classes.