Redis Streams Spout for Apache Storm

This project is an Apache Storm Spout for consuming from Redis Streams.

Features

Ability to consume from Redis Streams while maintaining state.
Ability to consume from a single Redis server or a RedisCluster.
Parallelism supported via unique Consumer Ids.

Usage & Configuration

Include the dependency in your POM.xml file:

<dependency>
    <groupId>org.sourcelab.storm.spout</groupId>
    <artifactId>redis-stream-spout</artifactId>
    <version>1.1.0</version>
</dependency>

Configuration

The spout is configured using the RedisStreamSpoutConfig class.

Common Configuration Properties

Property	Required	Description
`Group Name`	Required	The Consumer group name the Spout should use.
`Consumer Id Prefix`	Required	A prefix to use for generating unique Consumer Ids within the Consumer Group. To support multiple parallel consumers, the Spout instance will be appended to the end of this value.
`Stream Key`	Required	The Redis key to consume messages from.
`Tuple Converter`	Required	Defines how messages are transformed between being consumed from Redis, and being emitted into the topology
`Failure Handler`	Required	Defines how the spout handles failed tuples. See note below.

Example Configuration

    // Create config
    final RedisStreamSpoutConfig.Builder config = RedisStreamSpoutConfig.newBuilder()
        // If you want to connect to a single Redis instance:
        .withServer("localhost", 6759)

        // OR if you want to talk to a RedisCluster:
        .withClusterNode("node1.hostname.com", 6759)
        .withClusterNode("node2.hostname.com", 6759)
        ...
        
        // Consumer Properties
        .withGroupName("StormConsumerGroup")
        .withConsumerIdPrefix("StormConsumer")
        .withStreamKey("RedisStreamKeyName")

        // Tuple Converter instance (see note below)
        .withTupleConverter(..Your TupleConvertor implementation...)

        // Failure Handler instance (see note below)
        .withFailureHandler(new ExponentialBackoffFailureHandler(...));
        

    // Create Spout
    final ISpout redisStreamSpout = new RedisStreamSpout(config);

TupleConverter Implementation

In order to convert from the values consumed from RedisStream into Tuple values that can be emitted into the Storm Topology, an implementation of the TupleConverter must be defined and passed to the configuration.

TestTupleConverter is provided as an example implementation.

FailureHandler Implementations

The FailureHandler interface defines how the Spout will handle Failed Tuples. The following implementations are provided out of the box:

Implementation	Description
NoRetryHandler	Will never retry failed tuples.
ExponentialBackoffFailureHandler	Will attempt to retry failed messages using an exponential backoff strategy.
RetryFailedTuples	Rudimentary implementation that can be configured to replay failed tuples for a configured number of attempts.

Example Topology

ExampleLocalTopology is provided as a working example running on a Local Storm Topology.

Contributing

Releasing

Steps for performing a release:

Update release version: mvn versions:set -DnewVersion=X.Y.Z
Validate and then commit version: mvn versions:commit
Update CHANGELOG and README files.
Merge to master.
Deploy to Maven Central: mvn clean deploy -P release-redis-spout
Create release on Github project.

Changelog

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

View Changelog

SourceLabOrg/RedisStreams-StormSpout