/stream-reactor

Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.

Primary LanguageScalaApache License 2.0Apache-2.0

Actions Status

Join us on slack Alt text

Lenses for Apache Kafka

Lenses offers SQL (for data browsing and Kafka Streams), Kafka Connect connector management, cluster monitoring and more.

You can find more on lenses.io

Stream Reactor

Alt text FOSSA Status

A collection of components to build a real time ingestion pipeline.

Kafka Compatibility

  • Kafka 2.5+ (Confluent 5.5) - Stream reactor 2.0.0+
  • Kafka 2.0 -> 2.4 (Confluent 5.4) - Stream reactor 1.2.7

DEPRECATION NOTICE

The following connectors have been deprecated and are no longer included in any release from 3.0.

  • Blockchain
  • Bloomberg
  • Rethink
  • VoltDB

Connectors

Please take a moment and read the documentation and make sure the software prerequisites are met!!

Connector Type Description Docs
AWS S3 Sink Copy data from Kafka to AWS S3. Docs
AzureDocumentDb Sink Copy data from Kafka and Azure Document Db. Docs
Cassandra Source Copy data from Cassandra to Kafka. Docs
*Cassandra Sink Certified DSE Cassandra, copy data from Kafka to Cassandra. Docs
Coap Source Copy data from IoT Coap endpoints (using Californium) to Kafka. Docs
Coap Sink Copy data from Kafka to IoT Coap endpoints. Docs
Elastic 6 Sink Copy data from Kafka to Elastic Search 6.x w. tcp or http Docs
FTP/HTTP Source Copy data from FTP/HTTP to Kafka. Docs
Hazelcast Sink Copy data from Kafka to Hazelcast. Docs
HBase Sink Copy data from Kafka to HBase. Docs
Hive Source Copy data from Hive/HDFS to Kafka. Docs
Hive Sink Copy data from Kafka to Hive/HDFS Docs
InfluxDb Sink Copy data from Kafka to InfluxDb. Docs
Kudu Sink Copy data from Kafka to Kudu. Docs
JMS Source Copy data from JMS topics/queues to Kafka. Docs
JMS Sink Copy data from Kafka to JMS. Docs
MongoDB Sink Copy data from Kafka to MongoDB. Docs
MQTT Source Copy data from MQTT to Kafka. Docs
MQTT Sink Copy data from Kafka to MQTT. Docs
Pulsar Source Copy data from Pulsar to Kafka. Docs
Pulsar Sink Copy data from Kafka to Pulsar. Docs
Redis Sink Copy data from Kafka to Redis. Docs

Release Notes

Please see the Stream Reactor Release Notes at Lenses Documentation.

Building

To build:

sbt clean compile

To test:

sbt test

To create assemblies:

sbt assembly

To build a particular project:

sbt "project cassandra-kafka-2-8" compile

To test a particular project:

sbt "project cassandra-kafka-2-8" test

To create a jar of a particular project:

sbt "project cassandra-kafka-2-8" assembly

Running E2E tests

If not already built, you must first build the connector archives:

sbt "project cassandra-kafka-2-8" assembly
sbt "project elastic6-kafka-2-8" assembly 
sbt "project mongodb-kafka-2-8" assembly
sbt "project redis-kafka-2-8" assembly

sbt "project cassandra-kafka-3-1" assembly 
sbt "project elastic6-kafka-3-1" assembly 
sbt "project mongodb-kafka-3-1" assembly 
sbt "project redis-kafka-3-1" assembly

To run the tests:

sbt e2e:test

Contributing

We'd love to accept your contributions! Please use GitHub pull requests: fork the repo, develop and test your code, semantically commit and submit a pull request. Thanks!

License

FOSSA Status