StreamX

Make stream processing easier

A magical framework that make stream processing easier!

🚀 Introduction

The original intention of StreamX is to make stream processing easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

🎉 Features

Scaffolding
Out-of-the-box connectors
Support maven compilation
Configuration
Multi version flink support(1.12.x,1.13.x,1.14.x, 1.15.x)
Scala 2.11 / 2.12 support
restapi support.
All Flink deployment mode support(Remote/K8s-Native-Application/K8s-Native-Session/YARN-Application/YARN-Per-Job/YARN-Session)
start, stop, savepoint, resume from savepoint
Various companies and organizations use StreamX for production and commercial products.
Flame graph
Notebook
Project configuration and dependency version management
Task backup and rollback
Manage dependencies
UDF
Flink SQL Connector
Flink SQL WebIDE
Catalog、Hive
Full support from task development to deployment
...

🏳‍🌈 Components

Streamx consists of three parts,streamx-core,streamx-pump and streamx-console

1️⃣ streamx-core

streamx-core is a framework that focuses on coding, standardizes configuration, and develops in a way that is better than configuration by convention. Also it provides a development-time RunTime Content and a series of Connector out of the box. At the same time, it extends DataStream some methods, and integrates DataStream and Flink sql api to simplify tedious operations, focus on the business itself, and improve development efficiency and development experience.

2️⃣ streamx-pump

streamx-pump is a planned data extraction component, similar to flinkx. Based on the various connector provided in streamx-core, the purpose is to create a convenient, fast, out-of-the-box real-time data extraction and migration component for big data, and it will be integrated into the streamx-console.

3️⃣ streamx-console

streamx-console is a stream processing and Low Code platform, capable of managing Flink tasks, integrating project compilation, deploy, configuration, startup, savepoint, flame graph, Flink SQL, monitoring and many other features. Simplify the daily operation and maintenance of the Flink task.

Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Thanks to the above excellent open source projects and many outstanding open source projects that are not mentioned, for giving the greatest respect,Thanks to Apache Flink for creating a great project! Thanks to the Apache Zeppelin project for the early inspiration.

🚀 Quick Start

click Document for more information

💋 out users

Various companies and organizations use StreamX for research, production and commercial products. Are you using this project ? you can add your company

🏆 Our honor

We have received some precious honors, which belong to everyone who contributes to StreamX, Thank you !

🤝 Contribution

You can submit any ideas as pull requests or as GitHub issues.

If you're new to posting issues, we ask that you read How To Ask Questions The Smart Way (This guide does not provide actual support services for this project!), How to Report Bugs Effectively prior to posting. Well written bug reports help us help you!

Thank you to all the people who already contributed to StreamX!