/streamx

Make stream processing easier! Flink & Spark development scaffold, The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Primary LanguageJavaApache License 2.0Apache-2.0


Make stream processing easier!!!

Official Website | Change Log | Document

English | 中文

StreamX

Make stream processing easier

A magical framework that make stream processing easier!

🚀 Introduction

The original intention of StreamX is to make stream processing easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

StreamX video

🎉 Features

  • Scaffolding
  • Out-of-the-box connectors
  • Support maven compilation
  • Configuration
  • Multi version flink support(1.12.x,1.13.x,1.14.x, 1.15.x)
  • Scala 2.11 / 2.12 support
  • restapi support.
  • All Flink deployment mode support(Remote/K8s-Native-Application/K8s-Native-Session/YARN-Application/YARN-Per-Job/YARN-Session)
  • start, stop, savepoint, resume from savepoint
  • Various companies and organizations use StreamX for production and commercial products.
  • Flame graph
  • Notebook
  • Project configuration and dependency version management
  • Task backup and rollback
  • Manage dependencies
  • UDF
  • Flink SQL Connector
  • Flink SQL WebIDE
  • Catalog、Hive
  • Full support from task development to deployment
  • ...

🏳‍🌈 Components

Streamx consists of three parts,streamx-core,streamx-pump and streamx-console

1️⃣ streamx-core

streamx-core is a framework that focuses on coding, standardizes configuration, and develops in a way that is better than configuration by convention. Also it provides a development-time RunTime Content and a series of Connector out of the box. At the same time, it extends DataStream some methods, and integrates DataStream and Flink sql api to simplify tedious operations, focus on the business itself, and improve development efficiency and development experience.

2️⃣ streamx-pump

streamx-pump is a planned data extraction component, similar to flinkx. Based on the various connector provided in streamx-core, the purpose is to create a convenient, fast, out-of-the-box real-time data extraction and migration component for big data, and it will be integrated into the streamx-console.

3️⃣ streamx-console

streamx-console is a stream processing and Low Code platform, capable of managing Flink tasks, integrating project compilation, deploy, configuration, startup, savepoint, flame graph, Flink SQL, monitoring and many other features. Simplify the daily operation and maintenance of the Flink task.

Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Thanks to the above excellent open source projects and many outstanding open source projects that are not mentioned, for giving the greatest respect,Thanks to Apache Flink for creating a great project! Thanks to the Apache Zeppelin project for the early inspiration.

🚀 Quick Start

click Document for more information

💋 our users

Various companies and organizations use StreamX for research, production and commercial products. Are you using this project ? you can add your company

image

🏆 Our honor

We have received some precious honors, which belong to everyone who contributes to StreamX, Thank you !

🤝 Contribution

PRs Welcome

You can submit any ideas as pull requests or as GitHub issues.

If you're new to posting issues, we ask that you read How To Ask Questions The Smart Way (This guide does not provide actual support services for this project!), How to Report Bugs Effectively prior to posting. Well written bug reports help us help you!

Thank you to all the people who already contributed to StreamX!

⏰ Contributor Over Time

Contributor Over Time

💰 Donation

Are you enjoying this project ? 👋

If you like this framework, and appreciate the work done for it to exist, you can still support the developers by donating ☀️ 👊

WeChat Pay Alipay
Buy Me A Coffee Buy Me A Coffee

🏆 Our sponsors (Coffee Suppliers)

💜 Monthly Supplier

Welcome individuals and enterprises to sponsor, your support will help us better develop the project

🥇 Gold Supplier

🥈 Platinum Supplier

🥈 Silver Supplier

Thanks to JetBrains for supporting us free open source licenses.

JetBrains

🏅 Backers

Thank you to all our backers!


💬 Join us

StreamX enters the high-speed development stage, we need your contribution.

Stargazers over time

Join the Group