This repository contains examples of use cases that utilize Decodable streaming solution as well as demos for related open-source projects such as Apache Flink, Debezium, and Postgres.
Example | Description |
---|---|
Flink Learn | Apache Flink tutorials and webinar |
AsyncAPI | Publishing Data Products with AsyncAPI |
Opinionated Data Pipelines | Building data pipelines with schema on write streams. |
Postman | Building data pipelines with Postman. |
Change Streams | Using change streams to build materialized views in Postgres |
XML Processing | Parse XML and transform to JSON |
OSQuery Routing | Route OSQuery logs with SQL |
Masking | Ways to mask data |
Apache Pinot | Transforming osquery logs to Apache Pinot and Superset |
Apache Druid | This example sends covid 19 data to Decodable using it's REST API. The data is then cleansed using Decodable SQL and send the data to a Kafka sink. |
Rockset | We will be utilizing a cloud MQTT broker and AWS Kinesis to capture and stream data. Decodable will be responsible for preparing and aggregating the data prior to reaching the real-time analytical database (Rockset) |
Tinybird | We write data to Tinybird and build a simple real time web application. |
Apache Kafka | Installing Apache Kafka on EC2 and writing to S3 with Decodable |
Apache Kafka mTLS | We install Apache Kafka on EC2 and configure it with mTLS and configure Decodable to read from it |
Snowflake + Snowpipe | We setup a snowpipe at the end of a Decodable S3 sink. |
Confluent | Clickstream from Confluent Cloud joined with CDC user data from Postgres |
Snowflake + Snowpipe + Merge | Leveraging Snowpipe, we send CDC data from Postgres to be processed in Snowflake using an Append Only Stream in Snowflake to merge CDC data in a Snowflake table. Essentially mirroring the table in Postgres in a Snowflake table. |
Reading S3 Events in a Lambda Function | We configure an S3 bucket with a Lambda notification to send data to Kinesis to be processed in Decodable |
MSSQL CDC | We enable a MSSQL in Docker with CDC. We then stand up a Debezium server to read from MSSQL and write the change events into AWS Kinesis |
Oracle CDC | We configure a Oracle AWS RDS with LogMiner. We then stand up a Debezium server to read change events into AWS Kinesis |
DynamoDb CDC | We configure a DynamoDB to send change data to Kinesis. Then we read those changes into Decodable for transformation or replication. |
Logical Decoding Message Examples | We show how to retrieve logical decoding messages from the Postgres WAL |
GitHub Webhooks | We show how to process GitHub Webhook events using the Decodable REST source connector |
PyFlink | We run a basic PyFlink job on Kubernetes |
Kafka / Flink / Iceberg | Integrating Apache Kafka with Apache Iceberg through Apache Flink. As presented at Kafka Summit London 2024 |
Kafka / Flink / Iceberg (with Decodable) | Streaming from Apache Kafka to Apache Iceberg with Decodable |
Flink SQL Troubleshooting | A set of Docker Compose environments for demonstrating various Flink SQL troubleshooting scenarios (see related blog) |
Array Aggregation | Using the array_agg() UDF for denormalizing data in a pipeline from MySQL to OpenSearch |
Kafka with ngrok | Docker Compose for running Apache Kafka locally, accessible from the internet using ngrok |
PyFlink on Decodable | Running a PyFlink job as a Custom Pipeline on Decodable |
Delta Lake / Flink | Writing to Delta Lake with Apache Flink |
This code base is available under the Apache License, version 2.