data-pipelines
There are 296 repositories under data-pipelines topic.
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
apache/airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dagster-io/dagster
An orchestration platform for the development, production, and observation of data assets.
apache/dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Unstructured-IO/unstructured
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
mage-ai/mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
infinyon/fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
StructuredLabs/preswald
Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.
orchest/orchest
Build data pipelines, the easy way 🛠️
Netflix/maestro
Maestro: Netflix’s Workflow Orchestrator
ucbepic/docetl
A system for agentic LLM-powered data processing and ETL
meltano/meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
elementary-data/elementary
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
data-engineering-community/data-engineering-wiki
The best place to learn data engineering. Built and maintained by the data engineering community.
feldera/feldera
The Feldera Incremental Computation Engine
combust/mleap
MLeap: Deploy ML Pipelines to Production
pyper-dev/pyper
Concurrent Python made simple
opendatadiscovery/odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
fmind/mlops-python-package
Kickstart your MLOps initiative with a flexible, robust, and productive Python package.
OpenDCAI/DataFlow
Easy Data Preparation with latest LLMs-based Operators and Pipelines.
yobix-ai/extractous
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
amphi-ai/amphi-etl
Visual Data Preparation and Transformation. Low-Code Python-based ETL.
bruin-data/bruin
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
dataform-co/dataform
Dataform is a framework for managing SQL based data operations in BigQuery
raystack/optimus
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
artie-labs/transfer
Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time.
elementary-data/dbt-data-reliability
dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
vmware/versatile-data-kit
One framework to develop, deploy and operate data workflows with Python and SQL.
gabledata/recap
Work with your web service, database, and streaming schemas in a single format.
dataflint/spark
Drop-in replacement for Apache Spark UI
tuva-health/tuva
Main repo including core data model, data marts, data quality tests, and terminology sets.
dataplane-app/dataplane
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
terrytangyuan/awesome-kubeflow
A curated list of awesome projects and resources related to Kubeflow (a CNCF incubating project)
kevin-hanselman/dud
A lightweight CLI tool for versioning data alongside source code and building data pipelines.
datajoint/datajoint-python
Relational data pipelines for the science lab
koolreport/core
An Open Source PHP Reporting Framework that helps you to write perfect data reports or to construct awesome dashboards in PHP. Working great with all PHP versions from 5.6 to latest 8.0. Fully compatible with all kinds of MVC frameworks like Laravel, CodeIgniter, Symfony.