etl-framework
There are 179 repositories under etl-framework topic.
elastic/logstash
Logstash - transport and process your logs, events, or other data
cloudquery/cloudquery
The open source high performance ELT framework powered by Apache Arrow
noflo/noflo
Flow-based programming for JavaScript
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
DAGWorks-Inc/hamilton
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
san089/goodreads_etl_pipeline
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
singer-io/getting-started
This repository is a getting started guide to Singer.
marsupialtail/quokka
Making data lake work for time series
stitchfix/hamilton
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
Cinchoo/ChoETL
ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)
YotpoLtd/metorikku
A simplified, lightweight ETL Framework based on Apache Spark
seanharr11/etlalchemy
Extract, Transform, Load: Any SQL Database in 4 lines of Code.
apache/seatunnel-web
SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
flow-php/flow
Flow PHP - data processing framework
usc-isi-i2/kgtk
Knowledge Graph Toolkit
quintoandar/butterfree
A tool for building feature stores.
data-dot-all/dataall
A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.
Nextdoor/bender
Bender - Serverless ETL Framework
ceumicrodata/mETL
mito ETL tool
dalenewman/Transformalize
Configurable Extract, Transform, and Load
BitwiseInc/Hydrograph
A visual ETL development and debugging tool for big data
duoan/OpenKettleWebUI
一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中
halestudio/hale
(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)
globalbioticinteractions/globalbioticinteractions
Global Biotic Interactions provides access to existing species interaction datasets
frankframework/frankframework
The Frank!Framework is an easy-to-use, stateless integration framework which allows (transactional) messages to be modified and exchanged between different systems.
patterns-app/patterns-devkit
Data pipelines from re-usable components
usc-isi-i2/dig-etl-engine
Download DIG to run on your laptop or server.
restarone/violet_rails
an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL
geopython/stetl
Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.
socialpoint-labs/sqlbucket
Lightweight library to write, orchestrate and test your SQL ETL. Writing ETL with data integrity in mind.
maxim2266/csvplus
csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.
vim89/datapipelines-essentials-python
Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
MassStreetAnalytics/etl-framework
A framework for moving data into a data warehouse.
harry-s-grewal/mls-real-estate-scraper-for-realtor.ca
Python MLS and Real-Estate Data Scraper for the Realtor.ca Website
tharwaninitin/etlflow
EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.