data-processing
There are 1675 repositories under data-processing topic.
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
onceupon/Bash-Oneliner
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
johnkerl/miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
TomWright/dasel
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
NVIDIA/DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
modelscope/data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
deepseek-ai/smallpond
A lightweight data processing framework built on DuckDB and 3FS.
unionai-oss/pandera
A light-weight, flexible, and expressive statistical data testing library
cocoindex-io/cocoindex
Data transformation framework for AI. Ultra performant, with incremental processing. 🌟 Star if you like it!
dashbitco/broadway
Concurrent and multi-stage data ingestion and data processing with Elixir
microsoft/DialoGPT
Large-scale pretraining for dialogue
asyml/texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
numaproj/numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
bytewax/bytewax
Python Stream Processing
python-bonobo/bonobo
Extract Transform Load for Python 3.5+
pyper-dev/pyper
Concurrent Python made simple
OpenDCAI/DataFlow
Easy Data Preparation with latest LLMs-based Operators and Pipelines.
GoogleCloudPlatform/data-science-on-gcp
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
allenai/dolma
Data and tools for generating and inspecting OLMo pre-training data.
NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
microsoft/GODEL
Large-scale pretrained models for goal-directed dialog
GoogleCloudPlatform/DataflowJavaSDK
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
benibela/xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
asyml/texar-pytorch
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
hstreamdb/hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
ChenghaoMou/text-dedup
All-in-one text de-duplication
SebKrantz/collapse
Advanced and Fast Data Transformation in R
jofpin/synthBTC
A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.
infoslack/awesome-kafka
A list about Apache Kafka
kousun12/eternal
👾~ music, eternal ~ 👾
maykulkarni/Machine-Learning-Notebooks
Machine Learning notebooks for refreshing concepts.
Puchaczov/Musoq
SQL Syntax without any database
constellation-rs/amadeus
Harmonious distributed data analysis in Rust.
polyaxon/haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
keithorange/PatternPy
📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms, Supports & Resistances. For experts & beginners. #TradingMadeEasy 🔥
msamogh/nonechucks
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!