Pinned Repositories
incubator-samza
Mirror of Apache Samza
kafka-embedded
Runs embedded, in-memory Apache Kafka instances. Helpful for integration testing.
kafka-manager
A tool for managing Apache Kafka.
kafka-spark-consumer
kafka-storm-starter
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+, while using Apache Avro as the data serialization format.
kangaroo
Hadoop utilities for Kafka
klio
Smarter data pipelines for audio.
mpire
A Python package for easy multiprocessing, but faster than multiprocessing
Neuraxle
Build neat pipelines with the right abstractions to do AutoML. Let your pipeline steps have hyperparameter spaces. Enable checkpoints to cut duplicate calculations. Go from research to production environment easily.
rabit
Reliable Allreduce and Broadcast Interface for distributed machine learning
data-processing's Repositories
data-processing/kafka-embedded
Runs embedded, in-memory Apache Kafka instances. Helpful for integration testing.
data-processing/kafka-manager
A tool for managing Apache Kafka.
data-processing/kangaroo
Hadoop utilities for Kafka
data-processing/klio
Smarter data pipelines for audio.
data-processing/mpire
A Python package for easy multiprocessing, but faster than multiprocessing
data-processing/Neuraxle
Build neat pipelines with the right abstractions to do AutoML. Let your pipeline steps have hyperparameter spaces. Enable checkpoints to cut duplicate calculations. Go from research to production environment easily.
data-processing/rabit
Reliable Allreduce and Broadcast Interface for distributed machine learning
data-processing/akka-streams-example
data-processing/bloop
A hot bloop for your productivity
data-processing/crawler4j
Open Source Web Crawler for Java
data-processing/dask
Task scheduling and blocked algorithms for parallel processing
data-processing/dataduct
DataPipeline for humans.
data-processing/disque
Disque is a distributed message broker
data-processing/emr-bootstrap-actions
This repository hold the Amazon Elastic MapReduce sample bootstrap actions
data-processing/faust
Python Stream Processing
data-processing/fireant
Data analysis and reporting tool for quick access to custom charts and tables in Jupyter Notebooks and in the shell.
data-processing/flink
Mirror of Apache Flink
data-processing/gain
Web crawling framework based on asyncio for everyone.
data-processing/GoogleScraper
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, Baidu and others) by using proxies (socks4/5, http proxy) and with many different IP's, including asynchronous networking support (very fast).
data-processing/grpc-java
The Java gRPC implementation. HTTP/2 based RPC
data-processing/HiBench
HiBench is a Hadoop benchmark suite.
data-processing/hydra
Hydra is a framework for elegantly configuring complex applications
data-processing/Persimmon
A visual dataflow programming language for sklearn
data-processing/pyspider
A Powerful Spider System with Web UI
data-processing/realtime-analytics
data-processing/samoa
SAMOA (Scalable Advanced Massive Online Analysis) is a distributed streaming machine learning (ML) framework that contains a programing abstraction for distributed streaming ML algorithms.
data-processing/spark-redshift
Spark and Redshift integration
data-processing/Stream-Framework
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
data-processing/thredo
data-processing/ufora
Compiled, automatically parallel Python for data science