/Optimus

:rocket: Optimus is the missing framework for cleansing (cleaning and much more), pre-processing and exploratory data analysis in a distributed fashion with Apache Spark.

Primary LanguagePythonApache License 2.0Apache-2.0

Logo Optimus

PyPI version Build Status Documentation Status built_by iron Updates GitHub release Codacy Badge StackShare

Platforms Dependency Status Quality Gate Code Health Coverage Status

Join the chat at https://gitter.im/optimuspyspark/Lobby

Optimus is the missing framework for cleaning and pre-processing data in a distributed fashion. It uses all the power of Apache Spark (optimized via Catalyst) to do so. It implements several handy tools for data wrangling and munging that will make your life much easier. The first obvious advantage over any other public data cleaning library or framework is that it will work on your laptop or your big cluster, and second, it is amazingly easy to install, use and understand.

Click on Optimus to enter in our Website:

Click Me!

And if you want to see some cool information and tutorials for Optimus check out our blog:

https://hioptimus.com/category/blog/

Click below for the official documentation

Documentation

Survey

Please take a couple of minutes to help shape the Optimus' Roadmap:

Announcement!!

The first version of Optimus is out now! Check out the new examples and documentation to see all the changes. We want to thank the projects we rely on: pixiedust and spark-df-profiling and the help from our amazing community. We hope that you find the framework useful and interesting. Have fun!

Optimus for Spark 1.6.x

Optimus main stable branch will work now for Spark 2.2.0 The 1.6.x version is now under maintenance, the last tag release for this Spark version is the 0.4.0. We strongly suggest that you use the >2.x version of the framework because the new improvements and features will be added now on this version.

Requirements

  • Apache Spark 2.2.0
  • Python>=3.5

Installation (pip):

In your terminal just type:

pip install optimuspyspark

Contributing to Optimus (based on Webpack)

Contributions go far beyond pull requests and commits. We are very happy to receive any kind of contributions including:

Contributors:

Logo Data

License:

Apache 2.0 © Iron

Logo Iron

Optimus twitter