Optimus is the missing framework for cleaning and pre-processing data in a distributed fashion. It uses all the power of Apache Spark (optimized via Catalyst) to do so. It implements several handy tools for data wrangling and munging that will make your life much easier. The first obvious advantage over any other public data cleaning library or framework is that it will work on your laptop or your big cluster, and second, it is amazingly easy to install, use and understand.
Click on Optimus to enter in our Website:
And if you want to see some cool information and tutorials for Optimus check out our blog:
https://hioptimus.com/category/blog/
Please take a couple of minutes to help shape the Optimus' Roadmap:
The first version of Optimus is out now! Check out the new examples and documentation to see all the changes. We want to thank the projects we rely on: pixiedust and spark-df-profiling and the help from our amazing community. We hope that you find the framework useful and interesting. Have fun!
Optimus main stable branch will work now for Spark 2.2.0 The 1.6.x version is now under maintenance, the last tag release for this Spark version is the 0.4.0. We strongly suggest that you use the >2.x version of the framework because the new improvements and features will be added now on this version.
- Apache Spark 2.2.0
- Python>=3.5
In your terminal just type:
pip install optimuspyspark
Contributions go far beyond pull requests and commits. We are very happy to receive any kind of contributions including:
- Documentation updates, enhancements, designs, or bugfixes.
- Spelling or grammar fixes.
- README.md corrections or redesigns.
- Adding unit, or functional tests
- Triaging GitHub issues -- especially determining whether an issue still persists or is reproducible.
- Searching #optimusdata on twitter and helping someone else who needs help.
- Blogging, speaking about, or creating tutorials about Optimus and its many features.
- Helping others in our optimus gitter channel.
- Project Manager: Argenis León
- Original developers: Andrea Rosales, Hugo Reyes, Alberto Bonsanto
- Principal developer and maintainer: Favio Vázquez
Apache 2.0 © Iron
