/python-livy

Tool and library to interact with Apache Livy

Primary LanguagePythonMIT LicenseMIT

Python-Livy

test status

Still under developing. API might not be stable at this time.

Lightweight tool to interact with Apache Livy. Provide both CLI tools and library that works nicely in pure and native Python.

screenshot

features:

  1. Log watching and parsing

    It could parse livy's mixed-stdout logs back to log records and submit to Python's logging system. Then we could watch for the events in a much friendly way.

  2. Configurable

    For using CLI tools, it provides configuration system for saving common used variables on local storage. No need to set every option on each command.

  3. Human friendly log viewer

    On using CLI for reading logs, it could have colored logs and progress bar. Besides, we could highlight or hide specific logger via arguments (e.g. hide TaskSetManager as we've got progress bar).

  4. Extensible

    Custom function could be triggered during submission. So bring it to infinity...and beyond!

  5. Fully tested

    100% coverage. Both with and without extra dependencies are tested, thanks to Github action.

Installation

Install via pip:

pip install -U 'git+https://github.com/tzing/python-livy.git#egg=livy[pretty]'

Or get the wheel from release.

Docs

https://tzing.github.io/python-livy/