Python client to Cloudera Impala.
Impala allows you to rapidly analyze large, distributed data sets. But it doesn't integrate easily with your ad hoc (Python) analytical tools (pandas, scikit-learn). impyla aims to remedy this.
This package offers:
-
Lightweight,
pip-installable package for Impala-driven analytics anywhere -
Integration with pandas (and therefore the rest of the Python data stack, including scikit-learn and matplotlib)
-
Integration with MADlib, enabling scalable, in-database, distributed machine learning (see the Impala port of MADlib)
Eventually, we'll also support:
-
Integration with Spark
-
Running Python UDFs by compiling them to LLVM IR
For the latest (dev) version, clone the repo:
git clone git://github.com/laserson/impyla.git
cd impyla
python setup.py install
NOTE: running pip install impyla will install version 0.7 of this package,
which very different. We suggest you either clone or download the repo from
the GitHub page.
Impyla implements the Python DB API 2 (PEP 249)
import impala.dbapi
conn = impala.dbapi.connect(host='my.host.com', port=21050)
cursor = conn.cursor()
cursor.execute('SELECT * FROM table LIMIT 100')
for row in cursor:
process(row)
You can also get back a pandas DataFrame object
import impala.util
df = impala.util.as_pandas(cur)
# carry df through scikit-learn, for example