Hivemall: Hive scalable machine learning library
Hivemall is a scalable machine learning library that runs on Apache Hive. Hivemall is designed to be scalable to the number of training instances as well as the number of training features.
Supported Algorithms
Hivemall provides machine learning functionality as well as feature engineering functions through UDFs/UDAFs/UDTFs of Hive.
Binary Classfication
-
Passive Aggressive (PA, PA1, PA2)
-
Confidence Weighted (CW)
-
Adaptive Regularization of Weight Vectors (AROW)
-
Soft Confidence Weighted (SCW1, SCW2)
-
AdaGradRDA (w/ hinge loss)
-
Factorization Machine (w/ logistic loss)
My recommendation is AROW, SCW1, AdaGradRDA, and Factorization Machine while it depends.
Multi-class Classfication
-
Passive Aggressive (PA, PA1, PA2)
-
Confidence Weighted (CW)
-
Adaptive Regularization of Weight Vectors (AROW)
-
Soft Confidence Weighted (SCW1, SCW2)
-
Random Forest Classifier
-
Gradient Tree Boosting (Experimental)
My recommendation is AROW and SCW while it depends.
Regression
-
AdaGrad, AdaDelta (w/ logistic Loss)
-
Passive Aggressive Regression (PA1, PA2)
-
AROW regression
-
Random Forest Regressor
-
Factorization Machine (w/ squared loss)
My recommendation for is AROW regression, AdaDelta, and Factorization Machine while it depends.
Recommendation
-
Matrix Factorization (sgd, adagrad)
-
Factorization Machine (squared loss for rating prediction)
k-Nearest Neighbor
Anomaly Detection
Natural Language Processing
- English/Japanese Text Tokenzier
Feature engineering
-
Feature Hashing (MurmurHash, SHA1)
-
Feature scaling (Min-Max Normalization, Z-Score)
-
Feature instances amplifier that reduces iterations on training
-
TF-IDF vectorizer
-
Bias clause
-
Data generator for one-vs-the-rest classifiers
System requirements
-
Hive 0.12 or later
-
Java 7 or later
Note: It would work for Java 6 except hivemall-nlp but we recommend to use Java 7 or later.
Basic Usage
Find more examples on our wiki page and find a brief introduction to Hivemall in this slide.
Copyright
Copyright (C) 2015-2016 Makoto YUI
Copyright (C) 2013-2015 National Institute of Advanced Industrial Science and Technology (AIST)
Put the above copyrights for the services/softwares that use Hivemall.
Support
Support is through the issue list, not by a direct e-mail.
References
Please refer the following paper for research uses:
-
Makoto Yui. ``Hivemall: Scalable Machine Learning Library for Apache Hive'', 2014 Hadoop Summit, June 2014. [slide]
-
Makoto Yui and Isao Kojima. ``Hivemall: Hive scalable machine learning library'' (demo), NIPS 2013 Workshop on Machine Learning Open Source Software: Towards Open Workflows, Dec 2013.
-
Makoto Yui and Isao Kojima. ``A Database-Hadoop Hybrid Approach to Scalable Machine Learning'', Proc. IEEE 2nd International Congress on Big Data, July 2013 [paper] [slide]
Awards
Acknowledgment
This work was supported in part by a JSPS grant-in-aid for young scientists (B) #24700111 and a JSPS grant-in-aid for scientific research (A) #24240015.