GPBoost: Combining Tree-Boosting with Gaussian Process and Mixed Effects Models
Table of Contents
Get started
GPBoost is a software library for combining tree-boosting with Gaussian process and grouped random effects models (aka mixed effects models or latent Gaussian models). It also allows for independently applying tree-boosting as well as Gaussian process and (generalized) linear mixed effects models (LMMs and GLMMs). The GPBoost library is predominantly written in C++, it has a C interface, and there exist both a Python package and an R package.
For more information, you may want to have a look at:
- The Python package and R package with installation instructions for the Python and R packages
- The companion articles Sigrist (2020) and Sigrist (2021), this blog post on how to combine tree-boosting with mixed effects models, this blog post on how to combine tree-boosting with Gaussian processes for spatial data, or this blog post on how to use GPBoost for generalized linear mixed effects models (GLMMs)
- The GPBoost R and Python demo illustrating how GPBoost can be used in R and Python
- Detailed Python examples and R examples
- Main parameters presenting the most important parameters / settings for the GPBoost library
- Parameters an exhaustive list of all possible parametes and customizations for the tree-boosting part
- The CLI installation guide explaining how to install the command line interface (CLI) version
- Comments on computational efficiency and large data
Modeling background
The GPBoost library allows for combining tree-boosting with Gaussian process and grouped random effects models in order to leverage advantages of both techniques and to remedy drawbacks of these two modeling approaches.
Background on Gaussian process and grouped random effects models
Tree-boosting has the following advantages and disadvantages:
| Advantages of tree-boosting | Disadvantages of tree-boosting |
|---|---|
| - Achieves state-of-the-art predictive accuracy | - Assumes conditional independence of samples |
| - Automatic modeling of non-linearities, discontinuities, and complex high-order interactions | - Produces discontinuous predictions for, e.g., spatial data |
| - Robust to outliers in and multicollinearity among predictor variables | - Can have difficulty with high-cardinality categorical variables |
| - Scale-invariant to monotone transformations of the predictor variables | |
| - Automatic handling of missing values in predictor variables |
Gaussian process (GPs) and grouped random effects models (aka mixed effects models or latent Gaussian models) have the following advantages and disadvantages:
| Advantages of GPs / random effects models | Disadvantages of GPs / random effects models |
|---|---|
| - Probabilistic predictions which allows for uncertainty quantification | - Zero or a linear prior mean (predictor, fixed effects) function |
| - Incorporation of reasonable prior knowledge. E.g. for spatial data: "close samples are more similar to each other than distant samples" and a function should vary contiunuously / smoothly over space | |
| - Modeling of dependency which, among other things, can allow for more efficient learning of the fixed effects (predictor) function | |
| - Grouped random effects can be used for modeling high-cardinality categorical variables |
GPBoost and LaGaBoost algorithms
The GPBoost library implements two algorithms for combining tree-boosting with Gaussian process and grouped random effects models: the GPBoost algorithm (Sigrist, 2020) for data with a Gaussian likelihood (conditional distribution of data) and the LaGaBoost algorithm (Sigrist, 2021) for data with non-Gaussian likelihoods.
For Gaussian likelihoods (GPBoost algorithm), it is assumed that the response variable (aka label) y is the sum of a potentially non-linear mean function F(X) and random effects Zb:
y = F(X) + Zb + xi
where xi is an independent error term and X are predictor variables (aka covariates or features).
For non-Gaussian likelihoods (LaGaBoost algorithm), it is assumed that the response variable y follows some distribution p(y|m) and that a (potentially multivariate) parameter m of this distribution is related to a non-linear function F(X) and random effects Zb:
y ~ p(y|m)
m = G(F(X) + Zb)
where G() is a so-called link function.
In the GPBoost library, the random effects can consists of
- Gaussian processes (including random coefficient processes)
- Grouped random effects (including nested, crossed, and random coefficient effects)
- Combinations of the above
Learning the above-mentioned models means learning both the covariance parameters (aka hyperparameters) of the random effects and the predictor function F(X). Both the GPBoost and the LaGaBoost algorithms iteratively learn the covariance parameters and add a tree to the ensemble of trees F(X) using a gradient and/or a Newton boosting step. In the GPBoost library, covariance parameters can (currently) be learned using (Nesterov accelerated) gradient descent, Fisher scoring (aka natural gradient descent), and Nelder-Mead. Further, trees are learned using the LightGBM library.
See Sigrist (2020) and Sigrist (2021) for more details.
News
- See the GitHub releases page
- 04/06/2020 : First release of GPBoost
Open issues - contribute
Software issues
- Add Python tests (see corresponding R tests)
- Setting up a CI environment
Computational issues
- Add GPU support for Gaussian processes
- Add CHOLMOD support
Methodological issues
- Add a spatio-temporal Gaussian process model (e.g. a separable one)
- Add possibility to predict latent Gaussian processes and random effects (e.g. random coefficients)
- Implement more approaches such that computations scale well (memory and time) for Gaussian process models and mixed effects models with more than one grouping variable for non-Gaussian data
- Support sample weights
References
- Sigrist Fabio. "Gaussian Process Boosting". Preprint (2020).
- Sigrist Fabio. "Latent Gaussian Model Boosting". Preprint (2021).
- Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu. "LightGBM: A Highly Efficient Gradient Boosting Decision Tree". Advances in Neural Information Processing Systems 30 (NIPS 2017), pp. 3149-3157.
License
This project is licensed under the terms of the Apache License 2.0. See LICENSE for additional details.