This repository demonstrates the usage of Gurobi Machine Learning.
Gurobi Machine Learning is a library that integrates mathematical optimization and machine learning. This project explores its usage in the context of causal inference.
The integration of mathematical optimization and machine learning has gained increasing attention due to their complementary nature in solving complex problems.
Mathematical optimization involves the use of mathematical techniques to determine the best solution for a given problem, while machine learning involves the use of algorithms to learn patterns from data and make predictions or decisions. There are several ways in which mathematical optimization and machine learning can be combined.
One approach is to use trained machine learning models as variables in mathematical optimization. Gurobi Machine Learning, which was released in November 2022, is an example of this approach. This method can speed up the calculation of mathematical optimization when dealing with large datasets.
Another approach is to use mathematical optimization as inputs to machine learning. This is exemplified by MIPLearn, which combines mixed-integer programming (MIP) with machine learning algorithms. This approach can help to improve the performance of machine learning models by incorporating optimization techniques.
Machine learning can also be used to optimize tuning parameters in mathematical optimization, which is another way to integrate these fields. Additionally, mathematical optimization can be used to select features and choose the best machine learning model.
However, when machine learning models is used in the objective function or constraints of mathematical optimization, treating the machine learning model as a black box can be too slow to compute. Therefore, alternative methods need to be explored to integrate these fields more effectively.
In this project, we explore Gurobi Machine Learning for using machine learning models in mathematical optimization.
Gurobi Machine Learning formulates machine learning models in mixed-integer programming (MIP).
This library supports the use of machine learning models in mathematical optimization by translating them into mathematical expressions. However, some models may require approximation to fit into the MIP framework. For example, logistic regression cannot be represented directly in Gurobi using linear constraints, so piecewise-linear approximation is used instead.
There are limitations to the machine learning models that can be used in Gurobi Machine Learning. Currently, it only supports certain models from scikit-learn, such as LinearRegression, PLSRegression, LogisticRegression, MLPRegressor, DecisionTreeRegressor, GradientBoostingRegressor, and RandomForestRegressor. Keras and PyTorch are also supported, but only Linear and Relu layers for activation functions are supported.
Unsupported models, such as LightGBM, will generate following error message.
The application of Gurobi Machine Learning is particularly advantageous in the context of causal inference, as it allows for the incorporation of causal relationships between variables into the optimization process. Causal inference is a fundamental task in many fields, such as economics, social sciences, and healthcare, where understanding the effects of interventions is crucial.
One popular approach to causal inference in machine learning is uplift modeling, which is a technique used to identify the difference in treatment effect between two groups.
It helps determine whether a treatment will have an impact on each individual instance. However, uplift modeling may not always provide sufficient guidance for decision-making.
In contrast, the integration of mathematical optimization and machine learning with Gurobi Machine Learning can help answer more complex causal inference questions that involve decision-making under budget and other constraints. For example, in the student enrollment model provided by Gurobi Machine Learning, the goal is to determine which scholarships to offer to maximize enrollment while staying within budget limits.
In this way, Gurobi Machine Learning can offer a more comprehensive solution to the problem of causal inference.
1. Causal inference demo with Gurobi Machine Learning
The integration of mathematical optimization and machine learning has a lot of potential for causal inference in specific situations.
However, there are limitations such as the fact that only LogisticRegression is currently supported as a binary classifier in scikit-learn, which may not be sufficient for solving complex real-world problems.
During the investigation, it was discovered that Gurobi Machine Learning supports each regressor and classifier separately instead of treating everything as black boxes. This means that Gurobi must support each model individually, which can be time-consuming.
It is worth noting that this feature is a recent addition to Gurobi, introduced in version 10.0, released in November 2022. Therefore, it is essential to stay updated in this area and keep an eye out for any future developments.