MLOS is an ML-powered infrastructure and methodology to democratize and automate Performance Engineering. MLOS enables continuous, instance-based, robust, and trackable systems optimization.
From the MLOS paper at DEEM 2020
All systems software (e.g. SqlServer, MySQL, LevelDB, OpenSSL, etc.) is full of parameter choices.
Sometimes these are encoded in the software as constants embedded in the code (e.g. choice of abstract data structure implementation, buffer limit size or alignment, etc.). Other times they may be exposed as configuration parameters either at startup or runtime.
Careful selection of these parameters can yield dramatic performance differences for different contexts of a system (e.g. different workloads, hardware, etc.). Note that performance can be interpreted in different ways (e.g. reducing average/variability of latency/memory, increasing throughput, decreasing MTTR, etc.)
Generally speaking, this process is referred to as Software Performance Engineering, and typically involves a lot of manual effort that is brittle and not well tracked.
MLOS is about using machine-learning and data-science to optimize systems for a given context through these tunable choices.
Roughly, this can happen in two modes:
-
Offline (e.g. at development time)
In this case, developers can use (micro)benchmarks to explore a parameter space for a component either interactively or with a background CI/CD pipeline and then interact with that data through a notebook experience to select the right value to check in to the code, along with the results of the experiments and analysis, all encoded in the notebook.
-
Online (e.g. at runtime)
In this case a system component provides hooks to adjust its parameters at runtime and exports data about its current state/performance. These can be combined with additional contextual information from the system to build a model (or simple heuristics) to invoke the hooks to adjust the component to improve performance at runtime.
To achieve this MLOS provides:
-
Code Annotations to help describe additional settings metadata for tunables (a.k.a.
Settings).For instance, metadata can include things like constraints on acceptable values a Setting can take on as well as developer intuition to help guide the automated search process.
Currently these are implemented as C# Attributes to provide reflection and easy cross-platform and cross-compiler support for C++ projects.
-
Code Generation tools to use that metadata to expose those settings to different target systems/languages (e.g. Python Notebooks, C++, C#, etc.)
For instance, we generate efficient messages over shared memory communication channels for
-
exporting data about the component using that Setting
For instance, this may include performance statistics, workload traces, etc.
-
receiving feedback (e.g. to change the Setting's value)
This may involve a reconfiguration step or simply update a cache for the next instantiation to read.
-
-
An external agent (
Mlos.Agent.Server) which can consume the information exported by the target system (e.g. SqlServer, MySQL, LevelDB, etc.) with mimimal impact on the target system.The external agent can perform workload summarization, binning, cataloging, model inference, heuristic invocation, etc. based on the events exposed by the target system to then influence it.
Once hooks are created in the target system, iteration on the external agent can be more rapidly developed and deployed.
Some of the examples require only the installation of the mlos Python library. These examples do not use the shared memory infrastructure.
First, download the mlos code using git:
$ git clone https://github.com/microsoft/MLOS.git
For this simplified installation, it's recommended to use the Anaconda python distribution. For a slimer installation experience, you can also use the miniconda installer. After installing either anaconda or miniconda, you can create a new environment with all requirements for the examples using
$ conda env create -f MLOS/source/Mlos.Notebooks/environment.yml
The environment will be called mlos_python_environment and you can activate it as follows:
$ conda activate mlos_python_environment
Use pip to install the Python library:
$ pip install MLOS/source/Mlos.Python/
Alternatively, you can also install the package directly without checking out the code:
$ pip install "git+https://github.com/microsoft/MLOS.git#egg=mlos&subdirectory=source/Mlos.Python"
However, this does not include the examples and requires you to set up your environment manually.
After this installation, you can run any of the Python-only example notebooks. To do so you can:
$ python -m ipykernel install --user --name=mlos_environment
$ jupyter-notebook --notebook-dir=MLOS/source/Mlos.Notebooks
Jupyter will list a few notebooks. A good place to start is the BayesianOptimization.ipynb, which provides an Introduction to Bayesian Optimization.
MLOS supports Windows and Linux build environments.
For detailed instructions, please refer to:
Code and documentation for examples of using MLOS to optimize a system are described in the Notebooks section. Additional code is in the source/Examples source directory. You can find the source of the notebooks on github as well.
-
Additional overview documentation is available in the documentation tree.
-
Individual components may also include more detailed documentation in their respective subdirectories.
We welcome contributions! Please see Contributing and Code of Conduct for details.
Also, please see the Roadmap of planned features.
For more formal enquiries, you can contact us.