ML-DEECo is a machine-learning-enabled component model for adaptive component architectures. It is based on DEECo component model, which features autonomic components and dynamic component coalitions (called ensembles). ML-DEECo allows exploiting machine learning in decisions about adapting component coalitions at runtime.
The framework provides abstractions for getting predictions (estimates) about the future state of the system. It uses machine learning models trained in a supervised manner to obtain these predictions. A simulation of the system is run to collect data used for training the ML model. The simulation can then be run again with the trained model to see the impact of the learned model on the system.
- Installation
- Short summary of ensemble-based component systems
- Usage
- Notes to implementation
- Examples
Use pip
:
pip install .
For local development, installing with --editable
is recommended:
pip install --editable .
ML-DEECo is based on the DEECo component model (Bures et al.). We want to very briefly introduce the concepts of components and ensembles before describing the details of ML-DEECo usage.
Components are autonomous agents in the system. Based on the knowledge they have, they can independently operate in the environment. In ML-DEECo, the components have an actuate
method which is executed in every step of the simulation, and it thus comprises the behavior of the component.
An ensemble is a group of components created to express that the components are bound with the same objective (i.e., behavior). The ensembles are used for knowledge exchange among the member components and their coordination. The ensembles are formed and dismantled dynamically at runtime as the objectives emerge in the system and are completed. The member components of the ensemble are also selected dynamically at every step of the simulation.
The ML-DEECo framework provides abstractions for creating components and ensembles and assigning machine learning estimates to them. A simulation can then be run with the components and ensembles to observe behavior of the system and collect data for training the estimates. The trained estimate can then be used in the next run of the simulation.
The configuration of the simulation (lists of components and ensembles, number of steps of the simulation, etc.) is managed by a class derived from the Experiment
class. See section Running the simulation for more details.
A typical ML experiment in ML-DEECo consists of several iterations. In each iteration, the simulation is run multiple times to collect data, and then the training of the ML model is performed. Each run of the simulation consists of steps -- in each step, the ensembles are reformed and the components are actuated.
The configuration is stored in the Configuration
object which is passed to the Experiment
when creating its instance, and it is then accessible through the Experiment.config
field.
The configuration can either be created by passing the variables as keyword arguments of the constructor of the Configuration
object:
from ml_deeco.simulation import Configuration
configuration = Configuration(
iterations=2, # we run two iterations -- the ML model trains between iterations
simulations=1, # one simulation in each iteration
steps=80, # the simulation is run for 80 steps
)
or it can be loaded from a YAML file:
from ml_deeco.simulation import Configuration
configuration = Configuration()
configuration.loadConfigurationFromFile('config.yaml')
When loading a new configuration file, the values of the configuration are replaced in a recursive manner. Only the values which exist in the new file are updated and the values which were only in the previous configuration are kept.
The most important top-level keys of the configuration are:
name
– name of the experiment (it is used as a label in some plots produced by the framework),output
– folder for the outputs of the experiment run (plots, training data, ...),iterations
– number of iterations of the experiment,simulations
– number of simulation runs in each iteration,steps
– number of steps of the simulation,verbose
: verbosity level of the text output of the experiment (0 = no output, 2 = recommended, 4 = most verbose).plot
– set to true to produce evaluation plots for the ML models,estimators
– described in section Configuring estimators using the YAML files
This is an example YAML file with a configuration of an experiment (smart farm simulation featuring drones, birds and chargers).
name: 12drones
output: results/12drones
iterations: 2
simulations: 2
steps: 500
estimators:
batteryEstimator:
class: ml_deeco.estimators.LinearRegressionEstimator
args:
name: "Battery"
outputFolder: 'battery'
accumulateData: True
saveCharts: False
locals:
drones: 12
birds: 100
chargers: [
[17,29],
[22,21],
[28,13],
]
We first set the name of the experiment and the output folder. Then, we provide the number of iterations, simulations and steps. There is one estimator defined for predicting the battery of the drone – using the linear regression model. Lastly we set the values specific to this experiment to the locals
key – number of drones in the system, number of birds, locations of chargers.
The ml_deeco.simulation
module offers a base class Component
for defining components. Furthermore, we provide
StationaryComponent2D
and MovingComponent2D
(both derived from Component
) to represent components on a 2D map. These have a location
in a 2D space defined by an instance of ml_deeco.simulation.Point2D
class.
Each component has an actuate
method which is executed in every step of the simulation and should be implemented by the user.
The MovingComponent2D
offers the move
method which will move the component in a direction towards a target defined in the parameter. If the agent arrives at the target, the move
method will return True
.
from ml_deeco.simulation import StationaryComponent2D, MovingComponent2D
# Example of a stationary component -- a charging station
class Charger(StationaryComponent2D):
def __init__(self, location):
super().__init__(location)
self.charging_drones = []
# a drone at the location of the charger can start charging
def startCharging(self, drone):
if drone.location == self.location:
self.charging_drones.append(drone)
def actuate(self):
# we charge the drones
for drone in self.charging_drones:
drone.battery += 0.01
if drone.battery == 1:
# fully charged
drone.station = None
# Example of a moving component -- a flying drone
class Drone(MovingComponent2D):
def __init__(self, location, speed):
super().__init__(location, speed)
self.battery = 1
self.charger = None
def actuate(self):
# if the drone has an assigned charger
if self.charger:
# fly towards it
if self.move(self.charger.location):
# drone arrived at the location of the charger
self.charger.startCharging(self)
Ensembles are meant for coordination of the components. The base class for ensembles is ml_deeco.simulation.Ensemble
. Each ensemble has a priority specified by overriding the priority
method. Furthermore, ensemble can contain components in static and dynamic roles. Static roles are represented simply as variables of the ensemble.
The declaration of a dynamic role is done via the someOf
(meaning a list of components) or oneOf
(single component) function with the component type as an argument. The components are assigned and re-assigned to the dynamic roles (we say that a component becomes a member of the ensemble) by the framework in every step of the simulation. To select the members for the role, several conditions can be specified using decorators:
select
is a predicate, which the component must pass to be picked;utility
orders the components;cardinality
sets the maximum (or both minimum and maximum) allowed number of components to be picked.
The member selection works by first finding all components of the correct type that pass the select
predicate, then ordering them by the utility
(higher utility is better) and using the cardinality
to limit the number of selected members. The cardinality
can also be used to limit the minimal number of member components – if there are not enough components passing the selection, the ensemble cannot be active at the time.
from ml_deeco.simulation import Ensemble, someOf
class ChargingAssignment(Ensemble):
# static role
charger: Charger
# dynamic role
drones: List[Drone] = someOf(Drone)
# we select those drones which need charging
@drones.select
def need_charging(self, drone, otherEnsembles):
return drone.needs_charging
# order them by the missing battery (so the drones with less battery are selected first)
@drones.utility
def missing_battery(self, drone):
return 1 - drone.battery
# and limit the cardinality to the number of free slots on the charger
@drones.cardinality
def free_slots(self):
return 0, self.charger.free_slots
def __init__(self, charger):
super().__init__()
self.charger = charger
def actuate(self):
# assign the charger to each drone -- it will start flying towards it to charge
for drone in self.drones:
drone.station = self.charger
The framework performs ensemble materialization (selection of the ensembles which should be active at this time) in every step of the simulation. The ensembles are materialized in a greedy fashion, ordered by their priority (descending). Only those ensembles for which all roles were assigned appropriate number of members (conforming to the cardinality) can be materialized. For all materialized ensembles, the actuate
method is called.
There are two types of tasks our framework focuses on – value estimate and time-to-condition estimate.
In the value estimate, we use the currently available observations to predict some value that can be observed only at some future point. Based on the type of the estimated value, the supervised ML models are usually divided into regression and classification.
The time-to-condition estimates focuses on predicting how long it will take until some condition will become true. This is done by specifying a condition over some future values of component fields.
The definition of each estimate is split to three parts:
- The definition of the
Estimator
– machine learning model and storage for the collected data. - The declaration of the
Estimate
field in the component or ensemble. - The definition of inputs, target and guards. These are realized as decorators on component/ensemble fields and getter functions of the component.
All of these steps are realized using the ml_deeco.estimators
module.
Estimator represents the underlying machine learning model for computing the estimates. The framework currently provides implementation of several estimators:
ConstantEstimator
– Predicts a constant value, does not train at all. This serves as a baseline.NeuralNetworkEstimator
– Fully-connected feedforward neural network implemented using the TensorFlow framework.LinearRegressionEstimator
– Linear regression model (for regression only) implemented using the Scikit-learn framework.
Other estimators can be implemented by deriving from the Estimator
class. See the ml_deauto.estimators.Estimator
class for more details and the list of methods which must be implemented. This way, one can use other ML model with ML-DEECo.
The estimators can be either instantiated directly or they can be specified in the YAML configuration files.
Common parameters for the initializer of the Estimator
s are:
experiment
– TheExperiment
instance in which the estimator is used.outputFolder
– The collected training data and evaluation of the training is exported there. Set toNone
to disable export.name
– String to identify theEstimator
in the printed output of the framework (ifprintLogs
isTrue
and verbosity level was set byml_deeco.utils.setVerboseLevel
).accumulateData
– If set toTrue
, data from all previous iterations are used for training. If set toFalse
(default), only the data from the last iteration are used for training.
The ConstantEstimator
is initialized with a constant, which is then returned every time predictions are requested. It can serve as a baseline in experiments.
The NeuralNetworkEstimator
uses TensorFlow framework to implement a feedforward neural network. It is enough to specify the number of neurons in hidden layers using the hidden_layers
parameter. The model is constructed and trained appropriate to the target feature specified by the Estimate
.
from ml_deeco.estimators import NeuralNetworkEstimator
experiment = ...
futureBatteryEstimator = NeuralNetworkEstimator(
experiment,
hidden_layers=[256, 256], # two hidden layers
outputFolder="results/drone_battery", name="Drone Battery"
)
The LinearRegressionEstimator
does not have any specific constructor parameters. It is implemented using the Scikit-learn framework.
A top-level key estimators
can be defined in the YAML configuration files. This key is expected to be a dictionary of entries, each defining one of the estimators.
Each entry shall contain two fields: class
and args
. The class
defines the type of the estimator. It is instantiated at the beginning of the experiment by the ML-DEECo framework. A full path to the class is expected, e.g. for LinearRegressionEstimator
, the class
is specified as ml_deeco.estimators.LinearRegressionEstimator
. The args
field represents the parameters for the constructor of the estimator (as specified in the previous section; the experiment
is loaded automatically).
Example configuration:
estimators:
batteryEstimator:
class: ml_deeco.estimators.NeuralNetworkEstimator
args:
hidden_layers: [64]
name: "Battery"
outputFolder: 'battery'
The estimate is created by instantiating the ValueEstimate
class (future value estimate – both regression and classification) or TimeEstimate
(time-to-condition estimate) and assigned as class variables of the component (in fact, they are implemented as properties).
In case of value estimate, the number of time steps we want to predict into the future is set using the inTimeSteps
or inTimeStepsRange
methods. The inTimeSteps
sets a fixed number of time steps between the current time and the time of the predictions. The inTimeStepsRange
methods allows specifying a range of valid time differences between the current time and the time of the predictions (the desired time difference is then specified as the last argument when obtaining the estimate).
For both ValueEstimate
and TimeEstimate
, the Estimator
(described in the previous section) must be assigned. That is done by the using
method. The estimators specified in the YAML file are instantiated by our framework, and they are available via attributes of the Experiment
instance. For example, if the experiment is configured using the example YAML configuration above, one can assign the defined battery estimator like this: ValueEstimate().using(experiment.batteryEstimator)
.
As this requires a reference to the experiment object, we also provide an additional option to prevent cyclic dependencies in the Python files. The estimator can be specified using a string identifier – ValueEstimate().using('batteryEstimator')
– and connected to the experiment later using the initEstimates
class method on Component
or Ensemble
– Drone.initEstimates(experiment.)
. An example usage of this can also be seen in the simple_example
.
Multiple estimates can be assigned to a component.
from ml_deeco.estimators import ValueEstimate
class Drone(MovingComponent2D):
futureBatteryEstimate = ValueEstimate().inTimeSteps(50)\
.using(futureBatteryEstimator) # defined earlier
# more code of the component
The estimates can be added to ensembles in a same way as to components – as class variables (properties).
To assign an estimate to a role, use the withEstimate
(value estimate) or withTimeEstimate
(time-to-condition estimate) methods of someOf
(or oneOf
). Only one estimate can be assigned to a role.
The Estimator
must be also assigned by the using
method. In case of value estimate, the number of time steps we want to predict into the future is set using the inTimeSteps
method.
from ml_deeco.simulation import Ensemble, someOf
waitingTimeEstimator = NeuralNetworkEstimator(
hidden_layers=[256, 256], # two hidden layers
outputFolder="results/waiting_time", name="Waiting time"
)
class DroneChargingAssignment(Ensemble):
# dynamic role with time estimate
drones: List[Drone] = someOf(Drone).withTimeEstimate()\
.using(waitingTimeEstimator)
# more code of the ensemble
The definition of inputs, target and guards is realized as decorators and getter functions. For estimates assigned to components and ensembles, the decorator has a syntax @<estimateName>.<configuration>
. For estimates assigned to roles, the syntax is @<roleName>.estimate.<configuration>
.
The decorators are applied to methods of the component or ensemble. For estimates assigned to components and ensembles, these methods should only have the self
parameter. For estimates assigned to roles, these methods are expected to have the self
parameter and a second parameter representing a component (the potential role member).
The inputs of the estimate are specified using the input()
decorator, optionally with a feature type as a parameter. We offer a NumericFeature(min, max)
, which performs normalization of the inputs, a CategoricalFeature(enum|list)
for one-hot encoding categorical values, and a BinaryFeature()
to represent boolean attributes.
Example of inputs for an estimate in a component (continued from earlier):
from ml_deeco.estimators import ValueEstimate, NumericFeature, CategoricalFeature
from ml_deeco.simulation import MovingComponent2D
class Drone(MovingComponent2D):
# create the estimate (as described earlier)
futureBatteryEstimate = ValueEstimate().inTimeSteps(50)\
.using(futureBatteryEstimator)
def __init__(self, location):
self.battery = 1
self.state = DroneState.IDLE
# more code
# numeric feature
@futureBatteryEstimate.input(NumericFeature(0, 1))
def battery(self):
return self.battery
# categorical feature constructed from an enum
@futureBatteryEstimate.input(CategoricalFeature(DroneState))
def drone_state(self):
return self.state
Example of input for an estimate connected to a role (continued from earlier):
class DroneChargingAssignment(Ensemble):
# dynamic role with time estimate (as described earlier)
drones: List[Drone] = someOf(Drone).withTimeEstimate()\
.using(waitingTimeEstimator)
@drones.estimate.input(NumericFeature(0, 1))
def battery(self, drone):
return drone.battery
The target is specified similarly to the inputs using target()
decorator. A Feature
can again be given as a parameter – this is how classification and regression tasks are distinguished. The feature is then used to set the appropriate number of neurons and the activation function of the last layer of the neural network and the loss function used for training (more details in Notes to implementation).
class Drone(MovingComponent2D):
# create the estimate and inputs as described earlier
...
# define the target -- regression task
@futureBatteryEstimate.target(NumericFeature(0, 1))
def battery(self):
return self.battery
For the time-to-condition estimate, a condition must be specified instead of the target value. The syntax is again similar – using the condition
decorator. If multiple conditions are provided, they are considered in an "and" manner.
class DroneChargingAssignment(Ensemble):
# create the estimate and inputs as described earlier
...
# define the condition (drone is accepted for charging)
@drones.estimate.condition
def is_accepted(self, drone):
return drone in self.charger.acceptedDrones
Guard functions can be specified using inputsValid
, targetsValid
and conditionValid
decorators to assess the validity of inputs and targets. The data are collected for training only if the guard conditions are satisfied. This can be used for example to prevent collecting data from components which are no longer active.
class Drone(MovingComponent2D):
# create the estimate, inputs and targets as described earlier
...
@futureBatteryEstimate.inputsValid
@futureBatteryEstimate.targetsValid
def not_terminated(self):
return self.state != DroneState.TERMINATED
The Estimate
object is callable, so the value of the estimate based on the current inputs can be obtained by calling the estimate as a function. For estimate assigned to a role, a component instance is expected as an argument of the call. If the ValueEstimate
was created using the inTimeStepsRange
method, an additional argument is expected when calling the estimate to set the desired time of prediction.
Example in a component:
class Drone(MovingComponent2D):
# create the estimate, inputs and targets as described earlier
...
def actuate(self):
estimatedFutureBattery = self.futureBatteryEstimate()
Example for a role:
class DroneChargingAssignment(Ensemble):
# create the estimate and inputs as described earlier
drones: List[Drone] = someOf(Drone).withTimeEstimate()\
.using(waitingTimeEstimator)
...
@drones.select
def drones(self, drone, otherEnsembles):
# we obtain the estimated waiting time here
waitingTime = self.drones.estimate(drone)
# and use it to decide whether the drone should ask for a charging slot
return drone.needsCharging(waitingTime)
The whole simulation can be assembled by subclasses the Experiment
class from the ml_deeco.simulation
module.
The user is required to implement the prepareSimulation
method, which is called before each simulation run, and it is expected to provide the components and ensembles for the simulation.
To run the experiment, use the run
method of the derived experiment class. The number of iterations, simulation runs, and steps in each simulation is set in the configuration object when creating a new experiment instance. The run
method then runs iterations
iterations. In each iteration, the simulation is run simulations
times. Each simulation is run for steps
steps. After finishing all the simulation runs in one iteration, the estimators (ML models) are trained on the data collected during the iteration. The next iteration will use the updated models.
The Experiment
class also provides several optional callbacks to collect data from the simulation runs. These are implemented by overriding the appropriate methods in the Experiment
class.
stepCallback
is called after each simulation step. It can be used for example to log data from the simulation. The parameters are:- list of all components in the system,
- list of materialized ensembles (in this time step),
- current time step (int).
prepareIteration
is an optional function to be run at the beginning of each iteration. It can be used for example to initialize logs for logging data during simulations.simulationCallback
is ran after each simulation.iterationCallback
is ran at the end of iteration after the ML training finished.
For better control over the simulation, one can also run the simulation loop manually. The functions materialize_ensembles
and actuate_components
(in the ml_deeco.simulation
module) can be useful for that (and we use them in the implementation of the run
method).
When one runs the simulation manually, one must initialize the estimators before running the simulation by calling initEstimators
method of the Experiment
instance.
We use the target feature to automatically infer the activation function for the last layer of the neural network and the loss function for training.
Feature | Last layer activation | Loss |
---|---|---|
Feature (default) |
identity | Mean squared error |
NumericFeature |
sigmoid (+ scaling to proper range) | Mean squared error |
CategoricalFeature |
softmax (1 neuron for each category) | Categorical cross-entropy |
BinaryFeature |
sigmoid (only 1 neuron) | Binary cross-entropy |
TimeFeature (used by TimeEstimate ) |
exponential | Poisson |
For role-assigned estimates, we compute the estimated values for all potential member components at the same time and cache them. It saves time as the neural network is capable of processing all the potential members in one batch. This implies that the inputs of the model can't use the information about the already selected members for this role.
In the examples
folder, one example project is located:
simple_example
– a simple example showing basic usage of the ML-DEECo framework.