Vertical Elasticity on Marathon and Chronos Mesos frameworks

Overview

The main objective of this work is implements a tool that executes a Dockerized Job in order to accomplish a give Quality of Service (QoS) for Apache Mesos Frameworks: Chronos: or Marathon. To get this, the tool modifies resource allocation for each application. Furthemore, the tool sends metrics to Monasca OpenStack.

There are two possibles scenarios:

Chronos. Guarantee that the individual executions of the application are completed in a given.
Marathon. Guarantee that a minimum share of the CPU time has been allocated to that application.

Components

Launcher

Launcher is the component in charge of the submission of the job. Jobs are specified using JavaScript Object Notation (JSON) format, and Launcher assigns each job an unique identifier (UUID). According to the scenario’s specification, Launcher will submit the job with a modified job specification. Once submitted, Launcher will send to the Supervisor relevant information for the monitoring and scaling the application (e.g. the application deadline).

Dependencies:

Python module requests

CLI parameters:

-j <job-file>. Required parameter. <job-file> contains the job specification in JSON format. This specitication must contain the correct fields for each framework (you can check this creating a job by framework REST API) and information about QoS (it is described below).
-m <credentials-Marathon>. Required parameter (if -c is not being used). <credentials-Marathon> contains the credentials for Marathon and it is described below.
-c <credentials-Chronos>. Required parameter (if -m is not being used). <credentials-Chronos> is like <credentials-Marathon>
-i <supervisor-url>. Required parameter. <supervisor-url> it is the URL of the Supervisor REST API (e.g. “http://10.0.0.2:30000”).

Other parameters used:

executor_path: absolute path of the component Executor in Mesos agent nodes.
checkpoint_dir: shared path between Mesos agent nodes where checkpoints will be stored.
taskKillGracePeriodSeconds: quantity of seconds before Marathon executor remove the job once termination signal is received.

These parameters are in the main of Launcher.py.

Supervisor

Supervisor is a REST service that receives the information sent by Launcher and the containers. According to the scenario’s specification, it monitors the application in order to scale up and down the resources, according to an agreed quality of service. Furthermore, Supervisor sends metrics to Monasca for its visualization.

Dependencies:

Python module requests
Python module Flask

CLI parameters:

-m <credentials-Marathon>. Required parameter. <credentials-Marathon> contains the credentials for Marathon and it is described below.
-c <credentials-Chronos>. Required parameter. <credentials-Chronos> is like <credentials-Marathon>
-o <credentials-OpenStack>. Required parameter. <credentials-OpenStack> contains the credentials for KeyStone and the URL of Monasca. The file is described below.
-i <rest-ip>. Optional parameter. Default value is “0.0.0.0”.
-p <rest-port>. Optional parameter. Default value is “30000”.

Other parameters used:

logDirectory: absolute path of the directory where Supervisor stores their log.
cpu_max_slave: maximum shared CPU time available in Mesos agent node.
cpu_min_slave: minimum shared CPU time available in Mesos agent node.
monitoring_period: Monitoring period of Marathon applications by Supervisor.
cpu_increment
cpu_decrement

These parameters in Supervisor.py.

QoS definition

QoS is defined in the job specification file. Three parameters are needed:

duration: expected duration of one iteration of the application in seconds.
desv_deadline: maximum overprogress percentage.
deadline: interval of time given to the system for complete the job.

{
        "qos" : {
                "duration": 600,
                "desv_deadline": 0.8,
                "deadline": 350

        }
}

Credential files

The credential files required for the correct performance of the modules described above are:

Chronos and Marathon

{
        "url" : "http://IP:PORT",
        "user" : "USER",
        "password" : "PASSWD"
}

Monasca

{
        "keystone_url" : "http://IP:PORT",
        "monocasca_client_url" : "http://IP:PORT",
        "username" : "USER",
        "password" : "PASSWD",
        "project_name" : "myprojectname"
}

eubr-bigsea/vertical_elasticity