MConn

Notice:
This is a pre-release for friends and colleagues to check code styles, functions and usability. Currently, MConn undergoes an intensive development and testing. We will release our first version 0.1.0 in the next weeks. Please stay tuned!

MConn is an in Coffee-Script written HA Queue combined with a module management to work as an event-subscriber behind Mesosphere's Marathon to react on incoming task events and realize simple post processes (for Service-Discovery-Solutions) based on up/down scalings.

One of our targets has been a framework design that provides numerous custom post processes without wasting time on the individuell application of the Marathon-Handling, the HA and the fault-tolerance.

Features

Web UI
JSON/Web API
Queue
API to react on incoming task events
Inventory sync. (timeeven, MConn leader election, Marathon leader election or by apicall) to work fault-tolerance
Module-Management
HA
Zookeeper as datastore for states and module presets
Leader/Non-Leader election
request proxy to the current leader

For Developers

MConn is for free and based on the Apache v2-License. Don’t hesitate to fork and customize it! The first documentation for configuring Mconn is available here.

Demo-Deployment

Requirements:

Mesos (tested with 0.22.1)
Marathon (tested with 0.8.2)
Mesos-DNS (tested with 0.1.2)
Docker (tested with 1.6.2+)
Zookeeper (3.4.5+)

You have to assure that Mesos-DNS and each hostname can also be reached by Mesos-Slaves.

MConn's default enviroments for a fast deployment (inspired by DCOS) are:

Environment	Variable
MCONN_HOST	${HOST}
MCONN_PORT	${PORT0}
MCONN_MODULE_PATH	${MESOS_SANDBOX}
MCONN_MARATHON_HOSTS	leader.mesos:8080
MCONN_ZK_HOSTS	leader.mesos:2181
MCONN_ZK_PATH	${MARATHON_APP_ID}

You can modify this with the following Marathon-Config:

"env": {
    "MCONN_MARATHON_HOSTS": "admin:password@10.11.12.13:8080,admin:password@10.11.12.14:8080",
    "MCONN_MARATHON_SSL": "true",
    "MCONN_ZK_HOSTS": "10.11.12.13:2181,10.11.12.14:2181,10.11.12.15:2181"
}

More informations about the MConn-Enviroments, see the docs.

Deployment scheme:

1. Start MConn incl. the HelloWorld-Example:

The "uris"-paramenter in the following Marathon-Config will instruct Mesos to download the HelloWorkd-Example and extract it into the MESOS-SANDBOX:

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" \
    leader.mesos:8080/v2/apps -d '{
    "id": "mconn",
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "livespotting/mconn:0.0.10",
            "network": "HOST"
        },
        "volumes": []
    },
    "ports": [31999],
    "cpus": 1,
    "mem": 1024,
    "instances": 3,
    "healthChecks": [
        {
            "path": "/v1/ping",
            "portIndex": 0,
            "protocol": "HTTP",
            "gracePeriodSeconds": 30,
            "intervalSeconds": 30,
            "timeoutSeconds": 30,
            "maxConsecutiveFailures": 3
        }
    ],
    "uris": [
        "https://github.com/livespotting/mconn-helloworld/archive/master.zip"
    ],
    "env": {
        "MCONN_MODULE_START": "mconn-helloworld-master"
    },
    "upgradeStrategy": {
        "minimumHealthCapacity": 0.5,
        "maximumOverCapacity": 0.0
    },
    "constraints": [
        [
            "hostname",
            "UNIQUE"
        ]
    ]
}'

After that call up the MConn UI at mconn.marathon.mesos:31999 (or get it from the Marathon UI)

2. Add MConn as a Marathon Event-Subscriber:

To expand the "event_subscriber" on Marathon, start Marathon with "--event_subscriber http_callback" (more here) and afterwards install MConn as a HTTP-Endpoint:

curl -X POST leader.mesos:8080/v2/eventSubscriptions?callbackUrl=http://mconn.marathon.mesos:31999/v1/queue

3. Start a Demo-Application:

First you have to add a preset for the HelloWorld-Module instructing the modul what to do with our demo-app:

curl -H "Content-Type: application/json" -X POST -d '{"appId":"/bridged-webapp","moduleName":"HelloWorld","status":"enabled","options":{"actions":{"add":"Moin, Moin","remove":"Tschues"}}}' http://mconn.marathon.mesos:31999/v1/module/preset

The preset will generate a "Moin, Moin" if a task has got the status quo "TASK_RUNNING". It will generate a "Tschues", if the task is "TASK_FINISHED", "TASK_KILLED" or "TASK_FAILED".

Afterwards start the Demo-App to see how the MConn-Queue-Management will work:

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" \
    leader.mesos:8080/v2/apps -d '{
    "id": "bridged-webapp",
    "cmd": "python3 -m http.server 8080",
    "cpus": 0.5,
    "mem": 64,
    "instances": 4,
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "python:3",
            "network": "BRIDGE",
            "portMappings": [
                {
                    "containerPort": 8080,
                    "hostPort": 0,
                    "protocol": "tcp"
                },
                {
                    "containerPort": 161,
                    "hostPort": 0,
                    "protocol": "udp"
                }
            ]
        }
    },
    "healthChecks": [
        {
            "protocol": "HTTP",
            "portIndex": 0,
            "path": "/",
            "gracePeriodSeconds": 5,
            "intervalSeconds": 20,
            "maxConsecutiveFailures": 3
        }
    ]
}'

Or Mesos-Slave/Sandbox/Stdout:

[26-07-2015 18:36:35] [INFO] Task "bridged-webapp.8c6a13a9-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" created on queue (Module.HelloWorld)
[26-07-2015 18:36:35] [INFO] Task "bridged-webapp.8c6a3aba-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" created on queue (Module.HelloWorld)
[26-07-2015 18:36:35] [INFO] Starting worker for task "bridged-webapp.8c6a13a9-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" (Module.HelloWorld)
[26-07-2015 18:36:36] [INFO] Task "bridged-webapp.8c6a61cb-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" created on queue (Module.HelloWorld)
[26-07-2015 18:36:36] [INFO] Task "bridged-webapp.8c69ec98-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" created on queue (Module.HelloWorld)
[26-07-2015 18:36:36] [INFO] Moin, Moin "bridged-webapp.8c6a13a9-33b4-11e5-a556-2edad0a36318" (Module.HelloWorld)
[26-07-2015 18:36:36] [INFO] Task "bridged-webapp.8c6a13a9-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" state changed to "finished"! Task Queue is now "3" (Module.HelloWorld)
[26-07-2015 18:36:36] [INFO] Starting worker for task "bridged-webapp.8c6a3aba-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" (Module.HelloWorld)
[26-07-2015 18:36:37] [INFO] Moin, Moin "bridged-webapp.8c6a3aba-33b4-11e5-a556-2edad0a36318" (Module.HelloWorld)
[26-07-2015 18:36:37] [INFO] Task "bridged-webapp.8c6a3aba-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" state changed to "finished"! Task Queue is now "2" (Module.HelloWorld)
[26-07-2015 18:36:37] [INFO] Starting worker for task "bridged-webapp.8c6a61cb-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" (Module.HelloWorld)
[26-07-2015 18:36:38] [INFO] Moin, Moin "bridged-webapp.8c6a61cb-33b4-11e5-a556-2edad0a36318" (Module.HelloWorld)
[26-07-2015 18:36:38] [INFO] Task "bridged-webapp.8c6a61cb-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" state changed to "finished"! Task Queue is now "1" (Module.HelloWorld)
[26-07-2015 18:36:38] [INFO] Starting worker for task "bridged-webapp.8c69ec98-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" (Module.HelloWorld)
[26-07-2015 18:36:39] [INFO] Moin, Moin "bridged-webapp.8c69ec98-33b4-11e5-a556-2edad0a36318" (Module.HelloWorld)
[26-07-2015 18:36:39] [INFO] Task "bridged-webapp.8c69ec98-33b4-11e5-a556-2edad0a36318_TASK_RUNNING" state changed to "finished"! Task Queue is now "0" (Module.HelloWorld)

Author

License

Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/