When Google Kubernetes meets Apache Mesos
![GoDoc] (https://godoc.org/github.com/mesosphere/kubernetes-mesos?status.png)
Kubernetes and Mesos are a match made in heaven. Kubernetes enables the Pod (group of co-located containers) abstraction, along with Pod labels for service discovery, load-balancing, and replication control. Mesos provides the fine-grained resource allocations for pods across nodes in a cluster, and can make Kubernetes play nicely with other frameworks running on the same cluster resources.
Within the Kubernetes framework for Mesos the framework scheduler first registers with Mesos and begins accepting API requests. At some point Mesos offers the scheduler sets of available resources from the cluster nodes (slaves/minions). Once the scheduler is able to match a Mesos resource offer to an unassigned Kubernetes pod it binds the pod to the slave, first sending a launchTasks message to the Mesos master, and then updating the pod's state (desired host). Mesos marks the resources as claimed and then forwards the request onto the appropriate slave. The slave fetches the kubelet/executor and starts running it, spawning the pod's containers and communicating the pod/task status back to Mesos.
This is still very much a work-in-progress, but stay tuned for updates as we continue development. If you have ideas or patches, feel free to contribute!
- Launching pods (on local machine)
- Implement Kube-scheduler API
- Pick a Pod (FCFS), match it to an offer.
- Launch it!
- Kubelet as Executor+Containerizer
- Pod Labels: for Service Discovery + Load Balancing
- Running multi-node on GCE
- Replication Control
- Use resource shapes to schedule pods
- Even smarter (Marathon-like) scheduling
For a binary-only install of the Kubernetes-Mesos framework you can use the Docker-based builder:
# chcon needed for systems protected by SELinux
$ mkdir bin && chcon -Rt svirt_sandbox_file_t bin
$ docker run -rm -v $(pwd)/bin:/target jdef/kubernetes-mesos:build-latestInstructions to build and install from source are as follows:
NOTE: Building Kubernetes for Mesos requires Go 1.2+, protobuf 2.5.0, and Mesos 0.19+. Building the project is greatly simplified by using godep.
- To install Mesos, see mesosphere.io/downloads
- To install godep, see github.com/tools/godep
- See the development page for sample environment setup steps.
Once the prerequisites have been installed you can build the project:
$ cd $GOPATH # If you don't have one, create directory and set GOPATH accordingly.
$ mkdir -p src/github.com/mesosphere/kubernetes-mesos
$ git clone https://github.com/mesosphere/kubernetes-mesos.git src/github.com/mesosphere/kubernetes-mesos
$ cd src/github.com/mesosphere/kubernetes-mesos && godep restore
$ go install github.com/GoogleCloudPlatform/kubernetes/cmd/{proxy,kubecfg}
$ go install github.com/mesosphere/kubernetes-mesos/kubernetes-{mesos,executor}
$ go install github.com/mesosphere/kubernetes-mesos/controller-managerThe examples that follow assume that you are running the mesos-master, etcd, and the kubernetes-mesos framework on the same host, exposed on an IP address referred to hereafter as ${servicehost}.
If you are not running in a production setting then a single etcd instance will suffice.
To run etcd, see github.com/coreos/etcd, or run it via docker:
$ export servicehost=... # IP address of the framework host
$ sudo docker run -d --net=host coreos/etcd go-wrapper run \
-advertise-client-urls=http://${servicehost}:4001 \
-listen-client-urls=http://${servicehost}:4001 \
-initial-advertise-peer-urls=http://${servicehost}:7001 \
-listen-peer-urls=http://${servicehost}:7001Assuming your mesos cluster is started, and that the mesos-master and etcd are running on ${servicehost}, then:
$ ./bin/kubernetes-mesos \
-address=${servicehost} \
-mesos_master=${servicehost}:5050 \
-etcd_servers=http://${servicehost}:4001 \
-executor_path=$(pwd)/bin/kubernetes-executor \
-proxy_path=$(pwd)/bin/proxyFor simpler execution of kubecfg:
$ export KUBERNETES_MASTER=http://${servicehost}:8888To enable replication control, start a kubernetes replication controller instance:
$ ./bin/controller-manager -master=${KUBERNETES_MASTER#http://*}You can increase logging for both the framework and the controller by including, for example, -v=2.
This can be very helpful while debugging.
###Launch a Pod
Assuming your framework is running on ${KUBERNETES_MASTER}, then:
$ bin/kubecfg -c examples/pod-nginx.json create pods
# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/pods -XPOST -d @examples/pod-nginx.jsonAfter the pod get launched, you can check it's status via kubecfg, curl or your web browser:
$ bin/kubecfg list pods
ID Image(s) Host Labels Status
---------- ---------- ---------- ---------- ----------
nginx-id-01 dockerfile/nginx 10.132.189.242/10.132.189.242 name=foo Running
# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/pods{
"kind": "PodList",
"creationTimestamp": null,
"apiVersion": "v1beta1",
"items": [
{
"id": "nginx-id-01",
"creationTimestamp": "2014-10-17T02:46:00Z",
"labels": {
"name": "foo"
},
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "nginx-id-01",
"uuid": "ba48b2b2-55a7-11e4-8ec3-08002766f5aa",
"volumes": null,
"containers": [
{
"name": "nginx-01",
"image": "dockerfile/nginx",
"ports": [
{
"hostPort": 31000,
"containerPort": 80,
"protocol": "TCP"
}
],
"livenessProbe": {
"type": "http",
"httpGet": {
"path": "/index.html",
"port": "8081"
},
"initialDelaySeconds": 30
}
}
],
"restartPolicy": {
"always": {}
}
},
"status": "Running"
},
"currentState": {
"manifest": {
"version": "v1beta1",
"id": "nginx-id-01",
"uuid": "ba48b2b2-55a7-11e4-8ec3-08002766f5aa",
"volumes": null,
"containers": [
{
"name": "nginx-01",
"image": "dockerfile/nginx",
"ports": [
{
"hostPort": 31000,
"containerPort": 80,
"protocol": "TCP"
}
],
"livenessProbe": {
"type": "http",
"httpGet": {
"path": "/index.html",
"port": "8081"
},
"initialDelaySeconds": 30
}
}
],
"restartPolicy": {
"always": {}
}
},
"status": "Waiting",
"info": {
"net": {
"Id": "6c4dd34a8d213bc276103985baee053e49cec5236706b7d56acf24441e0f975d",
"Created": "2014-10-17T02:46:01.923547392Z",
"Path": "/pause",
"Config": {
"Hostname": "nginx-id-01",
"ExposedPorts": {
"80/tcp": {}
},
"Env": [
"HOME=/",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Image": "kubernetes/pause:latest",
"Entrypoint": [
"/pause"
]
},
"State": {
"Running": true,
"Pid": 13594,
"StartedAt": "2014-10-17T02:46:02.005492006Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "6c4579af347b649857e915521132f15a06186d73faa62145e3eeeb6be0e97c27",
"NetworkSettings": {
"IPAddress": "172.17.0.13",
"IPPrefixLen": 16,
"Gateway": "172.17.42.1",
"Bridge": "docker0",
"Ports": {
"80/tcp": [
{
"HostIP": "0.0.0.0",
"HostPort": "31000"
}
]
}
},
"ResolvConfPath": "/etc/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/6c4dd34a8d213bc276103985baee053e49cec5236706b7d56acf24441e0f975d/hostname",
"HostsPath": "/var/lib/docker/containers/6c4dd34a8d213bc276103985baee053e49cec5236706b7d56acf24441e0f975d/hosts",
"Name": "/k8s--net.fa4b7d08--nginx_-_id_-_01.etcd--ba48b2b2_-_55a7_-_11e4_-_8ec3_-_08002766f5aa--9acb0442",
"Driver": "aufs",
"HostConfig": {
"PortBindings": {
"80/tcp": [
{
"HostIP": "0.0.0.0",
"HostPort": "31000"
}
]
},
"RestartPolicy": {}
}
}
}
}
}
]
}Or, you can run docker ps on the appropriate Mesos slave to verify that the example container is running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ca87f2981e6d dockerfile/nginx:latest "nginx" 30 seconds ago Up 30 seconds k8s--nginx_-_01.e7078cc4--nginx_-_id_-_01.mesos--ab87fcc4_-_668e_-_11e4_-_bd1f_-_04012f416701--83e4f98d
78e9f83ed4a9 kubernetes/pause:go "/pause" 5 minutes ago Up 5 minutes 0.0.0.0:31000->80/tcp k8s--net.fa4b7d08--nginx_-_id_-_01.mesos--ab87fcc4_-_668e_-_11e4_-_bd1f_-_04012f416701--aa209b8e###Launch a Replication Controller
Assuming your framework is running on ${KUBERNETES_MASTER} and that you have multiple mesos slaves in your cluster, then:
$ bin/kubecfg -c examples/controller-nginx.json create replicationControllers
# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/replicationControllers -XPOST -d@examples/controller-nginx.jsonAfter the pod get launched, you can check it's status via kubecfg, curl or your web browser:
$ bin/kubecfg list replicationControllers
ID Image(s) Selector Replicas
---------- ---------- ---------- ----------
nginxController dockerfile/nginx name=nginx 2
# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/replicationControllers{
"kind": "ReplicationControllerList",
"creationTimestamp": null,
"resourceVersion": 7,
"apiVersion": "v1beta1",
"items": [
{
"id": "nginxController",
"creationTimestamp": "2014-10-17T11:10:29Z",
"resourceVersion": 3,
"desiredState": {
"replicas": 2,
"replicaSelector": {
"name": "nginx"
},
"podTemplate": {
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "",
"volumes": null,
"containers": [
{
"name": "nginx",
"image": "dockerfile/nginx",
"ports": [
{
"hostPort": 31001,
"containerPort": 80,
"protocol": "TCP"
}
]
}
],
"restartPolicy": {
"always": {}
}
}
},
"labels": {
"name": "nginx"
}
}
},
"currentState": {
"replicas": 2,
"podTemplate": {
"desiredState": {
"manifest": {
"version": "",
"id": "",
"volumes": null,
"containers": null,
"restartPolicy": {}
}
}
}
},
"labels": {
"name": "nginx"
}
}
]
}Run test suite with:
$ go test github.com/mesosphere/kubernetes-mesos/kubernetes-mesos -v