This project is an experiment to see if open source Kubernetes and supporting software from CoreOS (e.g. flannel) can be deployed and managed in a production-suitable manner, not just hacked together.
It is based on the Manual Installation Guide for deploying Kubernetes on CoreOS, but does not use CoreOS.
This is not presently a production-ready release. This is a work in progress. It is suitable for experimentation and may become unsupported in the future.
This goal of this project is to package up Kubernetes as a BOSH release, to be deployed in any configuration, on several possible supported IaaSes by a BOSH director.
BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems.
A BOSH release is a versioned collection of configuration properties, configuration templates, start up scripts, source code, binary artifacts, and anything else required to build and deploy software in a reproducible way.
The BOSH Director is deployed to orchestrate the deployments of other software.
For more on BOSH, visit bosh.io and check out the docs, especially What is BOSH? and What Problems Does BOSH Solve?
- Deploy a BOSH Director.
- For simple hacking, the easiest thing to do is
vagrant upa BOSH-Lite which deploys to Linux containers instead of IaaS VMs. - For a real deployment, use
bosh-init.
- Get the
boshCLI and target your Director. - Create a BOSH deployment manifest describing your desired deployment topology (e.g. number of etcd nodes, Kubernetes master nodes, and Kubernetes worker nodes); your manifest should reference this release, etcd release and an IaaS-appropriate BOSH stemcell.
- If deploying to BOSH-Lite, you can mostly crib off the BOSH-Lite example manifest.
- Run
bosh -d ${PATH_TO_MANIFEST} deploy. - Deploy the DNS Add-On Service:
bosh -d ${PATH_TO_MANIFEST} run errand deploy_dns_add_on.
You should be able to target your Kubernetes master with the kubectl CLI. It should be serving HTTPS traffic on port 443 of the master host, see here for details on TLS configuration of the CLI. The CA, certificate, and key mentioned in that document are the same ones used by the DNS Add-On errand, which uses kubectl to deploy the DNS service itself. If you bosh ssh onto the master host, you can also target localhost:8080 to reach the API without any TLS configuration.
Fun fact: If you use BOSH-Lite, you'll have a Vagrant VM, with nodes in the Kubernetes clusters running as Garden Linux containers inside the VM, and those nodes will be running Docker inside the Linux containers, which will then run your pods inside Docker containers!
You can deploy the example guestbook application as a BOSH errand. The simplest way to do this is to add the errand for it to your manifest (see the example BOSH Lite manifest for an example) and simply:
bosh -d ${PATH_TO_MANIFEST} run errand deploy_example_guestbook
It should be serving on port 30080 on all worker hosts.
Example Guestbook Application Not Working:
curl $WORKER_IP tends to give the error:
curl: (56) Recv failure: Connection reset by peer
Highly-Available Master Deployments:
This is not supported at this time. It is not known whether this scenario is well-supported within Kubernetes itself. See here:
Also, at this time high availability support for Kubernetes is not continuously tested in our end-to-end (e2e) testing.
"Error updating node status" in kubelet_master logs:
In kubelet_master error logs there appears to be a lot of the following type of output:
[2015-11-29 05:09:20+0000] E1129 05:09:20.092267 4947 kubelet.go:2284] Error updating node status, will retry: error getting node 10.244.5.2: node 10.244.5.2 not found
[2015-11-29 05:09:20+0000] E1129 05:09:20.093377 4947 kubelet.go:2284] Error updating node status, will retry: error getting node 10.244.5.2: node 10.244.5.2 not found
[2015-11-29 05:09:20+0000] E1129 05:09:20.094310 4947 kubelet.go:2284] Error updating node status, will retry: error getting node 10.244.5.2: node 10.244.5.2 not found
[2015-11-29 05:09:20+0000] E1129 05:09:20.095560 4947 kubelet.go:2284] Error updating node status, will retry: error getting node 10.244.5.2: node 10.244.5.2 not found
[2015-11-29 05:09:20+0000] E1129 05:09:20.096722 4947 kubelet.go:2284] Error updating node status, will retry: error getting node 10.244.5.2: node 10.244.5.2 not found
[2015-11-29 05:09:20+0000] E1129 05:09:20.096742 4947 kubelet.go:918] Unable to update node status: update node status exceeds retry count
This is likely a manifestation of the issue mentioned here:
Note that the kubelet running on a master node may log repeated attempts to post its status to the API server. These warnings are expected behavior and can be ignored.
"Failed to acquire subnet: out of subnets" in flannel logs:
It's not clear if this is actually breaking anything, but simply providing a pod_network larger than /23 should work around any potential issue.
Docker puts stuff in /var/run/docker/netns and /var/lib/docker/network:
There doesn't seem to be a way to tell the daemon command to use different directories, like you can for --graph and --execroot.
See daemon command documentation.
Everyone is encouraged to help improve this project. Here are some ways you can contribute:
- use it,
- suggest new features,
- report new issues or comment on existing ones, or
- submit pull requests or comment on existing ones; pull requests can include:
- writing or editing documentation
- writing unit, functional, or system tests
- writing code, even small things such as fixing typos, adding comments, cleaning up inconsistent whitespace, etc.
- refactoring code
When submitting a bug report, please include a Gist that includes a stack trace and any details that may be necessary to reproduce the bug, including your BOSH CLI version, BOSH Director version, BOSH stemcell version, etcd release version, and version of this release.
- Fork the project.
- Create a topic branch.
- Implement your feature or bug fix.
- Commit and push your changes.
- Submit a pull request.