A BOSH release for Kubernetes. Formerly named kubo.
- Slack: #cfcr on https://slack.cloudfoundry.org
- Pivotal Tracker: https://www.pivotaltracker.com/n/projects/2093412
- Tests
- Prerequisites
- Deploying CFCR
- Accessing the CFCR Cluster with kubectl
- Backup & Restore
- Monitoring
- DNS
- Deprecations
To support Kubernetes terminology changes and Community Values, master in CFCR documentation and property names has been changed to control plane and repo branches have been renamed from master to main.
To run tests, run these commands:
bundle install
gem install rspec
rspec
-
A BOSH Director configured with UAA, Credhub, and BOSH DNS runtime config. We recommend using BOSH Bootloader for this.
-
Accessing the control plane:
- Single Control Plane Node: Set up a DNS name pointing to your control plane's IP address
- Multiple Control Plane Nodes: A TCP load balancer for your control plane nodes.
- Use a TCP load balancer configured to connect to the master nodes on port 8443.
- Add healthchecks using either a TCP dial or HTTPS by looking for a
200 OK
response from/healthz
. - if you have used BOSH Bootloader on GCP then you need to manually create a firewall rule. Allow access to port TCP 8443 to VMs in your BBL network tagged
cfcr-master
from your load balancer's IP.
-
Cloud Config with
vm_types
namedminimal
,small
, andsmall-highmem
(See cf-deployment for reference)network
nameddefault
- three availability zones
azs
namedz1
,z2
,z3
Note: the cloud-config properties can be customized by applying ops-files. See
manifests/ops-files
for some examples.If using loadbalancers then apply the
vm_extension
calledcfcr-master-loadbalancer
to the cloud-config to add the instances to your loadbalancers. See BOSH documentation for information on how to configure loadbalancers.
Kubernetes uses etcd as its datastore. The official infrastructure requirements and example configurations for the etcd cluster can be found here.
-
Upload the latest Xenial stemcell to the director.
-
Untar the kubo-deployment tarball and rename it
kubo-deployment
-
Deploy
cd kubo-deployment bosh deploy -d cfcr manifests/cfcr.yml \ -o manifests/ops-files/misc/single-master.yml \ -o manifests/ops-files/add-hostname-to-master-certificate.yml \ -v api-hostname=[DNS-NAME]
cd kubo-deployment bosh deploy -d cfcr manifests/cfcr.yml \ -o manifests/ops-files/add-vm-extensions-to-master.yml \ -o manifests/ops-files/add-hostname-to-master-certificate.yml \ -v api-hostname=[LOADBALANCER-ADDRESS]
Note: Loadbalancer address should be the external address (hostname or IP) of the loadbalancer you have configured.
Check additional configurations, such as setting Kubernetes cloud provider, in docs.
-
Add Kubernetes system components
bosh -d cfcr run-errand apply-specs
-
Run the following to confirm the cluster is operational
bosh -d cfcr run-errand smoke-tests
Please check out our manifest and ops-files in kube-deployment for examples on how to configure kubo-release. Additionally, we have a doc page to describe how to configure Kubernetes components for the release.
CFCR can be deployed with Pod Security Policies. Check for more details in the doc
CFCR allows you to configure proxy for all components. Check recommendations for no proxy settings first.
CFCR clusters on BOSH Lite are intended for development. We run the deploy_cfcr_lite script to provision a cluster with the latest stemcell and master of kubo-release. This requires that the cloned kubo-release repository can be found from cd ../kubo-release
from within the kubo-deployment directory.
cd kubo-deployment
./bin/deploy_cfcr_lite
- Login to the Credhub Server that stores the cluster's credentials:
credhub login
- Find the director name by running
bosh env
- Configure the
kubeconfig
for yourkubectl
client:cd kubo-deployment ./bin/set_kubeconfig <DIRECTOR_NAME>/cfcr https://[DNS-NAME-OR-LOADBALANCER-ADDRESS]:8443
We use BBR to perform backups and restores of the etcd node within a CFCR cluster, for both single and three master deployments. Our backup currently takes an etcd snapshot without interruptions to the cluster. However, for restore we take both the kube-apiserver and etcd offline to restore the cluster with the specified snapshot. Restore is a destructive operation that will completely overwrite any existing data on the cluster. For a closer look at the bbr scripts, check out:
To run the bbr
cli against a CFCR cluster, follow the steps under "BOSH Deployment" on the BBR documentation page.
Follow the recommendations in etcd's documentation for monitoring etcd metrics.
By default CFCR runs with CoreDNS in preference of Kube-DNS.
If you are migrating from an earlier version of CFCR, Kube-DNS can be removed by running:
kubectl delete deployment -n kube-system kube-dns
You may notice that a kube-dns
service remains, this is also required by the CoreDNS spec.
CFCR had a set of scripts, including deploy_bosh
and deploy_k8s
, that were the primary mechanism we supported to deploy BOSH and Kubernetes clusters. We no longer support these and have removed the corresponding documentation from https://docs-cfcr.cfapps.io
The BOSH oriented method documented in this README.md is the supported method to deploy Kubernetes clusters with CFCR.