Keptn

This is the primary repository for the Keptn software and documentation. Keptn provides a “cloud-native” approach for managing the application release lifecycle metrics, observability, health checks, with pre- and post-deployment evaluations and tasks. It is an incubating project, under the umbrella of the Keptn Application Lifecycle working group.

Note Keptn was developed under the code name of "Keptn Lifecycle Toolkit" or "KLT" for short. The source code contains many vestiges of these names.

Goals

Keptn provides Cloud Native teams with the following capabilities:

Pre-requisite evaluation before deploying workloads and applications
Finding out when an application (not just a workload) is ready and working
Checking the Application Health in a declarative (cloud-native) way
Standardized way to run pre- and post-deployment tasks
Provide out-of-the-box Observability of the deployment cycle

Keptn can be seen as a general purpose and declarative Level 3 operator for your Application. For this reason, Keptn is agnostic to deployment tools that are used and works with any GitOps solution.

Status

Status of the different features:

Observability: expose OTel metrics and traces of your deployment.
K8s Custom Metrics: expose your Observability platform via the Custom Metric API.
Release lifecycle: handle pre- and post-checks of your Application deployment.
Certificate Manager: automatically configure TLS certificates with the Keptn Certificate Manager. You can instead configure your own certificate manager to provide secure communication with the Kube API.

The status follows the Kubernetes API versioning schema.

Installation

Keptn can be installed on any Kubernetes cluster running Kubernetes >=1.24.

For users running vCluster, please note that you may need to modify your configuration before installing Keptn; see Running Keptn with vCluster for more information.

Use the following command sequence to install the latest release of Keptn:

helm repo add keptn https://charts.lifecycle.keptn.sh
helm repo update
helm upgrade --install keptn keptn/keptn -n keptn-lifecycle-toolkit-system --create-namespace --wait

Installation with only certain namespaces allowed

Keptn lifecycle orchestration is by default enabled for all namespaces except the following ones:

kube-system
kube-public
kube-node-lease
cert-manager
keptn-lifecycle-toolkit-system
observability
monitoring
<Keptn installation namespace>

To restrict Keptn lifecycle orchestration to specific namespaces, you must specify those namespaces during installation via helm values. First you need to create a values.yaml file

lifecycleOperator:
  allowedNamespaces:
  - allowed-ns-1
  - allowed-ns-2

and add the values file to the helm installation command:

helm repo add keptn https://charts.lifecycle.keptn.sh
helm repo update
helm upgrade --install keptn keptn/keptn -n keptn-lifecycle-toolkit-system --values values.yaml --create-namespace --wait

Note Please be aware that you still need to correctly annotate the namespaces where Keptn lifecycle orchestration is allowed. To annotate them, use:

kubectl annotate ns <your-allowed-namespace> keptn.sh/lifecycle-toolkit='enabled'

Note Please be aware that, if this option is set, adding any additional namespace requires the helm installation to be updated by adding the name of the new namespace to the list.

Installation without scheduler

Keptn installed on Kubernetes cluster running Kubernetes >= 1.26 does not need scheduler for a proper functionality. With introduction of Pod scheduling gates Keptn can use this feature to substitute the functionality of Keptn scheduler.

As this functionality is still disabled by default, it can be enabled by setting up the correct helm values. This will lead to a Keptn installation without a scheduler and with scheduling gates feature enabled.

Use the following command sequence to install Keptn with scheduling gates enabled:

helm repo add keptn https://charts.lifecycle.keptn.sh
helm repo update
helm upgrade --install keptn keptn/keptn -n keptn-lifecycle-toolkit-system --set schedulingGatesEnabled=true --create-namespace --wait

Note Please be aware that scheduling gates functionality in Kubernetes is enabled by default in Kubernetes >= 1.27. To use it with Kubernetes 1.26, you need to enable it on your cluster.

More information

For more info about Keptn, please see our documentation, specifically:

Introduction to Keptn gives an overview of the Keptn facilities.
Getting started includes some short exercises to introduce you to Keptn.
Installation and upgrade provides information about preparing your Kubernetes cluster then installing and enabling Keptn.
Implementing Keptn applications documents how to integrate Keptn to work with your existing deployment engine and implement its various features.
Architecture provides detailed technical information about how Keptn works.
CRD Reference and API Reference provide detailed reference material for the custom resources used to configure Keptn.
Contributing to Keptn provides information about how to contribute to the Keptn project.

You can also find a number of video presentations and demos about Keptn on the YouTube Keptn channel. Videos that refer to the "Keptn Lifecycle Controller" are relevant for the Keptn project.

Architecture

Keptn is composed of the following components:

Keptn Lifecycle Operator
Keptn Scheduler

The Keptn Lifecycle Operator contains several controllers for Keptn CRDs and a Mutating Webhook. The Keptn Scheduler ensures that Pods are started only after the pre-deployment checks have finished successfully.

A Kubernetes Manifest. which is annotated with Keptn specific annotations, is applied to the Kubernetes Cluster. Afterward, the Keptn Scheduler is injected (via Mutating Webhook), and Kubernetes Events for Pre-Deployment are sent to the event stream. The Event Controller watches for events and triggers a Kubernetes Job to fullfil the Pre-Deployment. After the Pre-Deployment has finished, the Keptn Scheduler schedules the Pod to be deployed. The KeptnApp and KeptnWorkload Controllers watch for the workload resources to finish and then generate a Post-Deployment Event. After the Post-Deployment checks, SLOs can be validated using an interface for retrieving SLI data from a provider e.g, Prometheus. Finally, Keptn exposes Metrics and Traces of the entire Deployment cycle with OpenTelemetry.

Webhook

Annotating a namespace subjects it to the effects of the mutating webhook:

apiVersion: v1
kind: Namespace
metadata:
  name: podtato-kubectl
  annotations:
    keptn.sh/lifecycle-toolkit: "enabled"  # this line tells the webhook to handle the namespace

The mutating webhook only modifies specifically annotated resources in the annotated namespace. When the webhook receives a request for a new pod, it looks for the workload annotations:

keptn.sh/workload: "some-workload-name"

The mutation consists in changing the scheduler used for the deployment with the Keptn Scheduler. The webhook then creates a workload and app resource per annotated resource. You can also specify a custom app definition with the annotation:

keptn.sh/app: "your-app-name"

In this case the webhook does not generate an app, but it expects that the user will provide one. Additionally, it computes a version string, using a hash function that takes certain properties of the pod as parameters (e.g. the images of its containers). Next, it looks for an existing instance of a Workload CRD for the specified workload name:

If it finds the Workload, it updates its version according to the previously computed version string. In addition, it includes a reference to the ReplicaSet UID of the pod (i.e. the Pods owner), or the pod itself, if it does not have an owner.
If it does not find a workload instance, it creates one containing the previously computed version string. In addition, it includes a reference to the ReplicaSet UID of the pod (i.e. the Pods owner), or the pod itself, if it does not have an owner.

It uses the following annotations for the specification of the pre/post deployment checks that should be executed for the Workload:

keptn.sh/pre-deployment-tasks: task1,task2
keptn.sh/post-deployment-tasks: task1,task2

and for the Evaluations:

keptn.sh/pre-deployment-evaluations: my-evaluation-definition
keptn.sh/post-deployment-evaluations: my-eval-definition

After either one of those actions has been taken, the webhook sets the scheduler of the pod and allows the pod to be scheduled.

Scheduler

After the Webhook mutation, the Keptn-Scheduler handles the annotated resources. The scheduling flow follows the default scheduler behavior, since it implements a scheduler plugin based on the scheduling framework. For each pod, at the very end of the scheduling cycle, the plugin verifies that the pre deployment checks have terminated by retrieving the current status of the WorkloadInstance. Only when that is successful is the pod bound to a node.

Install a dev build

The GitHub CLI can be used to download the manifests of the latest CI build.

gh run list --repo keptn/lifecycle-toolkit # find the id of a run
gh run download 3152895000 --repo keptn/lifecycle-toolkit # download the artifacts
kubectl apply -f ./keptn-lifecycle-operator-manifest/release.yaml # install the lifecycle-operator
kubectl apply -f ./scheduler-manifest/release.yaml # install the scheduler

Instead, if you want to build and deploy the operator into your cluster directly from the code, you can type:

RELEASE_REGISTRY=<YOUR_DOCKER_REGISTRY>
# (optional)ARCH=<amd64(default)|arm64v8>
# (optional)CHART_APPVERSION=<YOUR_PREFERRED_TAG (defaulting to current time)>

# Build and deploy the dev images to the current kubernetes cluster
make build-deploy-dev-environment

License

Please find more information in the LICENSE file.

Thanks to all the people who have contributed 💜

Made with contrib.rocks.

knutgoetz/lifecycle-toolkit