kove: A Go repository from cmacrae

Watch your in cluster Kubernetes manifests for OPA policy violations and export them as Prometheus metrics.
Craft alerts & dashboards based on the structured data of the objects live in your environment(s).

About

Open Policy Agent provide the fearsome-but-trustworthy gatekeeper, which allows for admission control of Kubernetes manifests being submitted to the API. This is really nice and allows administrators to control the manifests coming in as fine-grained as they please.

However, administrators may not always want to take direct action (such as denial) on manifests arriving at the API. This is where kove comes in.
It allows administrators of Kubernetes clusters to define Rego policies that they want to flag violations for by exposing a Prometheus metric.

Some example use cases include monitoring the use of deprecated APIs, unwanted docker images, or container vars containing strings like API_KEY, etc.
Administrators can craft dashboards or alerts when such conditions are observed to better expose this information to users.

kove is built on an informer model, rather than admission control - so, it works on any existing objects in your cluster, instead of evaluating them when they arrive at the API (upon create/update). This means it'll expose policy violators that may otherwise go unnoticed if they're not updated often.

Example Implementations

kove-deprecations

A good example built on top of kove is the kove-deprecations Helm Chart.
It provides metrics for objects using APIs, annotations, and other such properties which are considered (or, soon to be) deprecated.

Simple alerts or Grafana dashboards can then be built for an overview:

This grants administrators automated visibility over the objects in their cluster that meet such criteria, which in turn allows for easier preparation of cluster upgrades and alignment with best practices.

Take a look at the policies/ directory (heavily based on the policies from swade1987/deprek8ion).

Metrics

Metric	Description
`opa_policy_violation`	Represents a Kubernetes object that violates the provided Rego expression. Includes the labels `name`, `namespace`, `kind`, `api_version`, `ruleset` and `data`
`opa_policy_violations_total`	Total number of policy violations observed
`opa_policy_violations_resolved_total`	Total number of policy violation resolutions observed
`opa_object_evaluations_total`	Total number object evaluations conducted

Usage

ConfigMap objects containing the Rego policy/policies and the application configuration can be mounted to configure what you want to evaluate and how you want to evaluate it.

Options

Option	Default	Description
`config`	`""`	Path to the config file. If not set, this will look for the file `config.yaml` in the current directory

`config`

A YAML manifest can be provided in the following format to describe how and what you want to watch for evaluation:

namespace: default
ignoreChildren: true
regoQuery: data.pkgname.blah
policies:
  - example/policies
objects:
  - group: apps
    version: v1
    resource: deployments
  - group: apps
    version: v1
    resource: daemonsets
  - group: apps
    version: v1
    resource: replicasets

Option	Default	Description
`namespace`	`""`	Kubernetes namespace to watch objects in. If empty or omitted, all namespaces will be observed
`ignoreChildren`	`false`	Boolean that decides if objects spawned as part of a user managed object (such as a ReplicaSet from a user managed Deployment) should be evaluated
`regoQuery`	`data[_].main`	The Rego query to read evaluation results from. This should match the expression in your policy that surfaces violation data
`policies`	none	A list of files/directories containing Rego policies to evaluate objects against
`objects`	none	A list of GroupVersionResource expressions to observe and evaluate. If empty all object kinds will be evaluated (apart from those defined in `ignoreKinds`)
`ignoreKinds`	`[` `apiservice` `endpoint` `endpoints` `endpointslice` `event` `flowschema` `lease` `limitrange` `namespace` `prioritylevelconfiguration` `replicationcontroller` `runtimeclass` `]`	A list of object kinds to ignore for evaluation
`ignoreDifferingPaths`	`[` `metadata/resourceVersion` `metadata/managedFields/0/time` `status/observedGeneration` `]`	A list of JSON paths to ignore for reevaluation when a change in the monitored object is observed

The above example configuration would instruct kove to monitor apps/v1/Deployment, apps/v1/DaemonSet, and apps/v1/ReplicaSet objects in the default namespace, but ignore child objects, yielding its results from the data.pkgname.blah expression in the provided policy.

`policies`

There are some important semantics to understand when crafting your Rego policies for use with kove.
The expression that you evaluate from your query must return structured data with the following fields:

Name: The name of the object being evaluated
Namespace: The namespace in which the object you're evaluating resides
Kind: The kind of object being evaluated
ApiVersion: The version of the Kubernetes API the object is using
RuleSet: A short description that describes why this is a violation
Data: Additional arbitrary data you wish to expose about the object

The above data are provided by kove when it evaluates an object, with the exception of RuleSet & Data which should be defined in the Rego expression. For instance, if we were to evaluate the query data.example.bad, our policy may look something like this:

package example

# Label matchers we want to look for.
labels["secure"] = ["nope"]

# Kinds of objects we care about evaluating.
# This isn't strictly necessary if you're satisfied with the 'objects' configuration
# option for kove; it'll only watch what it's told.
kinds = ["Deployment", "StatefulSet", "DaemonSet"]

bad[stuff] {
	# Assign our object manifest (input) to the variable 'r'
	r := input

	# Does our object's 'kind' field match any in our 'kinds' array?
	r.kind == kinds[_]

	# Does our object's 'secure' label match any in our 'labels.secure' array?
	r.metadata.labels.secure == labels.secure[_]

	# If the above conditions are true, express a set containing various pieces of
	# information about our object. As you can see, we're assigning this to the variable
	# 'stuff', which you may notice in the expression signature is what we're returning.
	# This information is then used to expose a Prometheus metric with labels using this
	# information.
	stuff := {
		"Name": r.metadata.name,
		"Namespace": r.metadata.namespace,
		"Kind": r.kind,
		"ApiVersion": r.apiVersion,
		"RuleSet": "Insecure object", # Explain why this is a violation
		"Data": r.metadata.annotations["something"] # Additional arbitrary information
	}
}

If you have a test cluster (perhaps built on kind), you can try out the evaluation of this policy against a violating Deployment.
Check out more examples/.

Deployment

A Helm Chart is available on Artifact HUB with accompanying implementation charts built on top of it, like kove-deprecations

cmacrae/kove