ClarkSource/k8t

Kubernetes manifest templating for multi cluster and environment deployments with code scaffolding.

k8t

Pronounced katie [ˈkeɪti]

Simple cluster and environment specific aware templating for kubernetes manifests.

Table of Contents generated with DocToc

• Installation
  • Docker
  • Completion
• Concepts
  • Clusters and Environments
  • Templating
    • Template helper functions
• Configuration inheritance
• Usage
  • Scaffolding
  • Config management
  • Validate templates
    • Shortcomings
      • is defined
  • Generate manifests
  • Overriding templates
  • Managing secrets
    • Providers
      • SSM
        • role assumption
      • Random
      • Hash
  • Using as a pre-commit hook
• TODO

Installation

run this

$ pip install --user --upgrade k8t

run the following to install ujson as a dependency

$ pip install --user --upgrade k8t[ujson]

note: k8t is not Python 2 compatible

Docker

You can also run k8t via docker

$ docker run clarksource/k8t:latest

hint: the docker image comes with aws-cli, and kubectl.

Completion

Run the following and store the file in your distribution/OS specific spot

bash:

$ _K8T_COMPLETE=source k8t > k8t-completion.sh

zsh:

$ _K8T_COMPLETE=source_zsh k8t > k8t-completion.sh

Concepts

By combining those concepts you can quickly add completely new environments to your deployment pipeline just by modifying specializing values and sharing the rest.

Check out our examples here.

Clusters and Environments

k8t comes with a builtin framework for clusters and environments (e.g. production, staging). This came from the need to be able to deploy the same application over multiple clusters and in different environments with completely different setups and values. This idea is helped by the fact that k8t deep-merges values and configs, allowing easy variation through different stages of your application deployment.

Both clusters and environments are intentionally working the same way and can be used to add another degree of freedom when combined. Environments however are also available globally, meaning clusters can share environment specific configuration while specifying differences in those environments.

Templating

Templating is supported via Jinja. k8t also comes with some additional helper functions and a validation function with verbose output to quickly verify the written templates.

Template helper functions

• random_password(N: int) - generate a random string of length N
• envvar(key: str, [default]) - get a value from any environment variable with optional default
• b64encode(value: str) - encodes a value in base64 (usually required for secrets)
• b64decode(value: str) - decodes a value from base64
• hash(value: str, [method: str]) - hashes a given value (default using sha256)
• get_secret(key: str) - provides a secret value from a given provider (see here)
• bool(value: Any) - casts value to boolean ("true", "on", "yes", "1", 1 are considered as True)
• sanitize_label(value: str) - sanitizes label values according to kubernetes spec
• sanitize_cpu(value: str | int) - sanitize cpu value to millicores
• sanitize_memory(value: str | int) - sanitize memory value to megabyte (note: values in scientific notation need to be converted to strings)
• standardize_cpu(value: str | int) - standardize cpu value to millicores (as int)
• standardize_memory(value: str | int) - standardize memory value to megabyte (as int; note: values in scientific notation need to be converted to strings)

Configuration inheritance

Configuration, values and templates are used according to the scope they are in. The following snippet shows an example project with low scores (1) and high scores (4) for evaluation order.

So variables and templates will be overridden from project -> environments -> clusters -> cluster-environments resulting in more specific configuration overriding lower values.

.                                           (1) # k8t new project .
├── clusters
│   ├── foo                                 (3) # k8t new cluster foo
│   │   ├── config.yaml
│   │   ├── values.yaml
│   │   ├── environments
│   │   │    ├── production                 (4) # k8t new environment production -c foo
│   │   │    │   ├── config.yaml
│   │   │    │   └── values.yaml
│   │   │    └── staging                    (4) # k8t new environment staging -c foo
│   │   │        ├── config.yaml
│   │   │        ├── values.yaml
│   │   │        └── templates
│   │   │           └── deployment.yaml.j2  (4) # k8t new template deployment -c foo -e staging
│   │   └── templates
│   │      └── deployment.yaml.j2           (3) # k8t new template deployment -c foo
│   └── bar                                 (3) # k8t new cluster bar
│       ├── config.yaml
│       └── values.yaml
├── environments
│   ├── production                          (2) # k8t new environment production
│   │   ├── config.yaml
│   │   └── values.yaml
│   └── staging                             (2) # k8t new environment staging
│       ├── config.yaml
│       └── values.yaml
├── config.yaml                             (1)
└── values.yaml                             (1)

Usage

Scaffolding

Create a new project folder with a cluster directory and an empty defaults file

$ k8t new project .

Create a new cluster

$ k8t new cluster MyCluster

Create a new global environment

$ k8t new environment staging

And a new cluster environment

k8t new environment staging -c MyCluster

Generate a new deployment template for cluster MyCluster (for a list of available templates see the k8t new template --help)

$ k8t new template deployment -c MyCluster -e staging

Config management

To ease file access a little bit k8t can open config and value files in your $EDITOR or fallback to a sensible default.

$ k8t edit values --environment staging

$ k8t edit config --cluster MyCluster

Validate templates

While validation is done before generating, templates can be validated for environment files easily.

$ k8t validate

To validate for clusters/environments the usual options can be used

$ k8t validate -c MyCluster -e production

Shortcomings

The validation is currently not a 100% correct and can miss certain edge cases. If you notice any other issues please let us know.

is defined

The following will result in a false negative for foobar being defined

{{ foobar }}

{% if foobar is defined %}
{{ foobar }}
{% endif %}

To avoid this make sure that the is defined test is applied to all instances of the variable.

The following may result in a false positive for bar being undefined

{% if foobar is defined %}
{{ bar }}
{% endif %}

Generate manifests

The --cluster flag will load variables from a directory. By default the file default.yaml in that directory will be loaded, however an environment can be specified with --environment.

$ k8t gen -c MyCluster -e staging

Additionally k8t will attempt to load a file defaults.yaml in the root directory. This way a set of default variables can be specified and selectively overriden via cluster and environment.

Additional values can be given via flag --value-file in the form of a file or --value KEY VALUE, both can be supplied multiple times.

Variables will be merged via deep merging. Default merge strategy is left-to-right.

Overriding templates

Templates can be overriden on a cluster/environment level.

If a file application.yaml exists in the root templates folder, simply add a file with the same name to the cluster/environment template folder.

Managing secrets

Secrets can be interpolated with the helper function get_secret. It requires a key as first argument and providers are configurable by environment/cluster.

foobar: "{{ get_secret('/my-key') }}"

Providers

SSM

Setup secrets on SSM

secrets:
  provider: ssm
  region: "eu-central-1"
  prefix: "/foobar"

Keep in mind that SSM parameter names can be formed as a path and they can only consist of sub-paths divided by slash symbol; each sub-path can be formed as a mix of letters, numbers and the following 3 symbols: .-_

Be careful to follow this format when setting up the provider prefix and get_secret(key).

Global secrets config can be overridden in get_secret helper function call by specifying config_override argument.

foobar: "{{ get_secret('/my-key', config_override={'prefix': '/dev'}) }}"

role assumption

You can optionally assume an IAM role to retrieve secrets by specyfing role_arn in the config:

secrets:
  provider: ssm
  region: "eu-central-1"
  role_arn: "arn:aws:iam::account:role/role-name-with-path"

Random

Random secrets can be generated easily by using the random provider. This provider uses a global dictionary to store results for the time of the run in python so keys should always produce the same result.

secrets:
  provider: random

Hash

In case consistent (fake) secrets are needed, the hash provider can be used that hashes the secret key for the value.

secrets:
  provider: hash

Using as a pre-commit hook

You can also use this repo as a https://github.com/pre-commit/pre-commit hook

Add this to your .pre-commit-config.yaml:

-   repo: https://github.com/ClarkSource/k8t
    rev: ''  # Use the sha / tag you want to point at
    hooks:
    -   id: k8t-validate
        # args: [ -e dev -c us-west-2 ]
    -   id: k8t-generate
        name: k8t(dev)
        args: [ -o dev.yaml -e dev ]

TODO

• testing needs to be expanded
• the ability to add additional template directories via the CLI
• validation functions for template values (e.g. memory/cpu values)