/helm-scalyr

A helm chart for the Scalyr agent.

Primary LanguageMustacheApache License 2.0Apache-2.0

scalyr-agent Helm Chart

Latest stable release of the Helm chart License Lint and Tests End to End Tests

Introduction

This helm chart installs the Scalyr Agent in a Kubernetes cluster. Two Agent monitors are enabled:

We implement the Kubernetes-recommended node-level logging architecture. The Agent runs as a DaemonSet; an Agent pod runs on each node and collects logs from other pods in the node. By default, the Agent collects pod logs, container metrics, and Kubernetes Events for all nodes.

You can also install the Agent to monitor other parts of the infrastructure, for example a hosted database service.

For more metrics and insight into your cluster, this chart can also install Kubernetes Explorer (in preview release). This enables a third, Openmetrics monitor. The standard installation has approximately 60 metrics; Kubernetes Explorer provides over 500 out of the box. You can easily collect more metrics from applications running in your cluster with open source metric exporters.

Installation

You must set some configuration options:

  • scalyr.apiKey: Must be a "Log Write Access" API key. Log into DataSet and select your account (email address). Then select "Api Keys".
  • scalyr.k8s.clusterName: You must set a name for your Kubernetes cluster, which shows in the UI.
  • By default data uploads to our US server. For EU customers, set scalyr.server="eu.scalyr.com".

To install:

helm install <name of release> scalyr-agent --repo https://scalyr.github.io/helm-scalyr/ --set scalyr.apiKey="<your write logs api key>" --set scalyr.k8s.clusterName="<your-k8s-cluster-name>"

Kubernetes Explorer

Kubernetes Explorer is our latest Kubernetes integration. (https://www.dataset.com/blog/introducing-dataset-kubernetes-explorer/).

To install:

helm install <name of release> scalyr-agent --repo https://scalyr.github.io/helm-scalyr/ --set scalyr.apiKey="<your write logs api key>" --set scalyr.k8s.clusterName="<your-k8s-cluster-name>" --set scalyr.k8s.enableExplorer=true

Kubernetes Explorer has two required dependencies, node-exporter and kube-state-metrics. If these are already installed in your cluster, see Configure Kubernetes Explorer to annotate the node-exporter DaemonSet, and the kube-state-metrics Deployment.

The helm chart can install these components for you; set scalyr.k8s.installExplorerDependencies to true. This is usually to evaluate Kubernetes Explorer in a fresh cluster, for example in minikube. To make cleanup easier, the components install in the same namespace as the Agent.

Note that minikube uses self-signed SSL certificates. You must set scalyr.k8s.verifyKubeletQueries to false, which disables certificate validation when talking to the Kubelet API. (Unless you have a very good reason, do not disable certificate validation in production.)

Also note that minikube runs a single-node (master) by default, and you must set scalyr.k8s.eventsIgnoreMaster to false for the Kubernetes Events monitor to run on master.

To install:

helm install <name of release> scalyr-agent --repo https://scalyr.github.io/helm-scalyr/ --set scalyr.apiKey="<your write logs api key>" --set scalyr.k8s.clusterName="<your-k8s-cluster-name>" --set scalyr.k8s.enableExplorer=true --set scalyr.k8s.installExplorerDependencies=true --set scalyr.k8s.verifyKubeletQueries=false --set scalyr.k8s.eventsIgnoreMaster=false

You can also consult our Minikube installation page for more information on the Service and DaemonSet for node-exporter; and the Deployment, Service, ServiceAccount, ClusterRole, and ClusterRoleBinding for kube-state-metrics.

Configuration

The chart's default values are set to monitor a Kubernetes cluster.

To monitor other parts of the infrastructure, for example a Database, set:

  • controllerType: It is usually best to set this to "deployment" instead of "daemonset".
  • scalyr.k8s.enableLogs and scalyr.k8s.enableEvents: Set these to false to remove the serviceaccount, clusterroles and other mounts to the Agent pods.

Configuration takes the confimap approach. This is basically a key-value hash. The keys refer to the configuration file name for grouping monitors. The value is the Scalyr json configuration for each monitor.

Set Custom Configuration Options

To set configuration options that are not in the chart's values.yaml, the Agent reads and parses JSON file fragments. These can be set with the scalyr.config option in the values file.

Since Helm cannot pass JSON strings as YAML key values, each JSON fragment must be base64 JSON. For example, if your custom config fragment is at ci/examples/agent.d/my-config.json:

1. Create the base64 encoded version of the JSON file content

cat ci/examples/agent.d/my-config.json | sed -e 's/^ *//' | tr -d '\n' | base64 | tr -d '\n' ; echo ""

sed and tr convert multi-line JSON to a single line, to prevent any issues with YAML formatting.

2. Update your values files

For example, in my-custom-config.yaml, add:

scalyr:
  config: {
    my-config.json: eyJtYXhfbG9nX29mZnNldF9zaXplIjogNTI0Mjg4MCwiZGVidWdfbGV2ZWwiOiA1fQ==
  }

Then pass the file to the install command:

helm install ... -f my-custom-config.yaml

Service Account Annotations

A common use case for Kubernetes ServiceAccounts is to provide pods with specific permissions to cloud resources. Consider the case where you wish to store the scalyr.apiKey in a secrets management service rather than plaintext or as a Kubernetes Secret. If you use a cloud-specific solution such as AWS Parameter Store or AWS Secrets Manager, the scalyr-agent pods will require specific IAM permissions to retrieve the value. (Note this will apply to any Kubernetes ServiceAccount-based permissions scheme)

AWS EKS

EKS's native pod permission management system is IAM Roles for Service Accounts (IRSA). IRSA documentation is here: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Assumption: You are using a Mutating Webhook or other solution that tells your pods to read secrets from Parameter Store or Secrets Manager upon startup.

Example:

  1. Create IAM Policy that can read the secrets in Param Store or Secrets Manager
  2. Create a service AWS IAM Role with appropriate OIDC template for this EKS cluster.
  3. Attach the IAM Policy to the IAM Role
  4. Override serviceAccount.annotations value in the helm chart with a value like the below:
# Values relevant to ServiceAccount
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>

This gives the pod permission to read the secret as defined in the IAM Policy. (Something in the cluster such as a MutatingWebhook will need to actually facilitate the secret lookup)

Using raw value for "SCALYR_API_KEY" pod environment variable

By default, scalyr.apiKey chart value is stored in a Kubernetes Secret and then this secret is referenced by the SCALYR_API_KEY pod environment variable.

In some situations, you may want to define a raw value for this environment variable. An example of that is using a tool like kube-secrets-init which relies on environment variable being set to a special prefixed value which will eventually get replaced with the actual secret by the tool itself.

Here is an example excerpt chart configuration for such use case:

# Values relevant to ServiceAccount
scalyr:
  apiKey: gcp:secretmanager:projects/$PROJECT_ID/secrets/myscalyragentapikey/versions/2
useRawApiKeyEnvValue: true

Changelog

For chart changelog, please see https://github.com/scalyr/helm-scalyr/blob/main/CHANGELOG.md.

For agent changelog, please see https://github.com/scalyr/scalyr-agent-2/blob/release/CHANGELOG.md.

Maintainers

Name Email Url
scalyr support@scalyr.com https://github.com/scalyr
dploeger develop@dieploegers.de https://github.com/dploeger

Source Code

Values

Key Type Default Description
affinity object {} optional affinity rules
controllerType string "daemonset" Wether to setup a daemonset or a deployment for the Scalyr agent A daemonset should be used for Kubernetes monitoring while a deployment should be used for single resource monitorings (e.g. hosted databases, etc.) Valid values: "daemonset" or "deployment"
deployment.replicaCount int 1 The count of replicas to use when using the deployment controller setup
existingSecretRef string "" Use this value if the Scalyr API key is already stored in a Kubernetes secret that was created by an external secrets operator or similar.
extraEnvVars string nil Additional environment variables to set
fullnameOverride string "" Override the default full name that helm calculates
image.pullPolicy string "IfNotPresent"
image.repository string "scalyr/scalyr-k8s-agent" Image to use. Defaults to the official scalyr agent image
image.tag string "" Tag to use. Defaults to appVersion from the chart metadata
image.type string "buster" Which image distribution to use - "buster" for Debian Buster and "alpine" for Alpine Linux based image. Alpine Linux images are around 50% smaller in size than Debian buster based ones.
imagePullSecrets list [] Image pull secrets to use if the image is in a private repository
livenessProbe.debug bool false set to true to enable printing additional debug information during the health check command.
livenessProbe.enabled bool true set to false to disable default liveness probe which utilizes scalyr-agent-2 status -H command
livenessProbe.timeoutSeconds int 10 timeout in seconds after which probe should be considered as failed if there is no response
nameOverride string "" Override the default name that helm calculates
nodeSelector object {} optional node selectors
podAnnotations object {} optional pod annotations
podLabels object {} optional arbitrary pod metadata labels
podSecurityContext object {} optional pod security context entries
resources object {"limits":{"cpu":"500m","memory":"500Mi"},"requests":{"cpu":"500m","memory":"500Mi"}} Pod resources. Defaults to the values documented in the official Installation guide
scalyr.apiKey string "" The Scalyr API key to use. Can also be used in combination with "useRawApiKeyEnvValue" when using something like kube-secrets-init. In that case, this should be a reference to the secret which will be replaced by kube-secrets-init.
scalyr.base64Config bool true As Helm is currently unable to correctly pass JSON strings, this can be set to true so all values of scalyr.config are expected to be base64 encoded and will be decoded in the chart
scalyr.config object {} A hash of configuration files and their content as documented in the Scalyr agent configmap configuration documentation
scalyr.debugLevel int 0 Set this to number between 1 and 5 (inclusive - 1 being least verbose and 5 being most verbose) to enable additional debug logging into agent_debug.log file. NOTE: If you want this debug log file to be ingested into Scalyr, you also need to set scalyr.ingestDebugLog option to true.
scalyr.ingestDebugLog bool false Set this to true to enable ingesting of agent_debug.log file. Keep in mind that depending on the debug log level set, this may result in large log volume.
scalyr.k8s.caCert string "" The path to the CA certificate to use to verify TLS-connection to the kubelet
scalyr.k8s.clusterName string "" The kubernetes cluster name (when using the kubernetes monitoring)
scalyr.k8s.enableEvents bool true Enable fetching Kubernetes events
scalyr.k8s.enableExplorer bool false Enable Kubernetes Explorer functionality (https://www.dataset.com/blog/introducing-dataset-kubernetes-explorer/). This functionality may require additional setup, for more information, please refer to the docs - https://app.scalyr.com/help/scalyr-agent-k8s-explorer NOTE: Explorer functionality is only supported when using DaemonSet agent deployment model.
scalyr.k8s.enableLogs bool true Enable fetching Pod/Container logs from Kubernetes
scalyr.k8s.enableMetrics bool true Enable fetching Kubernetes metrics. This requires scalyr.k8s.enableLogs to be true
scalyr.k8s.eventsIgnoreMaster bool true Set to false to also allow Kubernetes Events monitor to run on master node.
scalyr.k8s.explorerSampleInterval int 60 Sample internal (in seconds) for Kubernetes Explorer functionality.
scalyr.k8s.explorerScrapeInterval int 60 Scrape internal (in seconds) for Kubernetes Explorer functionality.
scalyr.k8s.installExplorerDependencies bool false Set to true to install additional dependencies which are needed for the complete Kubernetes Explorer experience. This includes node-exporter DaemonSet and kube-state-metrics Deployment. Both of the components are installed into the same namespace as the scalyr agent for easier cleanup. In production deployments, those two components usually get installed into monitoring or kube-system namespace. This functionality is only meant to be used on new clusters which don't already have those components running (e.g. local minikube cluster).
scalyr.k8s.verifyKubeletQueries bool true Set this to false to disable TLS cert validation of queries to k8s kubelet. By default cert validation is enabled and connection is verified using the CA configured via the service account certificate (/run/secrets/kubernetes.io/serviceaccount/ca.crt file). If you want to use a custom CA bundle, you can do that by setting scalyr.k8s.caCert config option to point to this file (this file needs to be available inside the agent container). In some test environments such as minikube where self signed certs are used you may want to set this to false.
scalyr.priorityClassName string ""
scalyr.server string "agent.scalyr.com" The Scalyr server to send logs to. Use eu.scalyr.com for EU
securityContext object {} optional security context entries
serviceAccount.annotations object {} optional arbitrary service account annotations
tolerations list [{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Exists"}] Pod tolerations. Defaults to the values documented in the official Installation guide
useRawApiKeyEnvValue bool false Set this to true if you want raw API key from "scalyr.apiKey" chart value to be used for the SCALYR_API_KEY pod environment variable. This comes handy in situations where you don't want to use a secret (e.g. you utilize something like kube-secrets-init which directly replaces environment variable value with the actual secret).
volumeMounts object {} Additional volume mounts to set up
volumes object {} Additional volumes to mount

Development, CI/CD

On each push to master and other branches Github Actions workflow runs which performs basic helm lint and helm install sanity checks against the changes.

chart-testing wrapper is used for running helm lint and helm install.

To run those checks locally, you need the following tools installed:

  • helm 3
  • chart-testing
  • minikube (or kind cluster against which helm install can run)
  • Python 3 with the following 3 libraries installed - yamllint, yamale
# 1. Install helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# 2. Install minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# 3. Install chart-testing
wget https://github.com/helm/chart-testing/releases/download/v3.4.0/chart-testing_3.4.0_linux_amd64.tar.gz
tar -xzvf chart-testing_3.4.0_linux_amd64.tar.gz
sudo mv ct /usr/local/bin
sudo mv etc ~/.ct

# 4. Create Python virtualenv and install libraries needed by chart testing
python3 -m venv .venv
source .venv/bin/activate
pip install yamale yamllint

# 5. Start minikube Kubernetes cluster
minikube start

# 6. Run actual lint and install task
ct lint --debug --config ci/ct.yaml

# To use valid API key
echo -e 'scalyr:\n  apiKey: "SCALYR_TEST_WRITE_API_KEY"' > charts/scalyr-agent/ci/test-values.yaml

ct install --debug --config ci/ct.yaml

You can find more example configs which are used by integration and end to end tests in ci/ directory.

As an alternative to manually installing those tools and setting up the environment, you can also use act tool which allows you to run GHA workflow locally inside Docker containers as shown below.

act lint_test
act

Keep in mind that it may take a while since it needs to pull down a large Docker image during the first run. This tool also may not work correctly on some operating systems since it relies on Docker inside Docker functionality for creating kind Kubernetes cluster.

Publishing new version

New version of the chart is automatically released by the Release Github Actions workflow on push to main branch when changes are detected in the chart (e.g. chart content or metadata has been updated).

Helm Chart repository is available at https://scalyr.github.io/helm-scalyr/.

Copyright, License, and Contributor Agreement

Copyright 2020-2021 DO! DevOps. Copyright 2021 SentinelOne, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

http://www.apache.org/licenses/LICENSE-2.0

By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the project, present and future, pursuant to the license of the project.

Thank You

The chart has been originally developed by Dennis Ploeger from dodevops. They have agreed to transfer the ownership to Scalyr so we can continue developing, improving and maintaining the chart.