New Relic Kubernetes Operator

Overview
Quick Start
Using the Operator to Provision Resources
- Create a New Relic alert policy
- Create a New Relic NRQL alert condition
Development

Overview

The newrelic-kubernetes-operator is a Kubernetes Operator that facilitates management of New Relic resources from within your Kubernetes configuration. Managing New Relic resources via custom Kubernetes objects can be done the same way you manage built-in Kubernetes objects.

Currently the operator supports managing the following resources:

Alert Policies
NRQL Alert Conditions.
Alert Conditions for APM, Browser and mobile
Alert Channels

Quick Start

Note: These quick start instructions do not require you to clone the repo.

Running Kubernetes in a Docker container locally with kind

Install docker, kubectl, kustomize, and kind

brew cask install docker
brew install kubernetes-cli kustomize kind

Create a test cluster with kind

kind create cluster --name newrelic
kubectl cluster-info

Install cert-manager
```
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml
```
Note: This takes a minute or two to finish so wait a minute before going on to the next step.

You can also confirm it's running with the command kubectl rollout status deployment -n cert-manager cert-manager-webhook
Install the operator in the test cluster.
```
kustomize build https://github.com/newrelic/newrelic-kubernetes-operator/config/default | kubectl apply -f -
```
Note: This will install operator on whatever kubernetes cluster kubectl is configured to use.

Using a custom container

If you want to deploy the operator in a custom container you can override the image name with a kustomize.yaml file.

Create a new kustomize.yaml file

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: newrelic-kubernetes-operator-system
resources:
  - github.com/newrelic/newrelic-kubernetes-operator/config/default
images:
  - name: newrelic/kubernetes-operator:snapshot
    newName: <CUSTOM_IMAGE>
    newTag: <CUSTOM_TAG>

The apply the file with:
```
kustomize build . | kubectl apply -f -
```

Provision New Relic resources with the operator

Once you've completed the Quick Start section, you can start provisioning New Relic resources with our New Relic Kubernetes objects.

Create a New Relic alert policy with NRQL alert conditions

We'll be using the following example policy configuration file. You will need to update the api_key field with your New Relic personal API key.

examples/example_policy.yaml

apiVersion: nr.k8s.newrelic.com/v1
kind: AlertsPolicy
metadata:
  name: my-policy
spec:
  # Add your account ID here
  account_id: <your New Relic account ID>
  # Add your API key here
  api_key: <your New Relic personal API key>
  name: k8s created policy
  incidentPreference: "PER_POLICY"
  region: "US"
  conditions:
    - spec:
        type: "NRQL"
        nrql:
          query: "SELECT count(*) FROM Transactions"
          evaluationOffset: 10
        enabled: true
        terms:
          - threshold: "75.0"
            threshold_occurrences: "ALL"
            threshold_duration: 60
            priority: "CRITICAL"
            operator: "ABOVE"
        name: "K8s generated alert condition"
    - spec:
        type: "apm_app_metric"
        enabled: true
        metric: "apdex"
        condition_scope: application
        entities:
          - "5950260"
        apm_terms:
          - threshold: "0.9"
            time_function: "all"
            duration: "30"
            priority: "critical"
            operator: "above"
        name: "K8s generated apm alert condition 2"

Once you've added your API key, we can apply it your local cluster.

kubectl apply -f examples/example_policy.yaml

Note: You can also use a Kubernetes secret for providing your API key. We've provided an example secret configuration file in case you want to use this method. You'll need to replace api_key with api_key_secret.

See your configured policies with the following command.
```
kubectl describe alertspolicies.nr.k8s.newrelic.com
```
Note: You should also see the newly created policy within your New Relic account.

The operator will create and update alert policies and NRQL alert conditions as needed by applying your configuration files with kubectl apply -f <filename>

Create a NRQL alert condition and add it to an existing alert policy

We'll be using the following example NRQL alert condition configuration file. You will need to update the api_key field with your New Relic personal API key.

examples/example_nrql_alert_condition.yaml

apiVersion: nr.k8s.newrelic.com/v1
kind: AlertsNrqlCondition
metadata:
  name: my-alert-condition
spec:
  # Add your account ID here
  account_id: <your New Relic account ID>
  # Add your API key here
  api_key: <your New Relic personal API key>
  name: "K8s generated alert condition"
  type: "NRQL"
  nrql:
    # Note: This is just an example.
    # You'll want to use a query with parameters that are
    # more specific to the needs for targeting associated
    # kubernetes objects.
    query: "SELECT count(*) FROM Transactions"
    evaluationOffset: 10
  enabled: true
  terms:
    - threshold: "75.0"
      threshold_occurrences: "ALL"
      threshold_duration: 60
      priority: "CRITICAL"
      operator: "ABOVE"
  existing_policy_id: "26458245" # Note: must match an existing policy in your account
  region: "US"

Create an Alerts Channel

We'll be using the following example alerts channel configuration file. You will need to update the api_key field with your New Relic personal API key.

examples/example_alerts_channel.yaml

apiVersion: nr.k8s.newrelic.com/v1
kind: AlertsChannel
metadata:
  name: my-channel1
  spec:
    api_key: <your New Relic personal API key>
    # api_key_secret:
    #   name: nr-api-key
    #   namespace: default
    #   key_name: api-key
    name:         "my alert channel"
    region:       "US"
    type:         "email"
    links:
      # Policy links can be by NR PolicyID, NR PolicyName AND/OR K8s AlertPolicy object reference 
      policy_ids: 
        - 1
      policy_names: 
        - "k8s created policy"
      policy_kubernetes_objects: 
        - name: "my-policy"
          namespace: "default"
    configuration: 
      recipients: "me@email.com"

Note: The New Relic Alerts API does not allow updating Alerts Channels. In order to change a channel, you will need to either rename the k8s AlertsChannel object to create a new one and delete the old one or manually delete the k8s AlertsChannel object and create a new one.

Monitoring the New Relic Operator

The New Relic Operator uses the New Relic Go Agent to report monitoring statistics. This can be activated by adding a Kubernetes Secret and an option ConfigMap to configure the agent. We'll be using Example_New_Relic_Agent_Config

apiVersion: v1
kind: Secret
metadata:
  name: nr-agent
  namespace: newrelic-kubernetes-operator-system
type: Opaque
stringData:
  license-key: <your New Relic license key, base64 encoded>
  
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nr-agent
  namespace: newrelic-kubernetes-operator-system
data:
  # defaults to New Relic Kubernetes Operator
  app-name: <your desired New Relic App name for the Operator>
  # Defaults to collector.newrelic.com
  host: <New Relic endpoint>
  # Defaults to false
  log-level: debug

Note: If the agent isn't reporting, make sure to check your base64 encoding didn't include a /n character.

Uninstall the operator

The Operator can be removed with the reverse of installation, namely building the kubernetes resource files with kustomize and running kubectl delete

kustomize build github.com/newrelic/newrelic-kubernetes-operator/config/default | kubectl delete -f -

Development

This section should get you set up properly for doing development on the operator.

Requirements

Go v1.13+
Docker (with Kubernetes enabled)
kubectl
kustomize
kubebuilder

Code

Clone the repo

git clone git@github.com:newrelic/newrelic-kubernetes-operator.git

Install kubebuilder following the instructions for your operating system. This installation will also get etcd and kube-apiserver which are needed for the tests.

Note: Do not install kubebuilder with brew. Homebrew's kubebuilder package will not provide all the necessary dependencies for running the tests.

Run the test suite, which uses the Ginkgo testing framework. Using the make targets is the quickest way to get started with testing.

Running tests with make

make test              # runs all tests
make test-unit         # only runs unit tests
make test-integration  # only runs integration tests

Linting the codebase
```
make lint
```

Perform the steps from the Quick Start section, except use the following instead of step 4 to install with the :snapshot image:
```
kustomize build config/development | kubectl apply -f -
```
Confirm your configuration was deployed to your local Kubernetes cluster (the one that we created with kind).
- Show your namespaces. You should see newrelic-kubernetes-operator-system in the list of namespaces.
```
kubectl get namespaces
```
- Show the nodes within the newrelic-kubernetes-operator-system namespace.
```
kubectl get nodes -n newrelic-kubernetes-operator-system
```
  You should see something similar to the following output:
```
NAME                     STATUS   ROLES    AGE    VERSION
newrelic-control-plane   Ready    master   163m   v1.18.2
```
Personalize your development alert policy

Edit examples/development/personal* with your name and API key.

Next, create your customized alert policy:
```
kustomize build examples | kubectl apply -f -
```
To delete your customized policy:
```
kustomize build examples | kubectl delete -f -
```

Helpful commands

# Describe the currently configured policies.
kubectl describe alertspolicies.nr.k8s.newrelic.com

# Describe the currently configured alert conditions.
kubectl describe alertsnrqlconditions.nr.k8s.newrelic.com

# Get the node being used for the newrelic operator.
kubectl get nodes -n newrelic-kubernetes-operator-system

# Describe the node being used for the newrelic operator.
kubectl describe node <your-node-name>

# Tail logs of the operator's manager container (useful during development).
# Use the `describe node` command above to locate your manager controller.
kubectl logs -f -n newrelic-kubernetes-operator-system -c manager newrelic-kubernetes-operator-controller-manager-<hash from>

tooooots/newrelic-kubernetes-operator