1. Getting started
1. What's more
- 2.1. Elasticsearch Cert
- 2.2. Overall
- 2.3. Pod Template
- 2.4. Build Your Own Elastalert Dockerfile.
- 2.5. Notice
1. Contact Me

Elastalert Operator for Kubernetes

The Elastalert Operator is an implementation of a Kubernetes Operator.

1. Getting started

Firstly, learn How to use elastalert, exactly how to setup a config.yaml and rule. The default command to start elastalert container is elastalert --config /etc/elastalert/config.yaml --verbose.

To install the operator, run:

kubectl create namespace alert
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/es.noah.domain_elastalerts.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/role.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/role_binding.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/service_account.yaml
kubectl create -n alert -f https://raw.githubusercontent.com/toughnoah/elastalert-operator/master/deploy/deployment.yaml

Args for Operator:

# /manager -h
-health-probe-bind-address string
The address the probe endpoint binds to. (default ":8081")

-leader-elect bool
Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager. (default false)

-metrics-bind-address string
The address the metric endpoint binds to. (default ":8080")

-zap-log-level string
Zap Level to configure the verbosity of logging. Can be one of debug, info, error. (default info)

Once the elastalert-operator deployment in the namespace alert is ready, create an Elastalert instance in any namespace, like:

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
  - name: error-messages
    type: any
    index: your-index-here
    filter:
    - query:
        query_string:
          query: "message: error"
    min_threshold: 1
  - name: error-status-code
    type: any
    index: your-index-here
    filter:
    - query:
        query_string:
          query: "http_status_code: 500"
    min_threshold: 1
  config:
    es_host: es.domain
    es_port: 9200
    es_username: username
    es_password: password
    writeback_index: your-elastalert-index
    run_every: 
      minutes: 5
    buffer_time: 
      minutes: 5
  overall:
    alert:
      - "post"
      http_post_url: "test.com"
      http_post_headers:
        Content-Type: |-
          "application/json"
      http_post_static_payload:
          ...
      
EOF

This will create an elastalert instance named elastalert, and the operator will create a deployment named the same, as:

# kubectl get -n alert deployment
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
elastalert        1/1     1            1           10m

Next you can check your elastalert pod.

# kubectl get -n alert pod
NAME                               READY   STATUS    RESTARTS   AGE
elastalert-76c95597f8-zmbzz         1/1    Running      0       10m

And check your configmaps that -config is created from config. -rule is created from rule.

# kubectl get configmap
NAME                     DATA   AGE
elastalert-config         1     10m
elastalert-rule           1     10m

Above mentioned configmaps will be mounted to /etc/elastalert and /etc/elastalert/rule as config.yaml and rule yaml named by its ["name"]

# /etc/elastalert  ls
config.yaml  rules

# /etc/elastalert/rules ls
error-message.yaml      error-status-code.yaml

2. What's more

2.1. Elasticsearch Cert

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
  - name: error-messages
    ...
  - name: error-status-code
    ...
  config:
    use_ssl: True
  overall:
    alert:
      ...
  cert: |-
        -----BEGIN CERTIFICATE-----
      MIIDYjCCAkqgAwIBAgIRAPUiB7MQDUsQlH5qOvwzaeQwDQYJKoZIhvcNAQELBQAw
      OzEZMBcGA1UECxMQc2ctZWxhc3RpY3NlYXJjaDEeMBwGA1UEAxMVc2ctZWxhc3Rp
      Y3NlYXJjaC1odHRwMB4XDTIwMTIxNjA2MTYyMloXDTIxMTIxNjA2MjYyMlowOzEZ
      MBcGA1UECxMQc2ctZWxhc3RpY3NlYXJjaDEeMBwGA1UEAxMVc2ctZWxhc3RpY3Nl
      YXJjaC1odHRwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6wgW0Al
      6txmJN0/ihVZkqxtmY9owIskB3CxOaaXva27LGqj0Rp5b/OXJIem2h9UKuYqTQCp
      xwToEVw7rmIuyGXRtC2qSKHDFDSFHSDJFHSDKFSDHKCTi6O6uqMcrgjWwATfT76i
      dzeceg4Ly8FINTd2MBi5IbB+UybT/V0T89CeRssjQHEjAX5qEJYW7iTG01nAvaVl
      qfmsMvsZVzU+T8K7ZWrfhkRI2y6ln3hfsE1rKeAzX788RwU8o3GA41Jk5md0yE3a
      y5h0odVoanM1GLH29nswl28t0UAvq/K7kg38V0Kzuocc09mStCXKTf+I74YG7LBu
      V8IHPfCliPSvCwSKDHFJSDHFJKSDNMXCWER72sfSDFETVNFQAheqoQwHQYDVR0lB
      KwYBBQUHAwEGCCsGAQUFBwMCMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFGVU
      OaiBC+86H7lAHu5xM95vSO+VMA0GCSqGSIb3DQEBCwUAA4IBAQCM8mvNRjDOj/kn
      7Fni7FVp6v7Oa7yXiK0knzRX9GoHkniA/a5rZN3Fau+i2y6g2vaUs4BtdsAAdyC7
      GHIImn2M91nJXxcCFD0sfSKDJHFJSKDHFsdfsfeuWwMZJpBHSIw4aNmfX3lsde4c
      6D4+FzlWBKg/JTCv/63HVJ+m+HMKDE8h12aUM1n2rTHiMtnuRkIBa9uoXydN+QCM
      trjLH8AHqXNEPpraKFPRsCVHHhlBlDfpTSkvFgQLAKCh+heXDLyG1e+NEeU+sw9X
      sVz/o0pQkfwUNQjKsPwxQCHxjjw0qZn02wwX8fHgvmCwNIOK/WgsUlNDpny8CLiC
      pxb0oEG7
      -----END CERTIFICATE-----
EOF

This action will create a new secret mounted as /ssl/elasticCA.crt, and operator will automatically create a secret.

# kubectl get -n alert secret
NAME                               TYPE                                  DATA   AGE
elastalert-es-cert                 Opaque                                1      10m

2.2. Overall

overall is used to config global alert settings. If you defined alert in a rule, it will override overall settings.

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
  - name: error-messages
    ...
  - name: error-status-code
    ...
  config:
    ...
  overall:
    alert:
      - "post"
      http_post_url: "test.com"
      http_post_headers:
        Content-Type: |-
          "application/json"
      http_post_static_payload:
          ...
  cert: |-
    ...
EOF

2.3. Pod Template

Define customized podTemplate

kubectl apply -n alert -f - <<EOF
apiVersion: es.noah.domain/v1alpha1
kind: Elastalert
metadata:
  name: elastalert
spec:
  rule:
    ...
  config:
    ...
  overall:
    ...
  cert: |-
    ...
  podTemplate:
      spec:
        containers:
        - name: elastalert
          imagePullPolicy: Always
          resources:
            limits:
              cpu: "2000m"
              memory: "4Gi"
            requests:
              cpu: "500m"
              memory: "1Gi" 
EOF

You can override the default command here.

2.4. Build Your Own Elastalert Dockerfile.

FROM docker.io/library/python:3.7.4-alpine

ENV TZ=Asia/Shanghai

ENV CRYPTOGRAPHY_DONT_BUILD_RUST=1

RUN apk add gcc g++ make libffi-dev openssl-dev

RUN apk add --no-cache tzdata

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN pip install --upgrade pip

RUN pip install elastalert

ENTRYPOINT ["elastalert", "--config", "/etc/elastalert/config.yaml", "--verbose"]

2.5. Notice

You don't have to specify rules_folder in config section, because operator will auto patch rules_folder: /etc/elastalert/rules/..data/ for your config. The reason why have to be ..data/ is the workaround when the configmap is mounted as file(such as /etc/elastalert/rules/test.yaml) in a pod, it will create a soft-link to /etc/elastalert/rules/..data/test.yaml. That is to say, you will receive duplicated rules name error that both files in rules and ..data would be loaded if you specify merely rules_folder: /etc/elastalert/rules

3. Contact Me

Any advice is welcome! Please email to toughnoah@163.com