Help ingesting events into ElasticSearch

[kkarrancsu]

Hello,
I am trying to setup the event exporter to send events to ElasticSearch, but am getting the following error, which I can observe by looking at the kubernetes-event-exporter logs which I also send to a file receiver:

{"level":"debug","sink":"file","event":"Created container event-exporter","time":"2021-09-22T22:34:35Z","caller":"/app/pkg/exporter/channel_registry.go:56","message":"sending event to sink"}
{"level":"debug","sink":"file","event":"Started container event-exporter","time":"2021-09-22T22:34:35Z","caller":"/app/pkg/exporter/channel_registry.go:56","message":"sending event to sink"}
{"level":"debug","error":"EOF","sink":"elasticsearch","event":"Created container event-exporter","time":"2021-09-22T22:34:35Z","caller":"/app/pkg/exporter/channel_registry.go:59","message":"Cannot send event"}

My setup is like this:
1 - I have a minikube cluster
2 - I followed the instructions here to setup an Elasticsearch instance, and a Kibana frontend inside my cluster:
https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html -- I am able to login the frontend of kibana, to ensure that it is running properly.
3 - I then get the username & password for kibana with the commands specified in the link above, and store the min my elastic search receiver config.
4 - I then deploy the event monitor, but get the errors. I see that it tries to send the event to elasticsearch, but I am getting the EOF error.

I am a K8 newbie, so likely it is my setup, but am wondering if someone can help me get this bootstrapped so that I can play with it. Thank you in advance.

[BBQigniter]

This is a config we use to ingest the kubernetes-cluster events to one of our ECK-ELKs - surely needs adaptions to work with your minikube

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-event-exporter
  namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubernetes-event-exporter
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes
  - pods
  - events
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-event-exporter
  namespace: monitor
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-event-exporter
subjects:
  - kind: ServiceAccount
    namespace: monitor
    name: kubernetes-event-exporter
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubernetes-event-exporter-cfg
  namespace: monitor
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
        - match:
            - receiver: "dump"
    receivers:
      - name: "dump"
        elasticsearch:
          hosts:
          - https://your-url-to-es
          index: kubernetes-events
          indexFormat: "kubernetes-events"
          username: kubernetes_events_writer
          password: your-password
          useEventID: false
          tls:
            insecureSkipVerify: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-event-exporter
  namespace: monitor
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kubernetes-event-exporter
        version: v1
    spec:
      serviceAccountName: kubernetes-event-exporter
      containers:
        - name: kubernetes-event-exporter
          image: ghcr.io/opsgenie/kubernetes-event-exporter:v0.10
          imagePullPolicy: IfNotPresent
          args:
            - -conf=/data/config.yaml
          volumeMounts:
            - mountPath: /data
              name: cfg
          resources:
            requests:
              memory: "100Mi"
            limits:
              memory: "200Mi"
      volumes:
        - name: cfg
          configMap:
            name: kubernetes-event-exporter-cfg
      # only run on master (controlplane) nodes
      nodeSelector:
        node-role.kubernetes.io/controlplane: "true"
      # this tells kubernetes to move pods if a node is unreachable or not ready for about 10 seconds 
      # default is 5 minutes
      tolerations:
      - key: "node.kubernetes.io/unreachable"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 10
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 10
  selector:
    matchLabels:
      app: kubernetes-event-exporter
      version: v1

But unfortunately the events are not in ECS - you maybe should have a look at https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html which also seems to be able to ingest kubernetes events nicely (haven't tried yet).

Also I suggest you create the kubernetes-events index manually with ILM ploicies with working rollover etc. and set kubernetes-events as index-alias.