opsgenie/kubernetes-event-exporter

Failed to export events after running for a while

ediezh opened this issue · 3 comments

ediezh commented

The only log I see from the container log is the below warning after the pod has been running for a while

W0504 07:57:16.735926       1 reflector.go:302] pkg/kube/watcher.go:95: watch of *v1.Event ended with: The resourceVersion for the provided watch is too old.

Restarting the pod would fix the problem.

But the event exporter should be able to automatically refresh the watched resources list.

timansky commented

having the same problem, is there any healthcheck or example of livinessprobe?

  • 👍1
aauren commented

Not sure that this will be the problem for everyone, but the linked issue above appears to be the solution for me when I encountered this problem.

Basically, you should not use the file exporter for stdout:

...
receivers:
  - name: "dump"
     file:
       path: "/dev/stdout"

This is because kubernetes-event-exporter uses a library called lumberjack which will automatically attempt to rotate the file path after ~12 hours. Obviously, /dev/stdout cannot rotated.

People that want to put events on stdout should instead use the stdout receiver:

receivers:
  - name: "dump"
     stdout: {}
  • 👍1
youyongsong commented

@mustafaakin-atl upgrade client-go version may solve this issue. kubernetes/kubernetes#95981 this pr add watch health check function.