Webhook for Grafana Loki
rajmohanram opened this issue · 2 comments
rajmohanram commented
- Presently Loki is not a supported receiver to push the events. I am trying to make use of Webhook with a layout confirming to the Loki json format as below
{
"streams": [
{
"stream": {
"label": "value"
},
"values": [
[ "<unix epoch in nanoseconds>", "<log line>" ],
[ "<unix epoch in nanoseconds>", "<log line>" ]
]
}
]
}
Loki requires the timestamp field to be in Unix epoch in nanoseconds. But the events exported by this exporter have timestamps in RFC3339 combined date-time format.
- Is it possible to control the timestamp format through the configmap? Until Grafana Loki is supported directly as a receiver, this feature to specify the timestamp format through configmap will help.
rajmohanram commented
Added another function at the end of https://github.com/opsgenie/kubernetes-event-exporter/blob/master/pkg/kube/event.go
func (e *EnhancedEvent) GetTimestampUnixNano() int64 {
return e.FirstTimestamp.UnixNano()
}
Called this function in the layout section of the webhook receiver for Grafana Loki like this:
receivers:
- name: "loki"
webhook:
endpoint: "http://192.168.0.30:3100/loki/api/v1/push"
headers:
Content-Type: application/json
User-Agent: "kube-event-exporter"
layout:
streams:
- stream:
source: event-exporter
app: kube-api
values:
- - "{{ .GetTimestampUnixNano }}"
- "Namespace: {{ .InvolvedObject.Namespace }}, {{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}, Reason: {{ .Reason }}, Message: {{ .Message }} "
ananace commented
You can sort of cheat this as it is right now by converting the millisecond timestamp to nanoseconds without needing to modify the code. The loss of precision may very well be acceptable for most people.
Something like this works in my use-case;
receivers:
- name: "loki"
webhook:
endpoint: 'http://loki:3100/loki/api/v1/push'
headers:
Content-Type: application/json
User-Agent: kube-event-exporter
layout:
streams:
- stream:
kind: kube-event
namespace: '{{ .InvolvedObject.Namespace }}'
type: '{{ .Type }}'
values:
- - "{{ mul .GetTimestampMs 1000000 }}"
- 'level={{ lower .Type | replace "normal" "notice" }} namespace={{ .InvolvedObject.Namespace }} object={{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }} reason={{ .Reason }} message={{ quote .Message }}'