kube-filebeat
is a Docker container running filebeat and kube-gen. kube-gen
watches for events on the Kubernetes API and generates filebeat configurations (based on Pod annotations) to harvest logs from applications running in Kubernetes and ship them to logstash.
Note: This project is mostly experimental. It relies on and exploits the mechanics of Docker's filesystem layer. The implementation here only works for Docker versions >= 1.10.0 and may break at any time.
Due to the mechanics of how kube-filebeat
operates, it needs to be running on any node from which you would like to collect logs. The recommended way to acheive this is to run kube-filebeat
as a Daemon Set. For example:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: "kube-filebeat"
annotations:
description: "automated log shipper powered by annotations"
spec:
template:
spec:
containers:
-
name: "kube-filebeat"
image: "kylemcc/kube-filebeat:latest"
env:
-
name: LOGSTASH_HOSTS
value: logstash.default.svc.cluster.local:5044
-
name: KUBERNETES_API_URL
value: http://10.1.2.3:8080
volumeMounts:
- name: docker
mountPath: /var/lib/docker
imagePullPolicy: "Always"
restartPolicy: "Always"
volumes:
- name: docker
hostPath:
path: /var/lib/docker
Annotations are used to inform kube-filebeat
of files that should be harvested. For example:
apiVersion: v1
kind: Pod
metadata:
annotations:
kube_filebeat: >
[
{
"log": "/var/log/example-app/output.log",
"ignore_older": "24h",
"close_older": "24h",
"fields": {
"app": "example-app",
"version": "1.2.3"
},
"multiline": {
"pattern": "^(([[:alpha:]]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2})|([0-9]{4}-[0-9]{2}-[0-9]{2}))",
"negate": true,
"match": "after"
}
},
{
"log": "/var/log/nginx/access.log",
"exclude_lines": [".*Go-http-client/1\\.1.*"],
"ignore_older": "24h",
"close_older": "24h",
"fields": {
"app": "example-app",
"version": "1.2.3",
"type": "access_log"
}
}
]
name: example-app
spec:
containers:
- image: example-app:1.2.3
name: example-app
For multi-container pods, specify the container name in each filebeat config. E.g.:
apiVersion: v1
kind: Pod
metadata:
annotations:
kube_filebeat: >
[
{
"container": "example-app",
"log": "/var/log/app/logfile",
...
},
{
"container": "nginx",
"log": "/var/log/nginx/access.log",
...
}
]
spec:
containers:
- image: example-app:1.2.3
name: example-app
- image: nginx:latest
name: nginx