[fluentd-elasticsearch] How could I parse the message from nginx ingress log?
Closed this issue · 1 comments
Is this a request for help?:
Yes. I want to know how to config to parse the nginx message. Thank you!
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST
Version of Helm and Kubernetes:
helm: v2.15.1
k8s: v1.16.2
Which chart in which version:
kiwigrid / helm-charts
fluentd-elasticsearch
What happened:
did not parse the mesage of nginx ingress logs.
the json in elasticsearch
{
"_index": "logstash-2019.11.18",
"_type": "_doc",
"_id": "d1BafW4BRywZvRE64q6F",
"_version": 1,
"_score": null,
"_source": {
"stream": "stdout",
"docker": {
"container_id": "01b3d5b246c2479e556d3edd131bf7bafe80090a4497850f8432e9f04041f282"
},
"kubernetes": {
"container_name": "nginx-ingress-controller",
"namespace_name": "kube-system",
"pod_name": "ingress-nginx-ingress-controller-59cc679bdb-6v2bp",
"container_image": "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1",
"container_image_id": "docker-pullable://quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:d0b22f715fcea5598ef7f869d308b55289a3daaa12922fa52a1abf17703c88e7",
"pod_id": "c8ca1454-7ef5-47a7-8403-aa6765e2b9a6",
"host": "izuf6c75kyx5u99dp8fvj5z",
"labels": {
"app": "nginx-ingress",
"component": "controller",
"pod-template-hash": "59cc679bdb",
"release": "nginx-ingress"
},
"master_url": "https://10.96.0.1:443/api",
"namespace_id": "6aba37d6-25be-48a9-bccc-c8b5cb32ffde",
"namespace_labels": {
"certmanager_k8s_io/disable-validation": "true"
}
},
"message": "210.13.118.218 - - [18/Nov/2019:07:12:17 +0000] \"GET /api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_6_0 HTTP/2.0\" 200 20857 \"https://mydomain\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\" 131 0.011 [elasticsearch-kibana-kibana-5601] [] 10.244.0.59:5601 20883 0.012 200 17807bad592d52e7b3acd117eb891a78\n",
"@timestamp": "2019-11-18T07:12:17.973304770+00:00",
"tag": "kubernetes.var.log.containers.nginx-ingress-nginx-ingress-controller-59cc679bdb-6v2bp_kube-system_nginx-ingress-controller-01b3d5b246c2479e556d3edd131bf7bafe80090a4497850f8432e9f04041f282.log"
},
"fields": {
"@timestamp": [
"2019-11-18T07:12:17.973Z"
]
},
"sort": [
1574061137973
]
}
What you expected to happen:
parse message
210.13.118.218 - - [18/Nov/2019:07:12:17 +0000] \"GET /api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_6_0 HTTP/2.0\" 200 20857 \"https://mydomain\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\" 131 0.011 [elasticsearch-kibana-kibana-5601] [] 10.244.0.59:5601 20883 0.012 200 17807bad592d52e7b3acd117eb891a78\n
field to elasticsearch
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know:
additionalPlugins:
- name: fluent-plugin-elasticsearch
version: 3.6.1
- name: fluent-plugin-kubernetes_metadata_filter
version: 2.4.0
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.