kiwigrid/helm-charts

[fluentd-elasticsearch] How could I parse the message from nginx ingress log?

Closed this issue · 1 comments

Is this a request for help?:

Yes. I want to know how to config to parse the nginx message. Thank you!

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

FEATURE REQUEST

Version of Helm and Kubernetes:

helm: v2.15.1
k8s: v1.16.2

Which chart in which version:
kiwigrid / helm-charts
fluentd-elasticsearch

What happened:
did not parse the mesage of nginx ingress logs.

the json in elasticsearch

{
  "_index": "logstash-2019.11.18",
  "_type": "_doc",
  "_id": "d1BafW4BRywZvRE64q6F",
  "_version": 1,
  "_score": null,
  "_source": {
    "stream": "stdout",
    "docker": {
      "container_id": "01b3d5b246c2479e556d3edd131bf7bafe80090a4497850f8432e9f04041f282"
    },
    "kubernetes": {
      "container_name": "nginx-ingress-controller",
      "namespace_name": "kube-system",
      "pod_name": "ingress-nginx-ingress-controller-59cc679bdb-6v2bp",
      "container_image": "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1",
      "container_image_id": "docker-pullable://quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:d0b22f715fcea5598ef7f869d308b55289a3daaa12922fa52a1abf17703c88e7",
      "pod_id": "c8ca1454-7ef5-47a7-8403-aa6765e2b9a6",
      "host": "izuf6c75kyx5u99dp8fvj5z",
      "labels": {
        "app": "nginx-ingress",
        "component": "controller",
        "pod-template-hash": "59cc679bdb",
        "release": "nginx-ingress"
      },
      "master_url": "https://10.96.0.1:443/api",
      "namespace_id": "6aba37d6-25be-48a9-bccc-c8b5cb32ffde",
      "namespace_labels": {
        "certmanager_k8s_io/disable-validation": "true"
      }
    },
    "message": "210.13.118.218 - - [18/Nov/2019:07:12:17 +0000] \"GET /api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_6_0 HTTP/2.0\" 200 20857 \"https://mydomain\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\" 131 0.011 [elasticsearch-kibana-kibana-5601] [] 10.244.0.59:5601 20883 0.012 200 17807bad592d52e7b3acd117eb891a78\n",
    "@timestamp": "2019-11-18T07:12:17.973304770+00:00",
    "tag": "kubernetes.var.log.containers.nginx-ingress-nginx-ingress-controller-59cc679bdb-6v2bp_kube-system_nginx-ingress-controller-01b3d5b246c2479e556d3edd131bf7bafe80090a4497850f8432e9f04041f282.log"
  },
  "fields": {
    "@timestamp": [
      "2019-11-18T07:12:17.973Z"
    ]
  },
  "sort": [
    1574061137973
  ]
}

What you expected to happen:
parse message 210.13.118.218 - - [18/Nov/2019:07:12:17 +0000] \"GET /api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_6_0 HTTP/2.0\" 200 20857 \"https://mydomain\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\" 131 0.011 [elasticsearch-kibana-kibana-5601] [] 10.244.0.59:5601 20883 0.012 200 17807bad592d52e7b3acd117eb891a78\n field to elasticsearch

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

additionalPlugins:
  - name: fluent-plugin-elasticsearch
    version: 3.6.1
  - name: fluent-plugin-kubernetes_metadata_filter
    version: 2.4.0

stale commented

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.