FEATURE REQUEST- Exposing fluentid-tail-refresh_interval in kiwigrid/fluentd-elasticsearch helm chart

[veenabenoy]

We are using fluentd, installed in the cluster using helm chart kiwigrid/fluentd-elasticsearch --version 6.1.1
we set elasticsearch.host as part of helm install command. So all other fluent id configuration take default values.

We have some pods in the system which will execute in less than 60sec.
We observed that Fluentd is not capturing the logs of such short-life pods.We assume this is happening because fluentd by default is configured(refresh_interval) to update the list of log files it monitors only once a minute. Due to the very short run time of the pods, they are already deleted when fluentd checks next time.

We tried manually adding this value in a configmap on a running system and it solved the problem (restart the fluentd pods after adding config map).
Unfortunately, this variable is not exposed in the kiwigrid helm chart.
So creating a request to add it part of configurable parameteres of Fluentd elasticsearch chart

helm_binary_version: 2.16.1
Kubernetes - v1.15.3

[monotek]

PR would be welcome ;-)

[veenabenoy]

I followed another approach.

1. Override default value and set containersInputConf as false in values yaml.
   e.g

configMaps:
  useDefaults:
    systemConf: true
    containersInputConf: false
    systemInputConf: true
    forwardInputConf: true
    monitoringConf: true
    outputConf: true
2. Defined customized containers Input Configuration in extraConfigMaps as :
extraConfigMaps: 
  containers.input.conf: |-
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/containers.log.pos
      tag raw.kubernetes.*      
      refresh_interval 3s    
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </pattern>
      </parse>
    </source>
...................other required sections of containers.input.conf..............

And I verified refresh interval is updated in fluentd configmaps after deploying the helm chart
(using kubectl describe configmaps

[veenabenoy]

The draw back is just for updating one flag 'refresh_interval" I had to define entire section of containers.input.conf inorder to pass it as extra config parameter

extraConfigMaps: 
  containers.input.conf: |-
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/containers.log.pos
      tag raw.kubernetes.*      
      refresh_interval 3s    
      read_from_head true
      <parse>
        @type multi_format        <pattern>
          format json
          time_key time
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </pattern>
      </parse>
    </source>

    # Detect exceptions in the log output and forward them as one log entry.
    <match raw.kubernetes.**>
      @id raw.kubernetes
      @type detect_exceptions
      remove_tag_prefix raw
      message log
      stream stream
      multiline_flush_interval 5
      max_bytes 500000
      max_lines 1000
    </match>

    # Concatenate multi-line logs
    <filter **>
      @id filter_concat
      @type concat
      key message
      multiline_end_regexp /\n$/
      separator ""
      timeout_label @NORMAL
      flush_interval 5
    </filter>

    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @id filter_kubernetes_metadata
      @type kubernetes_metadata
    </filter>

    # Fixes json fields in Elasticsearch
    <filter kubernetes.**>
      @id filter_parser
      @type parser
      key_name log
      reserve_time true
      reserve_data true
      remove_key_name_field true
      <parse>
        @type multi_format
        <pattern>
          format json
        </pattern>
        <pattern>
          format none
        </pattern>
      </parse>
    </filter>

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.