Getting kubernetes log input using fluent-bit-go-s3 docker image

[aroes]

Problem

I'm trying to get pod logs from kubernetes and output them to S3. The official fluent-bit helm chart has the correct configuration by default (using the tail INPUT plugin in a daemonset), so I've modified it to use your docker image instead of the official one (so the deamonset image is cosmo0920/fluent-bit-go-s3:v0.7.2 rather than fluent/fluent-bit:1.3.7, and env variables are added so the s3 output plugin can access the bucket).

The problem is, when using the fluent-bit-go-s3 image, I only get cpu logs for some reason, which you're meant to get when using the CPU metric input plugin. Below is an example of a log message:
{"system_p":3,"cpu0.p_system":3,"cpu1.p_system":2,"cpu0.p_user":14,"cpu1.p_user":9,"cpu_p":14.499999999999998,"cpu1.p_cpu":11,"user_p":11.5,"cpu0.p_cpu":17}

Steps to replicate

From official fluent bit helm chart, change the deamonset image to use cosmo0920/fluent-bit-go-s3:v0.7.2, and add required environment variables.

Expected Behavior or What you need to ask

I would expect the fluent-bit-s3 image to output tail kubernetes logs in the same way the official image does, as the input config is the same in both cases. I wonder if you know what might be going wrong? Any help you could offer would be much appreciated!

Using Fluentd and S3 plugin versions

• Kubernetes (helm)
• Fluent Bit v1.3.9
• cosmo0920/fluent-bit-go-s3:v0.7.2

Kubernetes log for one of the pods:

  Fluent Bit v1.3.9
Copyright (C) Treasure Data

[2020/09/03 12:59:06] [ info] [storage] version=1.0.1, initializing...
[2020/09/03 12:59:06] [ info] [storage] in-memory
[2020/09/03 12:59:06] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/09/03 12:59:06] [ info] [engine] started (pid=1)
[2020/09/03 12:59:06] [ info] [s3operator] id = 0
[2020/09/03 12:59:06] [ info] [flb-go 0] Starting fluent-bit-go-s3: (version=0.7.2, branch=master, revision=77750b7)
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin credential parameter = ''
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin accessKeyID parameter = 'A...S'
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin secretAccessKey parameter = 'Z...W'
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin bucket parameter = 'xx'
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin s3prefix parameter = 'fluent-bit'
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin suffixAlgorithm parameter = ''
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin region parameter = 'xx'
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin compress parameter = ''
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin endpoint parameter = ''
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin autoCreateBucket parameter = ''
[2020/09/03 12:59:06] [ info] [flb-go 0] plugin timeZone parameter = ''
[2020/09/03 12:59:06] [ warn] [flb-go 0] Not using suffix algorithm will cause object key collision. Please consider to use `suffixAlgorithm sha256`.
[2020/09/03 12:59:06] [ info] [sp] stream processor started

[cosmo0920]

This is because this repository's fluent-bit image uses in_cpu input:
https://github.com/cosmo0920/fluent-bit-go-s3/blob/master/docker/fluent-bit-s3.conf
You must write your in_tail pattern configuration with ConfigMap. And this configuration is out of scope of this issue.

[aroes]

Thanks, the problem was I didn't notice your dockerfile runs fluentbit with fluent-bit-s3.conf, which means the config generated by the official helm chart was ignored.