FluentBit pods go into CrashLoopBackOff

[steveellis]

My goal is to use this chart to deploy FluentBit to forward logs to CloudWatch logs as described in the docs here.

I'm setting my values to the following:

• adotCollector.daemonSet.service.metrics.receivers is awscontainerinsightreceiver
• adotCollector.daemonSet.service.metrics.exporters is awsemf.

Note these values are slightly different than what the doc referenced above says they should be, but I believe these are the correct ones. I've tried the other ones too of course. Its unclear to me if the adotcollector is only metrics or has something to do with logs as well.

I see that the collector has quite a bit of variety when it comes to components and collectors, processors etc. but I'm lost as to what the magic combination might be.

With the above config thefluent-bit pods fail to start and enter CrashLoopBackOff. Doing kubectl logs <fluent bit podname> yields the following but I have no way to debug this as I can't connect to the pod to view that application-log.conf file.

Fluent Bit v1.8.9
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

Error: Configuration file contains errors. Aborting

AWS for Fluent Bit Container Image Version 2.21.1[2022/04/08 16:52:33] [  Error] File application-log.conf
[2022/04/08 16:52:33] [  Error] Error in line 58: Key has an empty value

Questions

• Is the FluentBit portion of this chart known to work for my use case? My goal is to get the application logs for my pods forwarded to CloudWatch Logs in addition to getting the metrics. So metrics and logs. BTW the metrics pods are deploying fine.
• Is there a values change I can make to get logs/FB going? Perhaps I need to set something in the fluentBit portion of values. I have fluentBit.enabled set to true but nothing else seems like something I should set and I don't see anything in the docs about that.

Regarding the prerequisites, I have bound two worker role managed policies to my nodes: CloudWatchLogsFullAccess and CloudWatchAgentServerPolicy but since I'm able to see the metrics pods start up and send their metrics to CloudWatch perhaps this part is fine.

[sunnygoel87]

@steveellis , ADOT collector only supports metrics and tracing for now. Fluent-Bit agent is responsible for capturing and forwarding the logs to CloudWatch service.

Are you setting(either in values.yaml file or as parameters in helm install command itself) the ClusterName and Region while trying to install the Helm chart ?

[steveellis]

Thanks @sunnygoel87 for taking a look. Turns out I was passing in an undefined value for aws-region (hence the key has empty value message) which became clear when I noticed the fluentbit config file mentioned in my log is defined here.