aws/aws-for-fluent-bit

/var/lib/docker/containers mount in example can removed when EKS removes support for < 1.24 k8s [docker => containerd]

Opened this issue · 9 comments

radubarbos commented

Describe the question/issue

We wanted to migrate some apps to a kubernetes cluster using version 1.24.
The deploy fails because we are mounting the /var/lib/docker/containers and this folder does not exist as kubernetes 1.24 does not use docker runtime anymore.
For what is this used
Some of your examples in the use_cases are mounting this folder.
What is the purpose of it?

Configuration

Fluent Bit Log Output

Fluent Bit Version Info

Cluster Details

EKS on kubernetes 1.24 version.

Application Details

Steps to reproduce issue

Related Issues

Some of the use_cases

radubarbos commented

It seems the fluent-bit code uses this in case the 'docker' input plugin is enabled.
https://docs.fluentbit.io/manual/pipeline/inputs/docker-metrics

This can be closed.

PettitWesley commented

So at least in docker, we can mount paths that don't exist:

$ docker run -it -v /var/lib/doesnotexist:/var/lib/doesnotexit public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
... runs fine.. no error...
PettitWesley commented

I followed the steps here on a new eksctl created 1.25 kubernetes cluster: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html#Container-Insights-FluentBit-troubleshoot

It all worked fine, Fluent Bit was deployed. The config map still has the same /var/lib/docker/containers mount: https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml

@radubarbos What error did you see? Please give us the output. Thanks.

PettitWesley commented

Once EKS removes support for k8s < 1.24 we can update the examples to remove that volume.

radubarbos commented

The error that I got was that the volume /var/lib/docker/containers could not be mounted as it does not exits.
I see what is the difference, we had the k8s volume defined as 'Directory'

  • name: docker-logs
    hostPath:
    type: Directory
    path: "/var/lib/docker/containers"

but in your working example the volume has no type declared and there is no check if that exists or not.
What I found out is that with k8s 1.24 the actual log files are under /var/log/pods, linked from var/log/containers.
Also looking at the fluentbit repo I see that the /var/lib/docker/containers volume is needed if the 'docker' input plugin is set.

It's clear now why was the problem, this issue can be closed.

PettitWesley commented

I want to keep it open as a note that we could consider updating the example once EKS removes support for Docker k8s.

Or may be we'll just change the tutorial since non-EKS users might keep using Docker k8s, I'm not sure.