Failed to read textfile collector directory

[rohith-vallabhaneni]

Hi

I have a kubernetes cluster running in Amazon EKS, we are using the prometheus node-exporter docker image and running it as daemonset across all k8s nodes to fetch the node metrics. We had a use case to monitor the docker volumes usage on each node, I have written a shell script to fetch the data in the prometheus readable format

#! /bin/bash
hostip=$(curl -fs http://169.254.169.254/latest/meta-data/local-ipv4)

voulume_size=`df -Ph /var/lib/docker | awk '{if(NR>1)print $2}' | sed 's/G//g'`
voulume_usage=`df -Ph /var/lib/docker | awk '{if(NR>1)print $5}' | sed 's/%//g'`

`mkdir -p /home/ec2-user/node_exporter/textfile_collector`
`rm -f /home/ec2-user/node_exporter/textfile_collector/docker_volume.prom`

echo "# HELP _docker_volume_usage hostip usage." >> /home/ec2-user/node_exporter/textfile_collector/docker_volume.prom 2>&1
echo "# TYPE _docker_volume_usage gauge" >> /home/ec2-user/node_exporter/textfile_collector/docker_volume.prom 2>&1

echo "_docker_volume_usage{hostip=\"$hostip\",host=\"`hostname`\"} $voulume_usage" >> /home/ec2-user/node_exporter/textfile_collector/docker_volume.prom 2>&1


echo "# HELP _docker_volume_size hostip size." >> /home/ec2-user/node_exporter/textfile_collector/docker_volume.prom 2>&1
echo "# TYPE _docker_volume_size gauge" >> /home/ec2-user/node_exporter/textfile_collector/docker_volume.prom 2>&1

echo "_docker_volume_size{hostip=\"$hostip\",host=\"`hostname`\"} $voulume_size" >> /home/ec2-user/node_exporter/textfile_collector/docker_volume.prom 2>&1

I'm running the script on each node and adding the data to
/home/ec2-user/node_exporter/textfile_collector/docker_volume.prom file.

On the docker logs I see the below ERROR:

level=error ts=2021-01-21T08:44:45.125Z caller=textfile.go:197 collector=textfile msg="failed to read textfile collector directory" path=/home/ec2-user/node_exporter/textfile_collector err="open /home/ec2-user/node_exporter/textfile_collector: no such file or directory"

Let me know where am I going wrong, I now had a doubt that whether the textfile-collector from the docker container colect data from the node in which the container is running!

Any help over here would be appreciated.
Thanks,
Rohith Vallabhaneni.

[dswarbrick]

This sounds like a generic host system administration issue, and not something specific to the textfile collector(s) or even node_exporter. Looking at the error message you pasted, node_exporter is reporting that it cannot read the textfile collector directory, because it doesn't exist.

Have you verified that the directory is indeed being created by the environment?

[rohith-vallabhaneni]

@dswarbrick the directory is not getting created inside the container, the node_exporter expectes the textfile collector drectory to be availabile inside the container, the issue is fixed by mounting the volume in the k8s worker node to the container.
Please find the code snippent from the helm template that does the mount

  extraHostVolumeMounts:
    - name: text-file-collector
      hostPath: /home/ec2-user/node_exporter/textfile_collector
      mountPath: /var/lib/node_exporter/textfile_collector
      readOnly: true
      mountPropagation: HostToContainer
  extraArgs:
    - --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

with the above, the issue got fixed, thanks