Monitoring with node-exporter
edusouzaxx opened this issue · 1 comments
RancherOS Version: (ros os version)
v1.5.5
Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
Vmware
Why node-exporter does not take metrics from additional filesystems in RancherOS
RancherOS version
cat /etc/os-release
NAME="RancherOS"
VERSION=v1.5.5
ID=rancheros
ID_LIKE=
VERSION_ID=v1.5.5
PRETTY_NAME="RancherOS v1.5.5"
HOME_URL="http://rancher.com/rancher-os/"
SUPPORT_URL="https://forums.rancher.com/c/rancher-os"
BUG_REPORT_URL="https://github.com/rancher/os/issues"
BUILD_ID=
mounts configuration
ros config set mounts '[["/dev/sdb1","/var/lib/user-docker","ext4",""],["/dev/sdb2","/data","ext4",""],["/dev/sdb3","/var/lib/kubelet","ext4",""],["/dev/sdb4","/opt/rke","ext4",""]]'
filesystems
df -h
Filesystem Size Used Avail Use% Mounted on
overlay 4.7G 1.9G 2.6G 43% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 4.7G 1.9G 2.6G 43% /mnt
none 16G 1.1M 16G 1% /run
shm 64M 0 64M 0% /dev/shm
/dev/sdb1 98G 600M 93G 1% /var/lib/user-docker
/dev/sdb2 49G 53M 47G 1% /data
/dev/sdb3 9.8G 37M 9.3G 1% /var/lib/kubelet
/dev/sdb4 9.8G 37M 9.3G 1% /opt/rke
node-exporter service configuration
node_exporter:
image: prometheus/node-exporter:v0.18.1
command: |
--path.procfs=/host/proc
--path.sysfs=/host/sys
--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$$
--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($$|/)
--collector.ksmd
--collector.filesystem
--collector.mountstats
--path.rootfs=/host/root
--web.listen-address=0.0.0.0:9100
--log.level=debug
privileged: true
labels:
io.rancher.os.scope: "system"
io.rancher.os.after: "docker"
io.rancher.os.detach: "true"
ports:
- "9100:9100/tcp"
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/host/root:ro
restart: unless-stopped
metrics
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP node_filesystem_free_bytes Filesystem free space in bytes.
TYPE node_filesystem_free_bytes gauge
node_filesystem_free_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 2.98029056e+09
node_filesystem_free_bytes{device="none",fstype="tmpfs",mountpoint="/run"} 1.6853835776e+10
100 57003 0 57003 0 0 1562k 0 --:--:-- --:--:-- --:--:-- 1637k
problem
I would like to receive metrics for the filesystems:
/data
/var/lib/user-docker
/var/lib/kubelet
/opt/rke
but I only have metrics from filesystem:
/
The workaround I found:
Create the mountpoints under /mnt:
By default, /media and /mnt are mounted as shared in the console container. This means that mounts within these directories will propagate to the host as well as other system services that mount these folders as shared.
ros config set mounts '[["/dev/sdb1","/mnt/var/lib/docker","ext4",""],["/dev/sdb2","/mnt/data","ext4",""],["/dev/sdb3","/mnt/var/lib/kubelet","ext4",""],["/dev/sdb4","/mnt/opt/rke","ext4",""]]'
And create symbolic links on cloud-config like:
write_files:
path: /etc/rc.local
permissions: “0755”
owner: root
content: |
#!/bin/bash
ln -s /mnt/data /data
ln -s /mnt/var/lib/kubelet /var/lib/kubelet
ln -s /mnt/opt/rke /opt/rke