No Docker Metrics Available in zabbix
viveklanjekar opened this issue · 12 comments
viveklanjekar commented
below are info from logs
Docker image: monitoringartist/dockbix-agent-xxl-limited
Copyright (C) 2014-2017 Jan Garaj - www.monitoringartist.com
Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.
Starting Zabbix Agent [jira02.sg.aws.grabpay.com]. Zabbix 3.4.2 Dockbix Agent XXL (2017-09-26) (revision 72885).
Press Ctrl+C to exit.
7:20171214:074845.476 Starting Zabbix Agent [jira02.sg.aws.grabpay.com]. Zabbix 3.4.2 Dockbix Agent XXL (2017-09-26) (revision 72885).
7:20171214:074845.478 **** Enabled features ****
7:20171214:074845.478 IPv6 support: YES
7:20171214:074845.478 TLS support: YES
7:20171214:074845.478 **************************
7:20171214:074845.478 using configuration file: /etc/zabbix/zabbix_agentd.conf
7:20171214:074845.478 In zbx_load_modules()
7:20171214:074845.478 loading module "/var/lib/zabbix/modules/zabbix_module_docker.so"
7:20171214:074845.478 In zbx_module_api_version()
7:20171214:074845.478 In zbx_module_init()
7:20171214:074845.478 zabbix_module_docker v0.6.8, compilation time: Sep 26 2017 22:48:15
7:20171214:074845.478 In zbx_docker_dir_detect()
7:20171214:074845.479 Cannot detect used docker driver
7:20171214:074845.479 In zbx_docker_api_detect()
7:20171214:074845.479 In zbx_module_docker_socket_query()
7:20171214:074845.479 Docker's socket query: GET /_ping HTTP/1.0
7:20171214:074845.479 Docker's socket response: OK
7:20171214:074845.479 Docker's socket works - extended docker metrics are available
7:20171214:074845.479 In zbx_module_item_list()
7:20171214:074845.479 In zbx_module_item_timeout()
7:20171214:074845.479 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_docker.so": /var/lib/zabbix/modules/zabbix_module_docker.so: undefined symbol: zbx_module_history_write_cbs
7:20171214:074845.479 loading module "/var/lib/zabbix/modules/zabbix_module_stress.so"
7:20171214:074845.479 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_stress.so": /var/lib/zabbix/modules/zabbix_module_stress.so: undefined symbol: zbx_module_history_write_cbs
7:20171214:074845.479 loading module "/var/lib/zabbix/modules/zabbix_module_systemd.so"
7:20171214:074845.481 [systemd] starting v3.4.2, compiled: Sep 26 2017 22:47:47
7:20171214:074845.481 [systemd] in cgroup_dir_detect()
7:20171214:074845.481 [systemd] detected cgroup mount directory: /rootfs/var/lib/docker/overlay2/b3d89b8f3be768a040338eae39ca78d0ae2453df289ad80bac9665b4d761ea61/merged/sys/fs/cgroup/
7:20171214:074845.481 [systemd] cpu_cgroup is cpuacct
7:20171214:074845.481 [systemd] set timeout to 3000ms
7:20171214:074845.481 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_systemd.so": /var/lib/zabbix/modules/zabbix_module_systemd.so: undefined symbol: zbx_module_history_write_cbs
7:20171214:074845.481 loading module "/var/lib/zabbix/modules/zabbix_module_sockets.so"
7:20171214:074845.481 [sockets] starting v3.4.2
7:20171214:074845.481 [sockets] set timeout to 3000ms
7:20171214:074845.481 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_sockets.so": /var/lib/zabbix/modules/zabbix_module_sockets.so: undefined symbol: zbx_module_history_write_cbs
7:20171214:074845.481 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so, zabbix_module_systemd.so, zabbix_module_sockets.so
7:20171214:074845.481 End of zbx_load_modules():SUCCEED
7:20171214:074845.481 In init_collector_data()
7:20171214:074845.481 In zbx_dshm_create() size:0
7:20171214:074845.481 End of zbx_dshm_create():SUCCEED shmid:-1
7:20171214:074845.481 End of init_collector_data()
7:20171214:074845.481 agent #0 started [main process]
12:20171214:074845.485 agent #3 started [listener #2]
11:20171214:074845.485 agent #2 started [listener #1]
12:20171214:074845.485 In zbx_tls_init_child()
15:20171214:074845.485 agent #5 started [active checks #1]
11:20171214:074845.485 In zbx_tls_init_child()
14:20171214:074845.486 agent #4 started [listener #3]
15:20171214:074845.486 In zbx_tls_init_child()
14:20171214:074845.486 In zbx_tls_init_child()
11:20171214:074845.491 OpenSSL library (version OpenSSL 1.0.2g 1 Mar 2016) initialized
10:20171214:074845.503 agent #1 started [collector]
12:20171214:074845.503 OpenSSL library (version OpenSSL 1.0.2g 1 Mar 2016) initialized
10:20171214:074845.503 In init_cpu_collector()
11:20171214:074845.504 End of zbx_tls_init_child()
15:20171214:074845.504 OpenSSL library (version OpenSSL 1.0.2g 1 Mar 2016) initialized
12:20171214:074845.504 End of zbx_tls_init_child()
10:20171214:074845.504 End of init_cpu_collector():SUCCEED
11:20171214:074845.504 __zbx_zbx_setproctitle() title:'listener #1 [waiting for connection]'
14:20171214:074845.504 OpenSSL library (version OpenSSL 1.0.2g 1 Mar 2016) initialized
15:2017121
the above logs are from debug level 5
also please find the logs From normal logging
14:20171214:074414.510 Cannot open metric file: '/rootfs/var/lib/docker/overlay2/c889f774dc895802e1ae039183bbd1d65228868e480e1a23d4a53be89e71ede9/merged/sys/fs/cgroup/memory/1a732c7feb38d73087ab7b2d2f02ce3bcb148e8f2429bd1a6c0c45b1448a8357/memory.stat'`
docker info output
docker info
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 4
Server Version: 17.03.2-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.43-17.38.amzn1.x86_64
Operating System: Amazon Linux AMI 2017.03
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.5 MiB
Name: jira02.sg.aws.grabpay.com
ID: L75C:365N:J3CD:RGXY:NOTU:WBAA:SOHM:O5ZT:B7PJ:NDLP:W7UX:YUGO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
OS we are using is AWS linux
further debugging output
[root@jira02 ~]# docker exec -ti dockbix-agent-xxl zabbix_get -s 127.0.0.1 -k docker.mem[/dockbix-agent-xxl,rss]
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "no such file or directory"`
help would be appreciated
jangaraj commented
For the record: some problem zbx_docker_dir_detect():
7:20171214:074845.478 In zbx_docker_dir_detect()
7:20171214:074845.479 Cannot detect used docker driver
viveklanjekar commented
any suggestions, what Iam doing wrong? how would I make it work?
jangaraj commented
viveklanjekar commented
is this something related to permissions?
jangaraj commented
No, the problem is in the source code.
viveklanjekar commented
ok thanks
jangaraj commented
Could you provide outputs of these commands, please?
docker run --rm \
--net=host \
--privileged \
-v /:/rootfs \
-v /var/run:/var/run \
-ti ubuntu:16.04 \
bash -c "cat /proc/mounts | grep 'cpuset cgroup'"
docker run --rm \
--net=host \
--privileged \
-v /:/rootfs \
-v /var/run:/var/run \
-ti ubuntu:16.04 \
bash -c "cat /proc/mounts | grep 'cpuset cgroup' | grep -E '^[^[:blank:]]+[[:blank:]]+(/[^[:blank:]]+/)[^[:blank:]]+[[:blank:]]+cgroup[[:blank:]]+.*$'"
viveklanjekar commented
sorry for late response
below is out of command
[root@jira02 ~]# docker run --rm --net=host --privileged -v /:/rootfs -v /var/run:/var/run -ti ubuntu:16.04 bash -c "cat /proc/mounts | grep 'cpuset cgroup'"`
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/overlay2/aab5e773827dd2b811ef4da5c797e3218f8bac0a5b95a633007af1631e9db81d/merged/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
[root@jira02 ~]# docker run --rm --net=host --privileged -v /:/rootfs -v /var/run:/var/run -ti ubuntu:16.04 bash -c "cat /proc/mounts | grep 'cpuset cgroup' | grep -E '^[^[:blank:]]+[[:blank:]]+(/[^[:blank:]]+/)[^[:blank:]]+[[:blank:]]+cgroup[[:blank:]]+.*$'"
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/overlay2/c321de5e9c6d8d2e55477b1f7b4d720e6b1046fb5ea46ad289bc5f00f72cde2c/merged/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
eskogh commented
Any update on this? I got the same error.
Debug output (without --net=host)
root@host1:~# docker run --log-driver json-file --log-opt max-size=500k --name=dockbix-agent-xxl --privileged --restart=unless-stopped --userns=host --hostname=host1 -p 10050:10050 -v /:/rootfs -v /var/run:/var/run -v zabbix-data:/etc/zabbix/zabbix_agentd.d/ --env="ZABBIX_SERVER=10.206.3.98" --env="ZA_ServerActive=10.206.3.98" --env="ZA_Include=/etc/zabbix/zabbix_agentd.d/*.conf" --env="ZA_DebugLevel=5" -d monitoringartist/dockbix-agent-xxl:latest && docker logs -f dockbix-agent-xxl
10:20180125:143511.481 Starting Zabbix Agent [host1]. Zabbix 3.4.2 Dockbix Agent XXL (2017-10-03) (revision 72885).
10:20180125:143511.481 **** Enabled features ****
10:20180125:143511.481 IPv6 support: YES
10:20180125:143511.481 TLS support: YES
10:20180125:143511.481 **************************
10:20180125:143511.481 using configuration file: /etc/zabbix/zabbix_agentd.conf
10:20180125:143511.481 In zbx_load_modules()
10:20180125:143511.481 loading module "/var/lib/zabbix/modules/zabbix_module_docker.so"
10:20180125:143511.481 In zbx_module_api_version()
10:20180125:143511.481 In zbx_module_init()
10:20180125:143511.481 zabbix_module_docker v0.6.8, compilation time: Oct 3 2017 16:55:38
10:20180125:143511.481 In zbx_docker_dir_detect()
10:20180125:143511.482 Detected used docker driver dir: docker/
10:20180125:143511.482 Detected JoinController cpu,cpuacct
10:20180125:143511.482 In zbx_docker_api_detect()
10:20180125:143511.482 In zbx_module_docker_socket_query()
10:20180125:143511.482 Docker's socket query: GET /_ping HTTP/1.0
10:20180125:143511.482 Docker's socket response: OK
10:20180125:143511.482 Docker's socket works - extended docker metrics are available
10:20180125:143511.482 In zbx_module_item_list()
10:20180125:143511.482 In zbx_module_item_timeout()
10:20180125:143511.482 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_docker.so": /var/lib/zabbix/modules/zabbix_module_docker.so: undefined symbol: zbx_module_history_write_cbs
10:20180125:143511.482 loading module "/var/lib/zabbix/modules/zabbix_module_stress.so"
10:20180125:143511.482 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_stress.so": /var/lib/zabbix/modules/zabbix_module_stress.so: undefined symbol: zbx_module_history_write_cbs
10:20180125:143511.482 loading module "/var/lib/zabbix/modules/zabbix_module_systemd.so"
10:20180125:143511.483 [systemd] starting v3.4.2, compiled: Oct 3 2017 16:55:06
10:20180125:143511.483 [systemd] in cgroup_dir_detect()
10:20180125:143511.483 [systemd] detected cgroup mount directory: /rootfs/sys/fs/cgroup/
10:20180125:143511.483 [systemd] cpu_cgroup is JoinController cpu,cpuacct
10:20180125:143511.483 [systemd] set timeout to 3000ms
10:20180125:143511.483 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_systemd.so": /var/lib/zabbix/modules/zabbix_module_systemd.so: undefined symbol: zbx_module_history_write_cbs
10:20180125:143511.483 loading module "/var/lib/zabbix/modules/zabbix_module_sockets.so"
10:20180125:143511.483 [sockets] starting v3.4.2
10:20180125:143511.483 [sockets] set timeout to 3000ms
10:20180125:143511.483 cannot find "zbx_module_history_write_cbs()" function in module "zabbix_module_sockets.so": /var/lib/zabbix/modules/zabbix_module_sockets.so: undefined symbol: zbx_module_history_write_cbs
10:20180125:143511.483 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so, zabbix_module_systemd.so, zabbix_module_sockets.so
Docker info:
root@host1:~# docker info
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 3
Server Version: 1.12.6
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 52
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-75-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.859 GiB
Name: host1
ID: OY4J:4NWQ:UF7A:5EWP:TGY7:6V3G:RHLC:LA3I:DDEP:3YSG:JNH6:YPPA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: XXXXXXX
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
cat /proc/mounts
root@host1:~# docker run --rm --net=host --privileged -v /:/rootfs -v /var/run:/var/run -ti ubuntu:16.04 bash -c "cat /proc/mounts | grep 'cpuset cgroup'"
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/aufs/mnt/5cbb3edc744866e09739f94a39ca84919c415aa2dec4c1c435dadf99c1cb323e/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cat /proc/mounts | grep
root@host1:~# docker run --rm --net=host --privileged -v /:/rootfs -v /var/run:/var/run -ti ubuntu:16.04 bash -c "cat /proc/mounts | grep 'cpuset cgroup' | grep -E '^[^[:blank:]]+[[:blank:]]+(/[^[:blank:]]+/)[^[:blank:]]+[[:blank:]]+cgroup[[:blank:]]+.*$'"
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /rootfs/var/lib/docker/aufs/mnt/2f7d7ba5f2a372dd54631fccaff3f62621c2cbbad291acb3d356321fd8a0bd81/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
viveklanjekar commented
@eskogh for now iam using older version of agent, it works
eskogh commented
@viveklanjekar do you wanna share what version you're using so can I verify that?
viveklanjekar commented
@eskogh use this version docker pull monitoringartist/dockbix-agent-xxl-limited:3.2-2