The memory usage of "docker stats" is not equal with the summary of the memory usage of all processes in the docker container
samdai opened this issue · 7 comments
I run a Postgesql server in a docker container, when I run command "docker stats --no-stream", the memory usage of this container is 495.2MB as below:
c137012d8af9 0.00% 495.2 MB / 67.55 GB 0.73% 24.66 GB / 29.5 GB 0 B / 49.75 GB 0
But when I enter this container using command "docker exec -ti c137012d8af9 /bin/bash", then execute command "/usr/bin/top -bcn 1", the result is as below, the summation of the memory usage of all processes is about 381MB, the memory usage of "docker stats" is not equal with the summation of the memory usage of all processes in the docker container. Which one is correct?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 10 -10 17976 2816 2580 S 0.0 0.0 0:00.02 /bin/bash /scripts/run.sh /bin/sh -c chmod 775 /usr/bin/jq
7 root 10 -10 47120 3072 2688 S 0.0 0.0 0:00.00 sudo -u postgres /usr/lib/postgresql/9.6/bin/postgres -D /data
8 postgres 10 -10 271644 33736 32456 S 0.0 0.1 0:05.76 /usr/lib/postgresql/9.6/bin/postgres -D /data
10 postgres 10 -10 271956 142604 141084 S 0.0 0.2 0:11.32 postgres: checkpointer process
11 postgres 10 -10 271644 138852 137544 S 0.0 0.2 0:52.12 postgres: writer process
12 postgres 10 -10 271644 8064 6760 S 0.0 0.0 0:16.37 postgres: wal writer process
13 postgres 10 -10 272368 17524 15944 S 0.0 0.0 0:01.25 postgres: autovacuum launcher process
14 postgres 10 -10 107476 3876 2364 S 0.0 0.0 1:08.22 postgres: stats collector process
10003 root 10 -10 18140 3276 2828 S 0.0 0.0 0:00.02 /bin/bash
10564 root 10 -10 18140 3272 2820 S 0.0 0.0 0:00.02 /bin/bash
10952 root 10 -10 18140 3248 2792 S 0.0 0.0 0:00.02 /bin/bash
11392 root 10 -10 18140 3284 2832 S 0.0 0.0 0:00.02 /bin/bash
11524 root 10 -10 18140 3136 2852 S 0.0 0.0 0:00.02 /bin/bash
11539 root 10 -10 19740 2316 2056 R 0.0 0.0 0:00.00 /usr/bin/top -bcn 1
The docker version is 1.11.0
I have same issue for my container with chrome driver crawler.
my result
docker stats says my container consumes 2.498 GiB.
# docker stats 8804e404c26f
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
8804e404c26f 87.20% 2.546 GiB / 7.31 GiB 34.83% 0 B / 0 B 21.7 MB / 1.31 GB 66
But by checking from inside container, sum of RES is about 300MiB, so docker stats seems not correct.
# top (by docker exec --ti 8804e404c26f bash)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18143 root 20 0 1074.2m 143.0m 67.2m R 53.8 1.9 0:11.79 chrome
18089 root 20 0 584.6m 76.7m 61.6m S 4.7 1.0 0:01.96 chrome
18117 root 20 0 396.0m 46.8m 38.0m S 0.0 0.6 0:00.04 chrome
8 root 20 0 455.2m 44.4m 7.9m S 1.6 0.6 232:02.05 python
18098 root 20 0 344.1m 39.2m 31.2m S 0.0 0.5 0:00.02 chrome
18081 root 20 0 131.2m 13.8m 11.2m S 7.8 0.2 0:02.52 chromedriver
18158 root 20 0 21.5m 3.6m 3.1m S 0.0 0.0 0:00.00 bash
18162 root 30 10 23.1m 2.7m 2.3m R 0.0 0.0 0:00.00 top
1 root 20 0 4.2m 0.8m 0.7m S 0.0 0.0 0:02.35 sh
18095 root 20 0 5.9m 0.7m 0.6m S 0.0 0.0 0:00.00 cat
18096 root 20 0 5.9m 0.7m 0.6m S 0.0 0.0 0:00.00 cat
some investigation
According to #10824 and docker/cli#80, docker stats returns cgroup memory's usage_in_bytes - cache as RSS.
# cat memory.usage_in_bytes
2746667008 <== 2.619 GiB
# cat memory.stat
cache 27631616 <== 26 MiB
rss 177905664 <== 169 MiB
rss_huge 0
mapped_file 11689984
dirty 16384
writeback 0
swap 0
pgpgin 9180483715
pgpgout 9180433535
pgfault 9259023601
pgmajfault 377
inactive_anon 11468800
active_anon 177905664
inactive_file 4636672
active_file 11526144
unevictable 0
hierarchical_memory_limit 6028402688
hierarchical_memsw_limit 9223372036854771712
total_cache 27631616
total_rss 177905664
total_rss_huge 0
total_mapped_file 11689984
total_dirty 16384
total_writeback 0
total_swap 0
total_pgpgin 9180483715
total_pgpgout 9180433535
total_pgfault 9259023601
total_pgmajfault 377
total_inactive_anon 11468800
total_active_anon 177905664
total_inactive_file 4636672
total_active_file 11526144
total_unevictable 0
By looking /sys/fs/cgroup/memory/...<pid>/, memory.usage_in_bytes is much bigger than memory.stat's rss + cache,
so usage_in_bytes - cache won't match to rss.
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt says
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some optimization
to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz
value for efficient access. (Of course, when necessary, it's synchronized.)
If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
value in memory.stat(see 5.2).
Possibly docker stats should use stats's rss+cache directly, rather than usage_in_bytes - cache ? (especially for long running container ?)
Output of docker info:
Containers: 39
Running: 32
Paused: 0
Stopped: 7
Images: 134
Server Version: 17.03.2-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 595e75c212d19a81d2b808a518fe1afc1391dad5 (expected: 4ab9917febca54791c5f071a9d1f404867857fcc)
runc version: 54296cf (expected: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe)
init version: v0.13.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.111+
Operating System: Container-Optimized OS from Google
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.31 GiB
Name: gke-prd-cluster-4-pool-201706-e17094d4-lt74
ID: HXMQ:27QQ:IUFP:LGF7:U67C:HYT3:UZJG:XSTD:BX3K:47XM:YYNF:Z5OV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
10.0.0.0/8
127.0.0.0/8
Registry Mirrors:
https://mirror.gcr.io/
https://mirror.gcr.io/
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
- Running on GKE Kubernetes 1.9.6-gke.1 with Container-Optimized OS (cos)
BUILD_ID=10323.69.0
NAME="Container-Optimized OS"
KERNEL_COMMIT_ID=94bdaa2af49562e7aba0168cae050911d105f5cb
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=65
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Container-Optimized OS from Google"
VERSION=65
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=cos
I run php-fpm ,
but I have the same problem and I found the /sys/fs/cgroup/memory/memory.kmem.usage_in_bytes value in docker was match with the stats number.
I am still finding why ...
I find the cause issue and solution:
https://support.mesosphere.com/s/article/Critical-Issue-KMEM-MSPH-2018-0006
The docker cli also subtracts shared memory from the value before it is displayed.
I'm going to close this because it seems like a non-issue.
Thanks!
@yokomotod
It has been some time, but, related to issue you described about chrome driver container presumably consuming more memory than reported, did you ever solve the issue or had some insight why it happened? I have somewhat similar issue with headless chrome running in container, its used to generate PDF's, however, over time memory consumption reported by docker stats just grows and grows and its never freed, even though RSS + CACHE reports way lower values and well within limits defined for container.
so how to get the actual memory usage of containers in kubernetes ?
so how to get the actual memory usage of containers in kubernetes ?
Here's my solustion: https://github.com/docker/cli/blob/e57b5f78de635e6e2b688686d10b830c4747c4dc/cli/command/container/stats_helpers.go#L239