Warn when /var disk space is limited
tstromberg opened this issue · 8 comments
To replicate:
- Set Docker to 8GB
- Gather minikube dependencies:
minikube start --download-only - Fill the Docker image cache up until
docker system dfis full. If you need help doing so, I found this works:
seq 32 1 | xargs -n1 -I{} docker pull fedora:{}
seq 8 1 | xargs -n1 -I{} docker pull oraclelinux:{}
seq 20 1 | xargs -n1 -I{} echo docker pull ubuntu:{}:10
seq 20 1 | xargs -n1 -I{} echo docker pull ubuntu:{}:04
You'll probably see at this point errors, such as no space left on device -- that's a good thing!
- Run
minikube start
Here's what I see
Notice that there is no warning:
😄 minikube v1.12.2 on Darwin 10.15.5
✨ Automatically selected the docker driver. Other choices: hyperkit, vmware, parallels, virtualbox, vmwarefusion
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=3892MB) ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube"
/var is completely full!
% minikube ssh "df -h"
Filesystem Size Used Avail Use% Mounted on
overlay 7.9G 7.8G 0 100% /
tmpfs 64M 0 64M 0% /dev
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/vda1 7.9G 7.8G 0 100% /var
tmpfs 2.0G 8.5M 2.0G 1% /run
tmpfs 2.0G 12K 2.0G 1% /tmp
/dev/root 390M 390M 0 100% /usr/lib/modules
tmpfs 5.0M 0 5.0M 0% /run/lock
node shows as ready:
NAME STATUS ROLES AGE VERSION
minikube Ready master 3m21s v1.18.3
Describe shows no disk pressure:
Name: minikube
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=minikube
kubernetes.io/os=linux
minikube.k8s.io/commit=be7c19d391302656d27f1f213657d925c4e1cfc2-dirty
minikube.k8s.io/name=minikube
minikube.k8s.io/updated_at=2020_08_18T12_26_43_0700
minikube.k8s.io/version=v1.12.2
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 18 Aug 2020 12:26:40 -0700
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: minikube
AcquireTime: <unset>
RenewTime: Tue, 18 Aug 2020 12:29:54 -0700
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 18 Aug 2020 12:26:55 -0700 Tue, 18 Aug 2020 12:26:36 -0700 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 18 Aug 2020 12:26:55 -0700 Tue, 18 Aug 2020 12:26:36 -0700 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 18 Aug 2020 12:26:55 -0700 Tue, 18 Aug 2020 12:26:36 -0700 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 18 Aug 2020 12:26:55 -0700 Tue, 18 Aug 2020 12:26:55 -0700 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.17.0.3
Hostname: minikube
Capacity:
cpu: 2
ephemeral-storage: 8190392Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4035432Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 8190392Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4035432Ki
pods: 110
System Info:
Machine ID: b95ed2bfb4c74ec9b5b19a9e71395509
System UUID: 5e5d6d07-9c65-40ac-90e3-2e5e74257084
Boot ID: 2a9ce373-3ed6-4829-acc5-3a49f09ddb6f
Kernel Version: 4.19.76-linuxkit
OS Image: Ubuntu 20.04 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.8
Kubelet Version: v1.18.3
Kube-Proxy Version: v1.18.3
Non-terminated Pods: (7 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-66bff467f8-4vm7b 100m (5%) 0 (0%) 70Mi (1%) 170Mi (4%) 3m15s
kube-system etcd-minikube 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m18s
kube-system kube-apiserver-minikube 250m (12%) 0 (0%) 0 (0%) 0 (0%) 3m18s
kube-system kube-controller-manager-minikube 200m (10%) 0 (0%) 0 (0%) 0 (0%) 3m18s
kube-system kube-proxy-k6lnp 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m15s
kube-system kube-scheduler-minikube 100m (5%) 0 (0%) 0 (0%) 0 (0%) 3m18s
kube-system storage-provisioner 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m18s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 650m (32%) 0 (0%)
memory 70Mi (1%) 170Mi (4%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 3m20s kubelet, minikube Starting kubelet.
Normal NodeHasSufficientMemory 3m20s kubelet, minikube Node minikube status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3m20s kubelet, minikube Node minikube status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 3m20s kubelet, minikube Node minikube status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 3m19s kubelet, minikube Updated Node Allocatable limit across pods
Warning readOnlySysFS 3m14s kube-proxy, minikube CRI error: /sys is read-only: cannot modify conntrack limits, problems may arise later (If running Docker, see docker issue #24000)
Normal Starting 3m14s kube-proxy, minikube Starting kube-proxy.
Normal NodeReady 3m9s kubelet, minikube Node minikube status is now: NodeReady
It seems that the kubelet is not configured to track /var.
After starting minikube, docker system df shows:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 30 1 5.698GB 4.745GB (83%)
Containers 1 1 3.16MB 0B (0%)
Local Volumes 1 1 1.348GB 0B (0%)
Build Cache 0 0 0B 0B
Note that it doesn't say 100% - that's because minikube is running. Once you stop minikube it will say 100%:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 30 0 5.698GB 5.698GB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
FWIW, I've helped 3 different people this last week with this issue .. hence the priority of soon.
Also, this is unlikely to be specific to Docker or KIC, but it's almost certainly exacerbated by the fact that /var shares disk space with other images.
minikube start fails for me when /var is filled:
$ minikube start --driver docker --memory 2000
😄 minikube v1.12.3 on Darwin 10.15.6
✨ Using the docker driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=2000MB) ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
❌ Unable to load cached images: loading cached images: transferring cached image: sudo test -d /var/lib/minikube/images && sudo scp -t /var/lib/minikube/images && sudo touch -d "2020-07-07 22:39:55.97626885 -0400" /var/lib/minikube/images/pause_3.2: Process exited with status 1
output: scp: /var/lib/minikube/images/pause_3.2: No space left on device
💣 Failed to update cluster: updating control plane: copy: mkdir: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube: Process exited with status 1
stdout:
stderr:
mkdir: cannot create directory '/etc/systemd/system/kubelet.service.d': No space left on device
😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose
It would be nice if we could give the user a nicer error message when the "No space left on device" error comes up.
I think we should also warn them ASAP, which would be as soon as the container is created. This means we'll have to go for the "minikube ssh" approach since k8s won't be up yet.
I think we should also warn them ASAP, which would be as soon as the container is created. This mean
thats a good idea, I agree
X Docker is nearly out of disk space, which may cause deployments to fail! (98% of capacity). You can pass '--force' to skip this check.
-
Suggestion:
Try one or more of the following to free up space on the device:
- Run "docker system prune" to remove unused Docker data (optionally with "-a")
- Increase the storage allocated to Docker for Desktop by clicking on:
Docker icon > Preferences > Resources > Disk Image Size - Run "minikube ssh -- docker system prune" if using the Docker container runtime
-
Related issue: #9024
-
Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
-
Configuring Calico (Container Networking Interface) ...
-
Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
-
Enabled addons: storage-provisioner, default-storageclass
-
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Run "docker system prune" to remove unused Docker data (optionally with "-a")
Does not work. I get ZERO bytes reclaimed.
ropaflame@1dropaflame:~$ docker system prune
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
Update:
I had to do a
minikube ssh docker system prune
and then it worked
python hello