enram/data-repository

File transfer: make sure the EC2 machine doesn't get out of disk space

Closed this issue · 8 comments

niconoe commented

... that happened on June 20th, 2020, blocking further transfers. Fortunately we got informed thanks to a mail alert.

It appears a local copy of files happen on this machine, investigate if this is really necessary?

niconoe commented

I'd like to self-assign this issue, but can't now. @peterdesmet : can you give me some rights on this repo? Thanks!

niconoe commented

Update: disk appears full not because of size, but because the number of inodes gets exceeded. A reboot temporary solved it (tmp files deleted?):

$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
udev           124436    357 124079    1% /dev
tmpfs          126758    446 126312    1% /run
/dev/xvda1     786432 786401     31  100% /
tmpfs          126758      1 126757    1% /dev/shm
tmpfs          126758      3 126755    1% /run/lock
tmpfs          126758     16 126742    1% /sys/fs/cgroup
tmpfs          126758      4 126754    1% /run/user/1000
niconoe commented

Update: I was able to remove a tons of old kernel sources trees in /usr/src following this method, there's now a tons of free inodes: https://ubuntugenius.wordpress.com/2011/01/08/ubuntu-cleanup-how-to-remove-all-unused-linux-kernel-headers-images-and-modules/

Next step: reboot the instance to make sure it still has the needed kernel (but all looks right: menu.lst, vmlinuzimage, ...) => DONE

peterdesmet commented

Thanks for investigating Nico, you now have maintainer rights on this repo

adokter commented

I've run into inodes issues too recently on my local machine, because of the large number of small size when storing vp files for each radar over a long time series. I had to reformat the disk and assign more inodes to fix it:

mkfs.ext4 -T news /dev/device

formatting with this statement gives you one inode for every 4096 bytes, instead of default 16384 bytes.

  • 👍1
niconoe commented

Thanks @adokter, in that case it was just old linux (kernel) sources in /usr/src that were taking 90%+ of inodes, but that's good to know for the future!

niconoe commented

TODO:

Use AWS features to make sure I am informed by email next time the disk space/number of free inodes get low.

niconoe commented

Server back and running, and the appropriate CloudWatch have been set so I'll be informed the next time we get low on disk space or free inodes.