Limit log size on GitLab runners
Closed this issue · 3 comments
Logs (syslog and kern.log) on GitLab runners are reaching 100% disk, which causes the machine to break.
LogRotate to be installed on all hosts with root.
logrotate is already installed and running on our hosts. The issue is that, on the gitlab runner, syslog and kern.log are exploding the disk before they can be rotated. /var/log/kern.log is currently set to rotate weekly with 4 backups, /var/log/syslog rotates daily with 7 backups (both sets of backups are compressed).
I'll configure logrotate to deal with these logs more frequently. However, there is still an underlying issue here that something is logging way too much - these log files are growing much faster than those on any other host that I've looked at.
Implemented in db7157a.
The underlying issue of whatever is causing the excesive logging still remains.