Logrotate filled up all disk space (60Gb) in a single day
Opened this issue · 5 comments
christopheSeeka commented
Hello,
Few days ago, my webapp stopped work, i checked on the server and it was stopped in pm2 and didn't want restart.
I figured that the disk was full (25Gb) so i took an upgrade to 80Gb and installed logrotate but today, 2 days after, the disk is full again.
I did a flush logs command, which did bring it from 80Gb to 61Gb.
Then i figured that the .pm2/logs folder was 57Gb and i see 150 log files like "sign-out__2021-02-13_05-36-03.log" for the app for a single day, how can i fix that?
Here my logrotate default config:
"pm2-logrotate": {
"max_size": "10M",
"retain": "30",
"compress": false,
"dateFormat": "YYYY-MM-DD_HH-mm-ss",
"workerInterval": "30",
"rotateInterval": "0 0 * * *",
"rotateModule": true
},
"module-db-v2": {
"pm2-logrotate": {}
}
}
If i understand it well it shouldn keep more than 30 of 10Mb max so 57Gb is not normal.
Deleted all sign-out__2021-02-13_05-36-*.log files and down to 8Gb
I stopped logrotate for now and its ok but would like to know how to solve this issue
Thank you in advance
lindev0915 commented
This is not issue.
Your configuration is not prefer.
pm2 set pm2-logrotate:retain 50
It will keep only 50 logfiles, If the 51st file will try to perform the logrtate then it will delete the olf log file.
pm2 set pm2-logrotate:size 300M
Once the log file reached the size of 300M then the pm2-logrotate will work
You can avoid disk full issue by using these two configurations
christopheSeeka commented
Thank you very much for the answer, but isn't the config i shared saying 10Mb per log file and retain 30 max already?
"max_size": "10M",
"retain": "30",
How this ended filling out the entire disk in one day?
matthewmayer commented
i had a similar issue. New files created every 30 seconds of increasing size:
-rw-rw-r-- 1 ec2-user ec2-user 136M Mar 19 06:39 out-0__2021-03-19_06-09-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 117M Mar 19 06:39 out-0__2021-03-19_06-09-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 98M Mar 19 06:39 out-0__2021-03-19_06-10-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 90M Mar 19 06:39 out-0__2021-03-19_06-10-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 85M Mar 19 06:39 out-0__2021-03-19_06-11-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 81M Mar 19 06:39 out-0__2021-03-19_06-11-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 77M Mar 19 06:39 out-0__2021-03-19_06-12-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 75M Mar 19 06:39 out-0__2021-03-19_06-12-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 73M Mar 19 06:39 out-0__2021-03-19_06-13-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 71M Mar 19 06:39 out-0__2021-03-19_06-13-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 67M Mar 19 06:39 out-0__2021-03-19_06-14-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 63M Mar 19 06:39 out-0__2021-03-19_06-14-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 60M Mar 19 06:39 out-0__2021-03-19_06-15-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 57M Mar 19 06:39 out-0__2021-03-19_06-15-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 55M Mar 19 06:39 out-0__2021-03-19_06-16-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 52M Mar 19 06:39 out-0__2021-03-19_06-16-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 50M Mar 19 06:39 out-0__2021-03-19_06-17-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 48M Mar 19 06:39 out-0__2021-03-19_06-17-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 45M Mar 19 06:39 out-0__2021-03-19_06-18-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 43M Mar 19 06:39 out-0__2021-03-19_06-18-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 41M Mar 19 06:39 out-0__2021-03-19_06-19-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 39M Mar 19 06:39 out-0__2021-03-19_06-19-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 38M Mar 19 06:39 out-0__2021-03-19_06-20-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 36M Mar 19 06:39 out-0__2021-03-19_06-20-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 35M Mar 19 06:39 out-0__2021-03-19_06-21-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 33M Mar 19 06:39 out-0__2021-03-19_06-21-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 31M Mar 19 06:39 out-0__2021-03-19_06-22-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 30M Mar 19 06:39 out-0__2021-03-19_06-22-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 28M Mar 19 06:39 out-0__2021-03-19_06-23-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 27M Mar 19 06:39 out-0__2021-03-19_06-23-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 25M Mar 19 06:39 out-0__2021-03-19_06-24-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 24M Mar 19 06:39 out-0__2021-03-19_06-24-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 23M Mar 19 06:39 out-0__2021-03-19_06-25-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 22M Mar 19 06:39 out-0__2021-03-19_06-25-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 21M Mar 19 06:39 out-0__2021-03-19_06-26-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 20M Mar 19 06:39 out-0__2021-03-19_06-26-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 19M Mar 19 06:39 out-0__2021-03-19_06-27-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 18M Mar 19 06:39 out-0__2021-03-19_06-27-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 17M Mar 19 06:39 out-0__2021-03-19_06-28-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 16M Mar 19 06:39 out-0__2021-03-19_06-28-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 16M Mar 19 06:39 out-0__2021-03-19_06-29-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 15M Mar 19 06:39 out-0__2021-03-19_06-29-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 14M Mar 19 06:39 out-0__2021-03-19_06-30-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 13M Mar 19 06:39 out-0__2021-03-19_06-30-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 12M Mar 19 06:39 out-0__2021-03-19_06-31-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 11M Mar 19 06:39 out-0__2021-03-19_06-31-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 11M Mar 19 06:39 out-0__2021-03-19_06-32-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 9.3M Mar 19 06:39 out-0__2021-03-19_06-32-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 8.3M Mar 19 06:39 out-0__2021-03-19_06-33-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 7.4M Mar 19 06:39 out-0__2021-03-19_06-33-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 6.7M Mar 19 06:39 out-0__2021-03-19_06-34-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 6.2M Mar 19 06:39 out-0__2021-03-19_06-34-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 5.5M Mar 19 06:39 out-0__2021-03-19_06-35-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 4.8M Mar 19 06:39 out-0__2021-03-19_06-35-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 4.1M Mar 19 06:39 out-0__2021-03-19_06-36-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 3.3M Mar 19 06:39 out-0__2021-03-19_06-36-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 2.6M Mar 19 06:39 out-0__2021-03-19_06-37-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 2.0M Mar 19 06:39 out-0__2021-03-19_06-37-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 1.5M Mar 19 06:39 out-0__2021-03-19_06-38-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 1008K Mar 19 06:39 out-0__2021-03-19_06-38-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 500K Mar 19 06:39 out-0__2021-03-19_06-39-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:39 out-0__2021-03-19_06-39-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:40 out-0__2021-03-19_06-40-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:40 out-0__2021-03-19_06-40-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:41 out-0__2021-03-19_06-41-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:41 out-0__2021-03-19_06-41-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:42 out-0__2021-03-19_06-42-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:42 out-0__2021-03-19_06-42-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:43 out-0__2021-03-19_06-43-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:43 out-0__2021-03-19_06-43-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 860K Mar 19 06:44 out-0__2021-03-19_06-44-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:44 out-0__2021-03-19_06-44-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:45 out-0__2021-03-19_06-45-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:45 out-0__2021-03-19_06-45-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:46 out-0__2021-03-19_06-46-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:46 out-0__2021-03-19_06-46-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:47 out-0__2021-03-19_06-47-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:47 out-0__2021-03-19_06-47-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:48 out-0__2021-03-19_06-48-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:48 out-0__2021-03-19_06-48-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:49 out-0__2021-03-19_06-49-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:49 out-0__2021-03-19_06-49-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:50 out-0__2021-03-19_06-50-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:50 out-0__2021-03-19_06-50-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:51 out-0__2021-03-19_06-51-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:51 out-0__2021-03-19_06-51-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:52 out-0__2021-03-19_06-52-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:52 out-0__2021-03-19_06-52-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 0 Mar 19 06:53 out-0__2021-03-19_06-53-06.log.gz
pubmikeb commented
I have a similar issue (#174) with a configuration pm2 set pm2-logrotate:max_size 1K.
It looks like max_size is broken.
jaylee79 commented
I had the same issue; crashed my system after logs exploding and taking up 100% of disk space.