ChainSafe/forest-iac

Snapshot service log rotation

Closed this issue · 3 comments

Issue summary

A potential cause for an outage of the snapshot service is amassing too many logs.

root@forest-snapshot:~# uptime
 14:03:15 up 21 days, 46 min,  1 user,  load average: 0.13, 0.88, 1.94
root@forest-snapshot:~# du -sh logs/
4.5G    logs/

While this is not critical (we've got some slack when it comes to disk space, and we update the service often enough, e.g., for the network upgrades), it'd be nice to have it bounded to, e.g., 5 GiB, two last weeks or something along those lines.

Other information and links

@LesnyRumcajs, so basically we want to prune log files older than 2 weeks and more than 5GiB?

Perhaps just pruning files older than two weeks would be enough. If 2 weeks' worth of log files is more than 5 GB, we should fix the logging itself.

Screenshot 2024-01-11 at 8 36 51 PM

I think this is not bad for a service close to one month of uptime.