Does sometimes not pick up rotated logs
piotr1212 opened this issue · 5 comments
According to the readme: Rotating the log files while the exporter is running is OK. While this works most of the times, it already happened twice in two days that it does not and the exporter keeps tracking deleted files.
postfix+ 18282 1 0 May11 ? 00:07:50 /usr/local/bin/postfix_exporter --postfix.logfile_path=/var/log/mail.log
ls -l /proc/18282/fd/
total 0
<< skipped >>
lr-x------ 1 postfix-exporter postfix-exporter 64 May 11 11:55 3 -> '/var/log/mail.log.1 (deleted)'
rotate config:
/var/log/mail.log
{
rotate 24
hourly
missingok
notifempty
compress
delaycompress
sharedscripts
maxsize 200M
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
OS: Ubuntu 18.04 LTS
Running latest commit b050cdf
BTW could not get journald log working at all, probably related to #55
I see there is a bunch of known issues with hpcloud/tail, switched to a better maintained fork to see if that fixes our issues. Will post update if it works. piotr1212@6039f07
Been running the version with nxadm/tail on multiple mailservers for 5 days with hourly rotate and the issue did not occur anymore, before this patch it would happen every few hours.
Even with the updated code I'm still seeing failures. I'm using the busybox syslogd launched as such syslogd -n -b 0 -s 50000 where the daemon itself does log rotation on it's own (not logrotate). Any ideas what could be causing the sporadic behavior in this setup?
I also still see it happening once in a while, don't know why. Is much better than before though.
I toyed with using a named pipe but I didn’t have much luck getting the tail library to properly read data continuously (hacked the code and tried various options in the config struct). Ultimately I resorted to a cronjob to restart the collector on an interval (I run the container with supervisor so restarts are viable).