Missing log messages during pod startup
felixbuenemann opened this issue · 3 comments
The deis logger misses messages that appear during pod startup until the logger starts tailing the container log.
The logs can be retrieved using kubectl logs, so maybe when a new pod is started, fluentd could be instructed to parse the entire log.
There appears to be an option to cause fluentd to read the entire log:
http://docs.fluentd.org/articles/in_tail#readfromhead
In order to avoid re-parsing old log entries if fluentd is restarted on the node, it could use a pos_file on a host volume.
This problem is especially bad because if a container crashes during startup, those messages are usually missing from deis logs.
I looked into the configuration and there is already a pos_file written to /var/log/containers.log.pos on the host, so it appears the only think left to do is changing read_from_head falseto read_from_head true in the source definition for "/var/log/containers/*.log".
Looks like I got the wrong repo, I'll open a new issue in deis/fluentd.