Config / flag to make LS Forwarder follow / tail files

[gjvoit]

I've been working on the ELK stack for a couple weeks now and have a pretty good understanding of how the stack works. I'm beginning to implement a bit more scaling into my stack and the first step I've taken in this direction is to get LS Forwarder to push logs from a server into a Docker container hosted on the same server. I've managed to do this successfully, but as of yet I've found no solution for having LS Forwarder only ship the most recent additions to a log file.

Currently running the following command to start my LS Forwarder process:

./logstash-forwarder -config /etc/logstash-forwarder &

which works great, but it reships the entire logfile whenever a new entry is added.

I've also tried using

./logstash-forwarder -config /etc/logstash-forwarder -tail &

But this does not seem to do anything. Is there a happy medium where my logstash Forwarder process will monitor the log files specified in the config and ship only the new lines to Logstash? Currently it's creating a lot of duplication in my ES server.

Thanks in advance!

[tsprlng]

That's exactly what should (and normally does) happen.

It should be keeping track of the offset reached and only shipping new lines after that. Have you checked the logs (in /var/log/logstash-forwarder)? Perhaps something is stopping it keeping track of its state, such as lack of permissions on the working dir (which shows up as 'Failed to create tempfile').

[ruflin]

As the new release of logstash-forwarder is filebeat, I recommend to try out and check if the issue still exists. In case the issue still exist, please open a new issue here: https://github.com/elastic/beats