logstash-plugins/logstash-input-file

After restart logstash sincedb is not cleaned (sincedb that generated in previous running)

godhelpsus opened this issue · 2 comments

godhelpsus commented

I have an experience with the inode recycling problem, so I changed the sincedb_clean_after value like below.
sincedb_clean_after => "6 hour"

It is working fine.
But 6 hours after restarting the logstash , the old record which is generated previous running is not cleaned.
The record which generated current running is cleaned normally.

I think the logstash should clean old records in the sincedb no matter when they are created.

  • version : logstash-7.3.0
andsel commented

@godhelpsus I've some questions:

  • which is your mode? read or tail
  • if in read what value has file_completed_action
  • after the 6 hours, before restarting Logstash, did you still see the expired file?
  • after the 6 hours, if you stop Logstah witch is the content fo you sincedb file? If now you restart Logstash with --debug or --log.level trace command line switches do you see that input-file plugin discover again your expired file?
jsvd commented

looking at the code that performs sincedb cleanup, it only happens when the sincedb is serialized to disk or when a file is opened.
In the scenario where you're working with a single event or other files being watched that aren't being written to, then the cleanup code never executes, only during specific eventst, like a shutdown.