logstash-plugins/logstash-input-file

Final solution to inode reuse

rgr78 opened this issue · 8 comments

rgr78 commented

There are some scattered issues about the inode reuse problem, all closed, but the problem still stands. @guyboertje seems to understand the issue, this was the most promising solution:

We are planning to give the user the option of choosing their sincedb key: inode or path with fingerprinting a later option.

If the chosen key is path or fingerprint then the exact same content might be seen again.
(...)

#213 (comment)

But, where is the issue to implement this? If it doesn't exist, it can't be worked on, and the problem won't be fixed.

As far as mitigations go:

  • Setting sincedb_clean_after to a low value is difficult, because of this issue, also open: #231
  • In our usecase, a simple file_completed_action => "move" (to a different folder) would suffice, but it doesn't exist.

I only found this list with some information on this: #211

Can you please schedule some time to work on logstash-input-file? That list was created one year ago.

Thanks.

rnicksic commented

Does anyone have a solution to this? I'm fetching gzip log files every 10 minutes via external script and copying to ingest directory. I've tried a number of different values for sincedb_clean_after, but I'm still getting inode reuse.

input {
file {
path => "/var/spool/logstash/cloudflare*.gz"
sincedb_path => "/var/spool/logstash/cloudflare_logs.sdb"
sincedb_clean_after => "0.0056"
mode => "read"
file_completed_action => "log_and_delete"
file_completed_log_path => "/var/log/logstash/cloudflare_processed.log"
tags => ["cloudflare"]
}
}

rnicksic commented

So I updated to logstash file input plugin 4.1.16 yesterday and updated the value of sincedb_clean_after, which appears to be working properly now. The issue that I have noticed is that sincedb isn't being updated when in "read" mode if there are no files to be processed. This causes what should be expired records to remain and cause the inode reuse issue.

Here is my configuration and processing workflow and the monitoring script that I used to find this behavior:

input {
file {
path => "/var/spool/logstash/cloudflare*.gz"
sincedb_path => "/var/spool/logstash/cloudflare_logs.sdb"
sincedb_clean_after => "1min"
mode => "read"
file_completed_action => "log_and_delete"
file_completed_log_path => "/var/log/logstash/cloudflare_processed.log"
tags => ["cloudflare"]
}
}

I fetch cloudflare logs via a cron script every 10 minutes and dump them in the path directory. When I get a backlog of unprocessed files due to inode reuse, I stop logstash, delete the sincedb file and then restart logstash. I then watch the output of the following monitoring script and can see that while there are files being processed, sincedb is updated and old records expired appropriately. When all the files are processed, sincedb doesn't expire the few remaining records.

#/bin/bash

date_epoch=$(date +'%s')
if ls /var/spool/logstash/.gz 1> /dev/null 2>&1 ; then
ls /var/spool/logstash/.gz | wc -l
echo "\n"
ls -li /var/spool/logstash/*.gz
echo "\n"
else
echo "No cloudflare log files found\n"
fi

if [ -f /var/spool/logstash/cloudflare_logs.sdb ] ; then
cat /var/spool/logstash/cloudflare_logs.sdb
echo "\n"
for i in cat /var/spool/logstash/cloudflare_logs.sdb | cut -d ' ' -f5; do
echo $date_epoch-$i | bc
done
else
echo "/var/spool/logstash/cloudflare_logs.sdb not found"
fi

I'm not sure who needs to see this, so I'm tagging @guyboertje, hoping that whomever needs to see this will be notified.

I hope this helps. Please let me know if there are any questions.

ushmodin commented

I am sorry.

Why sincedb used in "read" mode?

Files are read once and then deleted.

rnicksic commented

My understanding is that sincedb keeps track of the position in the log file in case logstash is restarted while processing a file. Without doing so, you could miss some logs or get duplicates if logstash starts over on a partially processed file.

ushmodin commented

@rnicksic
thanks
I forgot about the restart case

Dukestep commented

Are there any updates regarding this issue? I am following more or less the same workflow as described in @rnicksic's comment and have noticed similar behaviour. We currently drop .gzip logs in a watched directory every day and let Logstash and the file input do the rest of the heavy lifting. Gzipped log files are then deleted when they are 30 days old.

I recently noticed Logstash was skipping some of the gzipped logs in the directory. Taking a look at the sincedb file, I noticed that for a new file, old entries for the inode associated to a deleted file (and now associated to a new incoming file) still existed in sincedb. As new files come in, they often are assigned an inode that was previously assigned to an old filed processed by Logstash. This results in the file being completed ignored.

I have now setup sincedb_clean_after => 3 and ignore_older => "2d" in my configuration in hopes that this will help overcome the inode reuse issue.

Any other suggested solutions or workarounds?

Mukul-Srivastava commented

Hi ,

Is there any update on this issue?
We are submitting CDN gzip log files to logstash.

I can see that thousands of gzip files are left without processing due to this issue.
I tried using file_completed_action => "log_and_delete" as well as "delete"," but the issue persists in both cases. I could not try with action "log" since I will end up with millions of files within an hour, which will be cumbersome for me to delete.
We are eagerly waiting for an early solution to this.
Any workaround will be greatly appreciated.

ylasri commented