file_completed_action delete not working for gzip files
hkelley opened this issue · 6 comments
I didn't get a reply on the forum so I'm posting here.
When I read .gz log files, the files are not being deleted after logstash processing. When I read an uncompressed file it seems to delete correctly.
- Windows 10
- logstash-input-file (4.1.11)
- logstash-7.4.2
input {
file {
path => "C:/temp/CS_dump/logstash-pickup/part-*.gz"
sincedb_path => "NUL"
codec => json
mode => "read"
file_completed_action => "delete"
}
}
I can create and delete files in the folder so I don't see how it could be a permission issue. I don't see any errors in the logs related to a failed delete attempt.
@hkelly when you delete the uncompressed file, do you also use the * shell expansion or point to a single file?
Could you try with the latest 4.1.14
version of the plugin published 27/1/2019 because contains the bugfix #254 that solves a delete on multiple files?
Thanks for your feedback after the test, it's welcome
When I delete the files myself, from the shell, either explicitly or by wildcard, the files are deleted.
I updated logstash-input-file 4.1.11 to 4.1.14 but I get the same result in logstash.
Is there a log file I can generate for more clues?
I've tried with this:
- created sample text file in
/tmp
with
echo "blabla1" > ls_file_test1.txt && gzip ls_file_test1.txt && echo "blabla2" > ls_file_test2.txt && gzip ls_file_test2.txt
- created this Logstash config file
input_file_gzipped.conf
:
input {
file {
path => '/tmp/ls_file_test*.gz'
sincedb_path => "NUL"
mode => "read"
file_completed_action => "delete"
}
}
output {
stdout {
codec => 'rubydebug'
}
}
- launched with
bin/logstash -f input_file_gzipped.conf --debug
And gzipped files are correctly deleted. Tried both version 7.4.2and latest
master`.
Probably the the user you use to run Logstash is different from the user you used to modify files in the path C:/temp/CS_dump/logstash-pickup/
and they have different permissions. Checkout your log to see if something is written about failed deletion, start Logstash with --debug
CLI switch
-
I am creating the files and running Logstash as the same user. This should eliminate any chance of a permission deficit. I confirmed that the user can delete the files.
-
Configuration is:
input { file { path => "C:/temp/CS_dump/logstash-pickup/*.txt.gz" sincedb_path => "NUL" mode => "read" file_completed_action => "delete" } } output { stdout { codec => rubydebug } }
-
Launched with
\elastic\logstash-7.4.2\bin\logstash.bat -l c:\temp\CS_dump\logstash-logs -f "logstash-test.conf" --debug
-
I do not see any any errors (though I'm not terribly familiar with logstash so I may be overlooking). Log attached.
logstash-plain.log
I suspect that
unlink using a File.delete in JRuby create a twin handler to the same file, the other is kept bylogstash-input-file/lib/filewatch/watched_file.rb
Lines 18 to 19 in 42e2ef2
I ran into this issue too. Setting logstash to debug output showed this:
The "file_completed_log_path" setting must be provided when the "file_completed_action" is set to "log" or "log_and_delete"
.
Once I did as the message said, the file was deleted and the information appeared in the log. The file input documentation: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-file_completed_action, suggests that both are necessary, but doesn't say it outright.