logstash-plugins/logstash-input-file

XML file is modified but logstash does not return to it

ConchuirLContract opened this issue · 1 comments

We are using logstash to read an xml file. This xml file is generated when a pipeline build commences and is written to with build data during the pipeline execution. We use file input mode 'read'.

CURRENT BEHAVIOR:
The xml file is created when the pipeline starts. Logstash discovers this xml file, reads it, logs it, and does not return to the xml file again.

PROBLEM:
Logstash has read the xml file prematurely and misses all the subsequent data that is written to it.

DESIRED BEHAVIOR:
Logstash allows us to apply some condition to tell it when to read the xml file. Ideally a trigger would tell logstash the xml file is completed and ready to be read and logged.

We want this to work with file input mode 'read'. The xml file is written to for around 1.5 hours.
Is there a filter, plugin or some other functionally that will allow logstash to return to the xml file when it is modified?

kares commented

this does not sounds like something the plugin was designed for esp. in read mode.
there's stat_interval => ... that allows configuring the interval changes are checked.
however, that won't solve your issue. ideally if the file is written to and you do not want to use tail mode than all it's left is to simply keep the file in a different location while it's written to and once done move/replace the file LS is monitoring ...