File input
gyscos opened this issue · 6 comments
I may have missed it, but I didn't find in the docs how to get input from a log file.
It would need to:
- Not read the whole file from the start, but seek to the end and read from there (like
tail -f
) - Keep working if a logrotate service renames the log file and creates a new file with the previous name (flowgger should start reading the new file).
Yes. This would let us use flowgger to replace many of our fluentd systemd.
To be able to read Docker logs, we use in_tail, with a path of /var/lib/docker/containers//-.json.log. Being able to "tail" (and as gyscos said resume tailing from a position) multiple files of input would be great. in_tail seems like an acceptable model.
For completeness, here's an example of one of our particular uses of in_tail to read multiple files:
<source>
@type tail
path /var/lib/docker/containers/*/*-json.log
pos_file /var/lib/fluent/fluentd-docker.pos
tag docker-tail.*
format json_in_json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%NZ
</source>
Is someone working on this? I can try to implement it
Hey @Trojan295, looks like nobody is, so if you can do it, that would be awesome!
I'm thinking what about multiple file support and the wildcard @rektide posted.
For now I would drop defining multiple input files (as this goes more to general support for multiple inputs), but wildcard support is very useful.
I believe the expected behaviour in this case is: watch any existing files and files, which were created during runtime.
@jedisct1, I've got a few questions, could you catch me on mail (visible in my profile)?
already merged in