Python "tail -F" like functionality, targeted for processing log data, with a goal of being reliable and robust.
Create an object to follow a file.
Features: - Detects if file is truncated and starts over at the beginning. - Detects file rename and new file created (for log rotation). - Continues reading old file after rotation to catch stragglers written there. - Can write an optional "offset" file so it can pick up from where it left off.
Follow(filename, offset_filename=None, watch_rotated_file_seconds=300):
- filename: Filename to open and read data from.
- offset_filename: If given, a file name to write offset information to. This file is written either whenever the end of file is reached during reading, or when the "save_offset()" method is called. If you break out of the read loop (say because of Control-C), it is best to save the offset.
- watch_rotated_file_seconds: After detecting the file has been rotated, watch the old file for this many seconds to see if new data has been written to it after the rotation.
Follow().readline(none_on_no_data):
- none_on_no_data: If true, instead of sleeping and continuing, it will return None if data is not ready.
Follow().save_offset()
- No arguments
f = ponytail.Follow('/var/log/syslog')
for line in f.readlines():
print(line.rstrip())
More exhaustive example:
f = ponytail.Follow('/var/log/syslog', offset_filename='/tmp/syslog.offset', watch_rotated_file_seconds=10)
for line in f.readlines():
print(line.rstrip())
The above will save the processed offset to a file, and later processing will pick up where the previous run left off. It will also stop watching the old file after a rotation, after 10 seconds (default is 300).