Support for reading compressed files?
Numpsy opened this issue · 6 comments
Hi,
I'm not sure if this idea is in scope for the library, or indeed just for Serilog bit (none-Serilog specific), but:
I recently had a look at some Serilog add-ons that compress the log files at various points (e.g. https://github.com/cocowalla/serilog-sinks-file-gzip, https://github.com/cocowalla/serilog-sinks-file-archive), and it came to mind that it could possibly be useful to be able to read the .gz compressed log files without having to unpack them first?
Doing so could possibly be as simple as wrapping the FileStreams in GzipStreams before reading the contents, though I do have a slight recollection that the .NET 4.7 version of GZipStream would only read the first chunk of the file when given a .gz file that contains multiple pieces (which was fixed in the .NET Core version). I haven't tested it though, this is just a thought at the moment.
Any thoughts?
Very interesting and worth checking. Especially for users the are using compressed files daily. The task of unpacking could get very tiresome after some time.
just summary:
One option (as you wrote) is to just unzip the content into temp folder, read all files and then delete them.
Another option, and more interesting, is to read directly the file stream and process it in memory (which is preferable to save IO).
It could be at the library level or the extension. Need to think about it a little :)
@Numpsy do you have an example gz file by any chance? I have added small implementation but don't have nominal archive to test it on.
This is a trivial clef/gz file that I created with a test app and the Serilog File and File.GZip sinks:
(I don't think there should be any difference to a plain .clef file compressed to a .gz using gzip/7-zip etc though)
Thanks :)
so I have added in memory parsing.
in case the data provider does not support compressed file I'm falling back to local unziping (and adding to the file open dialog the extensions):
Thanks, looks good.