Events can be dropped with --out-seq if the Seq sink's internal maximum buffer size is exceeded
Closed this issue · 5 comments
This package uses Serilog.Sinks.Seq, based on Serilog.Sinks.PeriodicBatching, which has a bounded internal queue to feed the background thread responsible for network calls.
When clef-tool imports a log file, it can overflow this bounded queue before the sink has a chance to catch up. Ideally, we'd configure the sink to provide backpressure instead of dropping events in this case, but increasing the queue size might be a reasonable mitigation.
This alternative implementation does not suffer the same issue; it's a little simplistic (does not do server-initiated minimum-level filtering, nor any payload size calculation), but the basic idea should be suitable for use here.
any updates for this? I am seeing lot of events being dropped which in turn makes seq useless for us
Hi! Thanks for the nudge - a new tool, https://github.com/datalust/seqcli - is a better fit for this use case, and doesn't suffer from this issue.
I think we're going to keep maintaining this repository as a general purpose manipulation tool for the CLEF format, but recommend seqcli for log import/ingestion tasks with Seq.
Thanks! seqcli worked like charm!
Good to hear :-) .. I might close this since the current functionality in this repo is acceptable for casual usage. We're including seqcli in the Seq v5 installer, too, so it'll be the more frequently-used tool.