Wildcard match without suffix algorithm seems like a hazard
groner opened this issue · 3 comments
Problem
Using Match foo.* with no suffix can lead to cause multiple uploads to the same file if there is more than one distinct tag being flushed in the same second.
#24 added a suffix mechanism to avoid this, but it isn't enabled by default. Because this issue can lead to data being lost silently, I think more should be done to avoid this scenario.
Mitigation
A warning could be generated at startup if a wildcard match is used and suffix algorithm is not specified explicitly.
Alternately, the default suffix algorithm could be changed when a wildcard match is used.
Mitigation
A warning could be generated at startup if a wildcard match is used and suffix algorithm is not specified explicitly.
Alternately, the default suffix algorithm could be changed when a wildcard match is used.
fluent-bit-go and its using go output plugin cannot handle tag information at startup. tag information will be handled in FLBPluginFlushCtx.
My thought was that the Match directive could be checked, but it seems this isn't exposed in the config property map or any other api.
Since opening this I've realized that collisions can happen due to the batching of records, even without the use of a wildcard match. It seems not using suffixes is problematic except in certain low throughput circumstances.