"Prefix" functionality for grok
SolomonShorser-OICR opened this issue · 4 comments
Feature Request
The kv plugin has a "prefix" function, which will prepend all extracted keys with a string:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-prefix
I think it would be useful if a similar prefix function existed for grok, to prefix the fields that are created by grok.
I am using grok to process/extract fields from messages that are calls to web services. Some of the services have similar arguments/parameters, but the differences are significant enough that I'd like to be able to keep them separate in elastic, so I can report on their usage separately.
A simple example:
/ServiceA/download/file/${IDENTIFIER}/svg?q=1234
/ServiceB/token/${TOKEN}/download/${IDENTIFIER}/jpeg/file?q=1234
The patterns (simplified), look something like this:
\/ServiceA\/download\/file\/(?<ServiceA_identifier>[^/]?)\/(?<ServiceA_imageType>[a-zA-Z]+).*
\/ServiceB\/token\/(?<ServiceB_token>[\/]?)\/download\/(?<ServiceB_identifier>[^/]?)\/(?<ServiceB_imageType>[a-zA-Z]+)\/file\?(?<args>.*)
...
# There are many more, I won't bore you with them all
I have one set of grok patterns for ServiceA and another set for ServiceB. We are interested in which identifiers get sent to ServiceA and ServiceB. So I have named capture groups that are very long such as ServiceA_identifier
, ServiceB_identifier
, ServiceC_identifier
, etc... it's rather long and ugly. It would be nice if I could do this:
grok {
match => ["service_request", "%{SERVICEPATTERN}"]
patterns_dir => ["/usr/share/logstash/pipeline/patterns/"]
prefix => "ServiceA_"
}
so that I could simplify my regular expressions and make them shorter and easier to read:
\/ServiceA\/download\/file\/(?<identifier>[^/]?)\/(?<imageType>[a-zA-Z]+).*
\/ServiceB\/token\/(?<token>[\/]?)\/download\/(?<identifier>[^/]?)\/(?<imageType>[a-zA-Z]+)\/file\?(?<args>.*)
It would also let me re-use the same patterns with different prefixes.
Great idea, I would also like an enhancment of this plugin to add a target
, just like kv
does.
Thanks! When does 4.3.0 come out?
@SolomonShorser-OICR you can just run ./bin/logstash-plugin update logstash-filter-grok
to upgrade to 4.3.0
That worked for me, and I am on logstash 7.6.1