logstash-plugins/logstash-filter-grok

Load grok patterns from `/etc/logstash/patterns`

joshuaspence opened this issue · 8 comments

joshuaspence commented

Currently this plugin loads patterns from the patterns/ directory within logstash-patterns-core and also from $LOGSTASH_HOME/patterns by default:

    @@patterns_path ||= Set.new
    @@patterns_path += [
      LogStash::Patterns::Core.path,
      LogStash::Environment.pattern_path("*")
    ]

On Logstash 5, $LOGSTASH_HOME/patterns is /usr/share/logstash/patterns. It seems more common, however, for users to define custom grok patterns in /etc/logstash/patterns rather than $LOGSTASH_HOME/patterns. The official elastic/logstash Puppet module, for example, provides a logstash::patternfile resource, which creates files in this directory. This works fine, but means that the user needs to add patterns_dir => "/etc/logstash/patterns" to every grok filter.

It seems to me that it would be better if the grok filter simply loaded patterns from /etc/logstash/patterns by default.

jordansissel commented

@joshuaspence I'm not against this, but I have some thoughts:

  1. /etc doesn't exist on windows.
  2. If this were a default, it would be a change in behavior we'd have to delay shipping with Logstash until Logstash 6.0 (but you could install it earlier)
  3. If this were a default, how would a user say "Don't load the patterns from /etc" ?
joshuaspence commented

Regarding Windows, maybe the Logstash core should provide a method for accessing the configuration directory (/etc on Linux, not sure about Windows)?

joshuaspence commented

Another option might be to allow plugins to be considered figured globally. As in, if I could set logstash.filters.grok.patterns_dir: '/etc/logstash/patterns' in /etc/logstash/logstash.yml

  • 👍1
jordansissel commented

I'm open to that. We used to have a flag (--patterns-path? I forget) but I removed it because Grok was they only plugin (out of hundreds) where there was a desire for a global setting.

joshuaspence commented

Another plugin where I have wanted to set a global default is logstash-filter-geoip, where I have wanted to set a global default for the path to the GeoIP database file.

jordansissel commented

@joshuaspence Since we have a path.settings now in Logstash (as of 5.0), I think having a namespace available for each plugin to set defaults might be doable.

Can you open an issue about this on elastic/logstash?

dminuoso commented

Couldn't we simply load an environment variable LOGSTASH_FILTER_GROK_PATTERN_DIRS, which would take a colon separated list of paths? That would require no change to logstash, is easy to inject (in case of systemd you can simply use drop-in units, or maybe you control the deployment of logstash anyway), and should require very little effort to implement.