Directory under patterns directory can crash logstash at startup
JalehD opened this issue · 3 comments
- Version: 5.3.1
- Operating System: any
- Config File (if you have sensitive info, please remove it):
- Sample Data: no need for data
- Steps to Reproduce:
1- Install logstash 5.3.1,
2- Create a patterns directory and reference it using pattern_directory in conf. below is a conf sample:
`input {
stdin { }
}
filter {
grok {
patterns_dir => "/opt/logstash/patterns"
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}`
3- Create an empty directory under the patterns directory.
4- start logstash and see it crash with below error
[2017-05-05T11:04:40,864][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x8b7e48c @id=\"c1ec15479fd638b036420c5eb13aa96522533774-2\", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x4212ce4 @metric=#<LogStash::Instrument::Metric:0x4e8f512b @collector=#<LogStash::Instrument::Collector:0x5a287b29 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x6d0c7764 @store=#<Concurrent::Map:0x60c7302d @default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x54f52317>, @fast_lookup=#<Concurrent::Map:0x4d9dfb93 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"c1ec15479fd638b036420c5eb13aa96522533774-2\", :events]>, @logger=#<LogStash::Logging::Logger:0x1abb0402 @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0xdffbf7a>>, @filter=<LogStash::Filters::Grok patterns_dir=>[\"/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/patterns\"], match=>{\"message\"=>\"%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\\\[%{HTTPDATE:timestamp}\\\\] \\\"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\\\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}\"}, id=>\"c1ec15479fd638b036420c5eb13aa96522533774-2\", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>\"*\", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>[\"_grokparsefailure\"], timeout_millis=>30000, tag_on_timeout=>\"_groktimeout\">>", :error=>"undefined method close' for nil:NilClass"}
[2017-05-05T11:04:40,872][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<NoMethodError: undefined method close' for nil:NilClass>, :backtrace=>["/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:83:in add_patterns_from_file'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:405:in add_patterns_from_files'", "org/jruby/RubyArray.java:1613:in each'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:401:in add_patterns_from_files'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:284:in register'", "org/jruby/RubyArray.java:1613:in each'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:280:in register'", "org/jruby/RubyHash.java:1342:in each'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.0/lib/logstash/filters/grok.rb:275:in register'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:257:in register_plugin'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:268:in register_plugins'", "org/jruby/RubyArray.java:1613:in each'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:268:in register_plugins'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:278:in start_workers'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/pipeline.rb:207:in run'", "/Users/Jaleh/ALL_MY_TESTS/LogS/crash-grok/logstash-core/lib/logstash/agent.rb:389:in start_pipeline'"]}
Additional info from @guyboertje
"
when using the standard patterns_files_glob of "*" a directory will be globbed in the list of files found but we don't check for directories later on when we open the file.
this ruby File.exists?(folder) returns true for a directory <- bug!!! we should check that its a file not a folder - this is in the grok filter line 402 of v3.4.0
"
@guyboertje on it as discussed :)