logstash giving warning and error for pipline and java

[hcwwadmin]

logstash     | [INFO ] 2020-10-22 02:36:14.817 [LogStash::Runner] runner - Logstash shut down.
logstash     | WARNING: An illegal reflective access operation has occurred
logstash     | WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby5029500797019692358jopenssl.jar) to field java.security.MessageDigest.provider
logstash     | WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
logstash     | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash     | WARNING: All illegal access operations will be denied in a future release
logstash     | Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
logstash     | [INFO ] 2020-10-22 02:36:26.616 [main] runner - Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +jit [linux-x86_64]"}
logstash     | [ERROR] 2020-10-22 02:36:28.252 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 6, column 1 (byte 6) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}
logstash     | [INFO ] 2020-10-22 02:36:28.324 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash     | [INFO ] 2020-10-22 02:36:33.348 [LogStash::Runner] runner - Logstash shut down.
logstash     | WARNING: An illegal reflective access operation has occurred
logstash     | WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby3221644865931324601jopenssl.jar) to field java.security.MessageDigest.provider
logstash     | WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
logstash     | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash     | WARNING: All illegal access operations will be denied in a future release
logstash     | Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
logstash     | [INFO ] 2020-10-22 02:36:45.650 [main] runner - Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +jit [linux-x86_64]"}
logstash     | [ERROR] 2020-10-22 02:36:47.318 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 6, column 1 (byte 6) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}
logstash     | [INFO ] 2020-10-22 02:36:47.383 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash     | [INFO ] 2020-10-22 02:36:52.403 [LogStash::Runner] runner - Logstash shut down.```

[a3ilson]

@hcwwadmin - Is this related to your previous issue (#14)?

I've tested with the currently posted conf files without issue but the error provided in your comment above points to a missing element within column 1 line 6. Did you change/amend the files?

[a3ilson]

Is this still an issue?

[xlordz]

Hi @a3ilson, yes, this still a issue. Same problem. Tested with:

• logstash 7.10.0 and 7.9.3.
• Docker version 19.03.13
• Debian Buster
• Kernel 4.19.0-12-amd64

It's missing Log4J2 configuration file but that's ok.
The ruby problem I solved with this: (elastic/logstash#10496), but removed to post here. The pipeline problem persists even solving the ruby warnings.

logstash | WARNING: An illegal reflective access operation has occurred
logstash | WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby11041480225367338055jopenssl.jar) to field java.security.MessageDigest.provider
logstash | WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
logstash | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash | WARNING: All illegal access operations will be denied in a future release
logstash | Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
logstash | [INFO ] 2020-11-27 11:09:24.028 [main] runner - Starting Logstash {"logstash.version"=>"7.9.3", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +jit [linux-x86_64]"}
logstash | [ERROR] 2020-11-27 11:09:25.285 [Converge PipelineAction::Create

] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", [A-Za-z_], "-", [0-9], "[", "{", "]" at line 5, column 24 (byte 107) after filter {\r\n if [process][name] =~ /^filterlog$/ {\r\n grok {\r\n patterns_dir => [", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in block in converge_state'"]}
logstash | [INFO ] 2020-11-27 11:09:25.365 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash | [INFO ] 2020-11-27 11:09:30.386 [LogStash::Runner] runner - Logstash shut down.

[xlordz]

Puzzle Solved.

logstash/conf.d/05-firewall.conf:5
-- patterns_dir => [/usr/share/logstash/etc/logstash/conf.d/patterns"]
++ patterns_dir => ["/usr/share/logstash/etc/logstash/conf.d/patterns"]

[a3ilson]

Thanks!

The file was updated and apologize for any inconvenience.

[xlordz]

Don't apologize, you've made a really great job!
Thanks!