Logstash JDBC input (v5.4.1) crash pipeline when configured server is unreachable for some reason

Question

Logstash JDBC input (v5.4.1) crash pipeline when configured server is unreachable for some reason

vintur opened this issue a year ago · 1 comments

Logstash information:

Please include the following information:

Logstash version:
logstash 8.6.2
Logstash installation source:
yum localinstall logstash-8.6.2-x86_64.rpm
How is Logstash being run:
as a service
How was the Logstash Plugin installed:
logstash-integration-jdbc (5.4.1) was installed by default after upgrade.

JVM (e.g. java -version):
openjdk 17.0.6 2023-01-17
OpenJDK Runtime Environment Temurin-17.0.6+10 (build 17.0.6+10)
OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (build 17.0.6+10, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):
Linux Servername 3.10.0-1160.88.1.el7.x86_64 #1 SMP Sat Feb 18 13:27:00 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.9 (Maipo)

Description of the problem including expected versus actual behavior:
The pipeline with configured jdbc inputs crashes at startup when the configured database server is unreachable for some reason, for example not reachable via DNS or if the datebase is not listening on the configured port.

Expected behavior would be to just ignore the non reachable database and try it again next scheduled run.

Steps to reproduce:
Input configuration:

jdbc {
        jdbc_driver_library => "/etc/logstash/ojdbc8.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        jdbc_connection_string => "${CONNECTION_STRING}"
        jdbc_user => "${USER}"
        jdbc_password => "${PWD}"
        statement_filepath => "/etc/logstash/statement/statement.txt"
        schedule => "${SCHEDULE}"
        record_last_run => true
        tracking_column_type => "timestamp"
        tracking_column => "completion_time"
        use_column_value => true
        last_run_metadata_path => "/var/lib/logstash/.logstash_last_run"
        tags => ["jdbc_input"]
}

Provide logs (if relevant):

#################################### log 1 #########################################
[2023-04-21T08:15:44,715][INFO ][logstash.javapipeline    ][nameofpipeline] Starting pipeline {:pipeline_id=>"nameofpipeline", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>375, "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x74cc1016 run>"}
[2023-04-21T08:15:44,995][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline Java execution initialization time {"seconds"=>0.28}
[2023-04-21T08:15:46,390][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T08:15:46,433][ERROR][logstash.inputs.jdbc     ][nameofpipeline] 
java.sql.SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known
	at oracle.jdbc.driver.T4CConnection.handleLogonNetException(oracle/jdbc/driver/T4CConnection.java:1099) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:879) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:1132) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:104) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:837) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:664) ~[ojdbc11.jar:23.2.0.0.0]
.... 400 more stack trace lines
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143) ~[?:?]
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]
	at java.lang.Thread.run(java/lang/Thread.java:833) [?:?]
Caused by: oracle.net.ns.NetException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known
	at oracle.net.resolver.NavAddress.parseAddressParamsAndNavigate(NavAddress.java:208) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.NavAddress.navigate(NavAddress.java:83) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.NavDescription.navigate(NavDescription.java:214) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.NavServiceAlias.navigate(NavServiceAlias.java:61) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.resolveAddrTree(AddrResolution.java:1106) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.resolveTNSAddress(AddrResolution.java:918) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.initConnStrategy(AddrResolution.java:613) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.<init>(AddrResolution.java:430) ~[ojdbc11.jar:23.2.0.0.0]
.... 400 more stack trace lines
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143) [?:?]
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141) [jruby.jar:?]
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64) [jruby.jar:?]
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby.jar:?]
	at org.jruby.runtime.Block.call(Block.java:143) [jruby.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:309) [jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107) [jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2023-04-21T08:15:46,456][ERROR][logstash.inputs.jdbc     ][nameofpipeline] Unable to connect to database. Tried 1 times {:message=>"Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known", :exception=>Sequel::DatabaseConnectionError, :cause=>#<Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known>, :backtrace=>["oracle.jdbc.driver.T4CConnection.handleLogonNetException(oracle/jdbc/driver/T4CConnection.java:1099)", "oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:879)", "oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:1132)", "oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:104)", "oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:837)", "oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:664)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:472)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:330)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/adapters/jdbc.rb:237)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.new_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:248)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool.rb:151)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:139)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:91)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:293)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:302)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/misc.rb:188)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:57)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1507)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.inputs.jdbc.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_mixin_minus_ecs_compatibility_support_minus_1_dot_3_dot_0_minus_java.lib.logstash.plugin_mixins.ecs_compatibility_support.target_check.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:309)", "java.lang.Thread.run(java/lang/Thread.java:833)"]}
[2023-04-21T08:15:46,469][WARN ][logstash.inputs.jdbc     ][nameofpipeline] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known", :cause=>"#<Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known>"}
[2023-04-21T08:15:46,680][ERROR][logstash.javapipeline    ][nameofpipeline] Pipeline error {:pipeline_id=>"nameofpipeline", :exception=>#<LogStash::ConfigurationError: Can't create a connection pool to the database>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:318:in `register'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x74cc1016 run>"}
[2023-04-21T08:15:46,684][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline terminated {"pipeline.id"=>"nameofpipeline"}


#################################### log 2 #########################################


9:28,891][INFO ][logstash.javapipeline    ][nameofpipeline] Starting pipeline {:pipeline_id=>"nameofpipeline", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>375, "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x37d669c2 run>"}
[2023-04-21T15:39:29,200][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline Java execution initialization time {"seconds"=>0.31}
[2023-04-21T15:39:30,453][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:31,040][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:31,565][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:32,105][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:32,761][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:51:02,833][ERROR][logstash.inputs.jdbc     ][nameofpipeline] 
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
	at oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:774) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:688) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:39) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:691) ~[ojdbc8.jar:12.2.0.1.0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:472) ~[jruby.jar:?]
	at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:330) ~[jruby.jar:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/adapters/jdbc.rb:237) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.new_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:248) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool.rb:151) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:209) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:139) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:91) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:293) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:302) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/misc.rb:188) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:57) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118) ~[?:?]
	at org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1507) ~[jruby.jar:?]
	at org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen) ~[jruby.jar:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.inputs.jdbc.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_mixin_minus_ecs_compatibility_support_minus_1_dot_3_dot_0_minus_java.lib.logstash.plugin_mixins.ecs_compatibility_support.target_check.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234) ~[?:?]
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865) ~[jruby.jar:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143) ~[?:?]
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]
	at java.lang.Thread.run(java/lang/Thread.java:833) [?:?]
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660) ~[ojdbc8.jar:12.2.0.1.0]
.... 400 more stack trace lines
	at org.jruby.runtime.Block.call(Block.java:143) [jruby.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:309) [jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107) [jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.IOException: Operation interrupted, socket connect lapse 15000 ms. dbserver.domain.example.com/10.0.0.1 1521 0 1 true
	at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209) ~[ojdbc8.jar:12.2.0.1.0]
.... 400 more stack trace lines
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.InterruptedIOException: Operation interrupted
	at oracle.net.nt.TimeoutSocketChannel.handleInterrupt(TimeoutSocketChannel.java:311) ~[ojdbc8.jar:12.2.0.1.0]
.... 400 more stack trace lines
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2023-04-21T15:51:02,844][ERROR][logstash.inputs.jdbc     ][nameofpipeline] Unable to connect to database. Tried 1 times {:message=>"Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection", :exception=>Sequel::DatabaseConnectionError, :cause=>#<Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection>, :backtrace=>["oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:774)", "oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:688)", "oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:39)", "oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:691)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:472)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:330)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/adapters/jdbc.rb:237)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.new_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:248)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool.rb:151)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:139)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:91)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:293)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:302)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/misc.rb:188)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:57)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1507)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.inputs.jdbc.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_mixin_minus_ecs_compatibility_support_minus_1_dot_3_dot_0_minus_java.lib.logstash.plugin_mixins.ecs_compatibility_support.target_check.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:309)", "java.lang.Thread.run(java/lang/Thread.java:833)"]}
[2023-04-21T15:51:02,847][WARN ][logstash.inputs.jdbc     ][nameofpipeline] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection", :cause=>"#<Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection>"}
[2023-04-21T15:51:03,544][ERROR][logstash.javapipeline    ][nameofpipeline] Pipeline error {:pipeline_id=>"nameofpipeline", :exception=>#<LogStash::ConfigurationError: Can't create a connection pool to the database>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:318:in `register'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x37d669c2 run>"}
[2023-04-21T15:51:03,547][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline terminated {"pipeline.id"=>"nameofpipeline"}
[2023-04-21T15:51:03,551][ERROR][logstash.agent           ] Failed to execute action {:id=>:nameofpipeline, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<nameofpipeline>, action_result: false", :backtrace=>nil}

Something similar to this:
logstash-plugins/logstash-input-jdbc#144

My thread in discussion:
https://discuss.elastic.co/t/logstash-jdbc-input-v5-4-1-crash-pipeline-when-configured-server-is-unreachable-or-if-the-server-is-not-listening-on-db-port/330530

Answer 1 · 2023-05-05T13:00:49.000Z

My workaround for now is to install an older version:
bin/logstash-plugin install --version '5.2.4' logstash-integration-jdbc

Getting similar errors but the pipeline does not crash:
[2023-05-05T15:11:31,051][WARN ][logstash.inputs.jdbc ][pipelinename [6faf299d28a81ec8695e11ca859aaf26d751639446c01b99ea916ea01c0c5cc5] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection", :cause=>"#<Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection>"}