Connections timeouts cause input plugin to crash

[jsvd]

Currently there's no exception handling when performing the search or scroll requests to Elasticsearch.

This means that a SocketTimeout or ConnectionRefused error bubbles outside the plugin.

In a normal scan/scroll the input worker will catch the exception and restart the plugin:

[2020-05-05T23:09:47,551][ERROR][logstash.javapipeline    ][main] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Elasticsearch hosts=>["127.0.0.1:9200"], id=>"07db66e000aebf72500bbae2c946402277c14307fa361994723a96aa89a97a33", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_25f1e1a2-417a-4521-8c20-bb21a5c410e8", enable_metric=>true, charset=>"UTF-8">, index=>"logstash-*", query=>"{ \"sort\": [ \"_doc\" ] }", size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
  Error: Connection refused (Connection refused)
  Exception: Manticore::SocketException

When using slices, new threads are started for each slice, and exceptions will cause logstash to fatally crash:

[2020-05-05T23:11:02,823][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<Manticore::ConnectTimeout: connect timed out>, :backtrace=>["/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.5.0/lib/logstash/inputs/elasticsearch.rb:298:in `search_request'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.5.0/lib/logstash/inputs/elasticsearch.rb:246:in `do_run_slice'", "/private/tmp/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.5.0/lib/logstash/inputs/elasticsearch.rb:234:in `block in do_run'"]}