Bulk request without checking supported action causing ES 400.

[mashhurs]

Short Description

Plugin receives 400 Validation Failed: 1: no requests added error when sending any random action.

Expectation

Plugin should validate the action before sending bulk request to ES.

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version)
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
4. How was the Logstash Plugin installed

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (java -version)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Setup any Logstash version (tested 7.15.1 & 8.3.0) and read json file

input {
   file {
       path => "/path/to/files/*.json"
       sincedb_path => "/since/db/path/sincedb"
       start_position => "beginning"
       codec => "json"
   }
}

3. logstash-output-elasticsearch 11.6.0 version
4. Set any value in file for action (ex: partialupdate) except index, delete, create & update and save

{
   "event":
   {
       "name": "TEST",
       "type": "TEST",
       "action": "partialupdate",
       "source": "template",
       "version": 1
   },
   "eventData":
   {
       "hello": "world"
   }
}

Plugin receives 400 Validation Failed: 1: no requests added error.

Provide logs (if relevant):

[2022-08-08T11:58:50,017][ERROR][logstash.outputs.elasticsearch][main][1] Encountered a retryable error (will retry with exponential backoff) {:code=>400, :url=>"https://{ES_HOST}/_bulk", :content_length=>882, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: no requests added;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: no requests added;\"},\"status\":400}"}

[mashhurs]

11.9.1 version of the plugin is released. Now, it filters out the invalid actions before sending to ES. Rejected events can be send to DLQ (if enabled, also warns in the log) or warns events in the log (if DLQ disabled).
Further note: ES v8.5 rejects entire bulk request if any of request item has invalid action (not in [index, delete, update, create]) and there is not any change required from LS side since this version opts out invalid actions.