logstash-plugins/logstash-output-elasticsearch

Invalid data stream configuration when using dlq_custom_codes option

ckauf opened this issue · 0 comments

ckauf commented

Logstash information:

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version)
    8.7.1

  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
    tar.gz

  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
    via CLI

  4. How was the Logstash Plugin installed
    Bundled

JVM (e.g. java -version):
Bundled

OS version (uname -a if on a Unix-like system):
MacOS 13.4

Description of the problem including expected versus actual behavior:
Actual behavior: When adding dlq_custom_codes to an output with data_stream => true the pipeline does not start but instead displays the following error message:

Invalid data stream configuration, the following parameters are not supported: {"dlq_custom_codes"=>[401]}

Expected behavior: Configuring custom codes works with Data Streams

Steps to reproduce:
Create a pipeline config that looks like this:

input {
  generator {}
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200/"]
    user => "elastic"
    password => "test"
    dlq_custom_codes => [401]
    data_stream => "true"
  }
}

Provide logs (if relevant):

[2023-05-22T16:10:21,618][ERROR][logstash.outputs.elasticsearch][main] Invalid data stream configuration, the following parameters are not supported: {"dlq_custom_codes"=>[401]}
[2023-05-22T16:10:21,619][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid data stream configuration: ["dlq_custom_codes"]>, :backtrace=>["/Users/user/Documents/elastic/logstash-8.7.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.13.1-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:76:in `check_enabled_data_stream_config!'", "/Users/user/Documents/elastic/logstash-8.7.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.13.1-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:59:in `check_data_stream_config!'", "/Users/user/Documents/elastic/logstash-8.7.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.13.1-java/lib/logstash/outputs/elasticsearch/data_stream_support.rb:32:in `data_stream_config?'", "/Users/user/Documents/elastic/logstash-8.7.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.13.1-java/lib/logstash/outputs/elasticsearch.rb:307:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:69:in `register'", "/Users/user/Documents/elastic/logstash-8.7.1/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/Users/user/Documents/elastic/logstash-8.7.1/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/Users/user/Documents/elastic/logstash-8.7.1/logstash-core/lib/logstash/java_pipeline.rb:607:in `maybe_setup_out_plugins'", "/Users/user/Documents/elastic/logstash-8.7.1/logstash-core/lib/logstash/java_pipeline.rb:249:in `start_workers'", "/Users/user/Documents/elastic/logstash-8.7.1/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/Users/user/Documents/elastic/logstash-8.7.1/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/Users/user/Documents/elastic/logstash-8.7.1/config/logstash-dlq.conf"], :thread=>"#<Thread:0x245522bb@/Users/user/Documents/elastic/logstash-8.7.1/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}