logstash-plugins/logstash-output-elasticsearch

Failing to provide a + in the date pattern of the index property silently fails.

nicenemo opened this issue · 0 comments

Logstash information:

  1. Using bundled JDK: /usr/share/logstash/jdk logstash 8.4.1 But also fails on older versions
  2. Debian packages from Elastic. But also fails on Mac.
  3. Either as service or as command line
  4. plugin came packaged with logstash

We run Logstash both as systemd service as well as commandline tool.
We run on Debian Bullseye and MacOS.
Debian is ran on AWS ARM & Intel based systems. It is run on Windows WSL2 based Debian instances too.

JVM (e.g. java -version):

Bundled

Steps to reproduce:

Use a config like this:

input {
  # We read from backup files. Note that the start of the path MUST be absolute
  file {
    path => "/opt/data/backup/ecorp-fluffy-data-2022.33.json"
    sincedb_path => "/tmp/sincedb_restore-ecorp.db"
    start_position => "beginning"
    file_chunk_size => 268435456
    mode => "read"
    codec => "json"
    file_completed_action => "log"
    file_completed_log_path => "/opt/data/logstash/logs/restore-ecorp-fluffy-data.log"
  }
}
filter {
  if "_jsonparsefailure" in [tags] {
    drop { }
  }
# sort
  json {
    source => "message"
  }
}

output {
  # We write to the "new" cluster
  elasticsearch {
    manage_template => false
    sniffing => false
    ilm_enabled => false
     # does not work and fails silent because of missing +.
     index => "ecorp-fluffy-%{YYYY.MM}.01"
     # Does work. the added fixed 01 is to work around another bug/feature
     # index => "ecorp-fluffy-%{+YYYY.MM}.01"
    doc_as_upsert => true
    action => "update"
    document_id => "%{identifier}"
    http_compression => true
    ssl => true
    ssl_certificate_verification => false
    hosts => ["https://localhost:9200"]
    api_key => " not going to tell you"
  }
  # We print dots to see it in action
  stdout {
    codec => "dots"
  }
}

It will fail silently.

Provide logs (if relevant):

That's the problem.... Logstash is silent ;)