Monthly date patterns are not possible with providing a value for the day because of a regex check
nicenemo opened this issue · 0 comments
nicenemo commented
Logstash information:
- Using bundled JDK: /usr/share/logstash/jdk logstash 8.4.1 But also fails on older versions
- Debian packages from Elastic. But also fails on Mac.
- Either as service or as command line
- plugin came packaged with logstash
We run Logstash both as systemd service as well as commandline tool.
We run on Debian Bullseye and MacOS.
Debian is ran on AWS ARM & Intel based systems. It is run on Windows WSL2 based Debian instances too.
JVM (e.g. java -version
):
Bundled
Steps to reproduce:
Use a config like this:
input {
# We read from backup files. Note that the start of the path MUST be absolute
file {
path => "/opt/data/backup/ecorp-fluffy-data-2022.33.json"
sincedb_path => "/tmp/sincedb_restore-ecorp.db"
start_position => "beginning"
file_chunk_size => 268435456
mode => "read"
codec => "json"
file_completed_action => "log"
file_completed_log_path => "/opt/data/logstash/logs/restore-ecorp-fluffy-data.log"
}
}
filter {
if "_jsonparsefailure" in [tags] {
drop { }
}
# sort
json {
source => "message"
}
}
output {
# We write to the "new" cluster
elasticsearch {
manage_template => false
sniffing => false
ilm_enabled => false
# does not work and fails because of a regex check.
index => "ecorp-fluffy-%{+YYYY.MM}."
# The added fixed 01 is to work around another bug/feature
# index => "ecorp-fluffy-%{+YYYY.MM}.01"
doc_as_upsert => true
action => "update"
document_id => "%{identifier}"
http_compression => true
ssl => true
ssl_certificate_verification => false
hosts => ["https://localhost:9200"]
api_key => " not going to tell you"
}
# We print dots to see it in action
stdout {
codec => "dots"
}
}
It will fail with a regex error that suggests I Must add a day.
Adding a fixed 01
out of the pattern seems to work as a workaround.
Provide logs (if relevant):
[....] "reason"=>"index name [ecorp-fluffy-2022.09] does not match pattern '^.*-(\\d{4}.\\d{2}.\\d{2})(-[\\d]+)?$'"}}}}