10x speed differences between v5.2.2 and v5.2.3
xia0pin9 opened this issue · 0 comments
Logstash information:
Please include the following information:
- Logstash version: 8.2.0
- Logstash installation source: docker
- How is Logstash being run: docker
- How was the Logstash Plugin installed: ./bin/logstash-plugin install --version xxx logstash-output-http
JVM (e.g. java -version
): using the bundled JDK
OS version (uname -a
if on a Unix-like system): Linux hostname 4.15.0-1065-aws #69-Ubuntu SMP Thu Mar 26 02:17:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
We have a logstash pipeline that reads from kafka and write to https endpoint. After upgrading logstash version, I noticed that the pipeline throughput was dramatically reduced. The logstash node stats api showed that majority time were spent in http output plugin. After manually uninstall logstash-filter-http, logstash-input-http_poller, logstash-output-http, and gradually upgrade logstash-output-http version, I noticed the performance degradation started from v5.2.3.
Stats for v5.2.2:
"events" : {
"out" : 328875,
"filtered" : 328875,
"duration_in_millis" : 1474697,
"queue_push_duration_in_millis" : 115194,
"in" : 331876
}
"outputs" : [ {
"events" : {
"out" : 328875,
"duration_in_millis" : 1474551,
"in" : 330375
},
"name" : "http"
} ]
Stats for v5.2.3:
"events" : {
"out" : 21000,
"queue_push_duration_in_millis" : 119697,
"filtered" : 21000,
"in" : 24001,
"duration_in_millis" : 1437976
}
"outputs" : [ {
"name" : "http",
"events" : {
"out" : 21000,
"in" : 22500,
"duration_in_millis" : 1437954
}
} ]
Steps to reproduce:
pipeline config:
input {
kafka {
codec => "json"
consumer_threads => 2
}
}
filter {
}
output {
http {
url => "https_target"
http_method => post
format => "json"
content_type => "application/json"
#ssl_verification_mode => "none"
pool_max => 200
pool_max_per_route => 200
}
}
logstash.yml:
cat logstash.yml
log.level: info
pipeline.batch.delay: 5
pipeline.workers: 12