anthonygauthier/jmeter-elasticsearch-backend-listener

Unable to post data from Jmeter to Elasticsearch using backend listener

sg86-sourav opened this issue · 9 comments

sg86-sourav commented

Hi,

I am facing an issue for long time while trying to post data from Jmeter to Elasticsearch using Backend Listener. The Elasticsearch [version : 6+] is installed on remote server and accessible through HTTPS connection. We have created the index in the Elasticsearch using console and the mapping is showing as below and this is the default mapping set by project team.

index : test1234

{
"mapping": {
"_doc": {
"dynamic": true
}
}
}

While executing the test, Jmeter backend listener is using "https://:443/test1234/SampleResult/_bulk" URL to post data into elasticsearch and the request is getting timed out. I believe the URL is not correct and that is why the request is getting timed out.

So, how we can update the POST url for pushing the data into elasticsearch.

Please let me know if any further information is required.

anthonygauthier commented

Hi,

Please provide me with a screenshot of your backend listener configuration.

Thanks,

sg86-sourav commented

Hi,

Please find below the backend listener snapshoot-

image

anthonygauthier commented

Mmmh, everything seems to be okay. Is the port you guys are using really 443? What exactly made you think the URL wasn't right to begin with?

I know several people who use this plugin in their day-to-day operations and have no issues with it. If you look at the code, the URL is formed with the first three (3) parameters;

  • es.scheme
  • es.host
  • es.port

If the URL you see doesn't correspond to what you need, then there must be an error in one of the 3 parameters listed above.

anthonygauthier commented

Also, on the same machine you're using for the JMeter GUI, can you open up a browser and ping your elasticsearch instance?

scheme://host:port (https://bla-bla-bla:443)

sg86-sourav commented

Yes, the port is correct as while giving an index in backend listeners which is not present in ElasticSearch, in that case the index is getting created in ES but data is not getting posted.

Also, we can POST the data into ElasticSearch using Postman from the same machine from where I am running JMeter.

While checking the source code for ElasticSearchMetricsSender.java, I can see the following line where it is appending "/SampleResult/" in the URL in case the ES version is less than 7. Can we edit this URL ?

if(elasticSearchVersionPrefix < 7) {
request = new Request("POST", "/" + this.esIndex + "/SampleResult/_bulk");
actionMetaData = String.format(SEND_BULK_REQUEST, this.esIndex, "SampleResult");
}
else {
request = new Request("POST", "/" + this.esIndex + "/_bulk");
actionMetaData = String.format(SEND_BULK_REQUEST, this.esIndex);
}

Please find below the response I am getting while hitting the host in the browser

{
"name" : "es-client-646xxxxxx9b-hqxxxj",
"cluster_name" : "es-cluster",
"cluster_uuid" : "78Bpxxxxxxxxfy2tsWMCwxxxx",
"version" : {
"number" : "6.7.2",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "56xxxxx8",
"build_date" : "2019-xx-xxT09:05:50.290371Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

anthonygauthier commented

The SampleResult shouldn't cause any issues in ES version >= 6.X.X. The issue seems to be something else, not the URL. The fact that your requests result in timeouts is quite strange, it's as if your JMeter instance was unable to reach the ES instance. The problem definitely doesn't seem related to the plugin itself.

Could you supply me with the HTTP request you're doing and the JMeter log as well as the console log (2 different things).

Try changing your HTTP Sampler so that it sends requests to a well known website, try google.com for example. See if these requests go through.

Thanks,

anthonygauthier commented

Comment deleted as it was a double of the issue in #75

anthonygauthier commented

@sg86-sourav , was the issue solved? If I don't have an answer by 4PM EST I'll close this issue.

debajyotichanda commented

Hi Anthony!!
I am facing the same issue. Can you please let me know a workaround. Jmeter test is able to create an index but not able to write anything. I am able to post data from Postman though.
I much appreciate your kind support.
Elasticsearch version: 7.5
JMeter Version: 5.2
Backend Listener version: 2.6.11
Console Log:

2020-06-06 16:14:43,730 INFO o.a.j.e.StandardJMeterEngine: Running the test!
2020-06-06 16:14:43,730 INFO o.a.j.s.SampleEvent: List of sample_variables: []
2020-06-06 16:14:43,731 INFO o.a.j.v.b.BackendListener: Backend Listener: Starting worker with class: class io.github.delirius325.jmeter.backendlistener.elasticsearch.ElasticsearchBackendClient and queue capacity: 5000
2020-06-06 16:14:43,732 INFO o.a.j.v.b.BackendListener: Backend Listener: Started  worker with class: class io.github.delirius325.jmeter.backendlistener.elasticsearch.ElasticsearchBackendClient
2020-06-06 16:14:43,849 INFO i.g.d.j.b.e.ElasticSearchMetricSender: Index already exists!
2020-06-06 16:14:43,896 ERROR i.g.d.j.b.e.ElasticSearchMetricSender: Exceptionorg.elasticsearch.client.ResponseException: method [GET], host [https://test.internal.intient.com:443], URI [/], status line [HTTP/1.1 404 Not Found]

2020-06-06 16:14:43,896 ERROR i.g.d.j.b.e.ElasticSearchMetricSender: ElasticSearch Backend Listener was unable to perform request to the ElasticSearch engine. Check your JMeter console for more info.
2020-06-06 16:14:43,896 INFO o.a.j.g.u.JMeterMenuBar: setRunning(true, *local*)
2020-06-06 16:14:43,900 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : SampleTestPlan
2020-06-06 16:14:43,900 INFO o.a.j.e.StandardJMeterEngine: Starting 1 threads for group SampleTestPlan.
2020-06-06 16:14:43,901 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error
2020-06-06 16:14:43,901 INFO o.a.j.t.ThreadGroup: Starting thread group... number=1 threads=1 ramp-up=60 delayedStart=false
2020-06-06 16:14:43,901 INFO o.a.j.t.ThreadGroup: Started thread group number 1
2020-06-06 16:14:43,901 INFO o.a.j.e.StandardJMeterEngine: All thread groups have been started
2020-06-06 16:14:43,905 INFO o.a.j.t.JMeterThread: Thread started: SampleTestPlan 1-1
2020-06-06 16:14:44,373 INFO o.a.j.t.JMeterThread: Thread is done: SampleTestPlan 1-1
2020-06-06 16:14:44,373 INFO o.a.j.t.JMeterThread: Thread finished: SampleTestPlan 1-1
2020-06-06 16:14:44,373 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test
2020-06-06 16:14:44,388 INFO o.a.j.v.b.BackendListener: Worker ended
2020-06-06 16:14:44,388 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, *local*)

Backend Listener Setup 👍
image