Tried to send variable from threadGroup to elastic

Question

Tried to send variable from threadGroup to elastic

gimbal04 opened this issue 6 years ago · 18 comments

Hello !

I have few questions about the Backend Listener like a discussion it is why I open an issue for it.

Indeed, I need to call differents url and retrieve the environment linked with them In the backend listener to exploit it in Kibana.
I used a file which contains lines like : "env url apikey" and I make a loop on this file which call a request for each lines.

The ${url} in my threadGroup is good and the request works for each environment called but when I try to get it with custom parameter in the backendListener like : name:"environment", value:"${url}" and in Kibana I see the value as ${url} and not deployment for exemple.

Can you tell me if I make something wrong ?

Answer 1 · 2018-08-14T16:35:55.000Z

Hey @cprevot04 ,

So this is related to #27. Basically, the way that JMeter and its backend listener are built makes it so that the backend listener doesn't have access to JMeter variables that are declared anywhere else but in the JMeter's "Test Plan" element.

I would really love to work around this, but I'm afraid I have no way of accessing these variables unless the way it works is changed directly into the JMeter's source code. 😢

So to answer your question; no you didn't do anything wrong - it's simply a limitation of the software :( !

Answer 2 · 2018-08-15T21:06:12.000Z

Hello,
An option would be to submit a patch to jmeter implementing this .
I guess having variables declared in sample_variables available in SampleResult would do what you need.

See for contributing :

http://jmeter.apache.org/building.html

Answer 3 · 2018-08-15T21:14:29.000Z

Very true, this could very well solve the issue. I personally don't really have the time to implement it right now, but I'll try to work it in my schedule in the following weeks/months.

Thanks!

Answer 4 · 2018-08-27T08:35:00.000Z

Hello,
An other solution for us will be to exploit the RequestHeaders.

Would it be possible that your plugin parse and forward the request&response headers as "real" key/values in ElasticSearch ?
Hence, those headers would be much easier to exploit in Kibana.

Thanks !

Answer 5 · 2018-08-27T20:43:06.000Z

@cprevot04 Wow that's actually a great idea! I could sure as hell do that 😄 ! I'm on vacation right now, but I'll try to work on it ASAP!

Thanks for the feedback!

Answer 6 · 2018-09-04T18:55:07.000Z

@cprevot04 Just to let you know that I'm working on the headers method as I'm writing this. Should be out sometime this week 😄 !

Answer 7 · 2018-09-11T19:26:59.000Z

This was added with version 2.5.0! It is currently being pushed to the central repository and will be merged to JMeter-Plugins ASAP. For the meantime, you can manually uninstall your old version and install the new with the JAR available on GitHub! 😄

Cheers all,

Answer 8 · 2018-09-11T19:27:19.000Z

Hello,
Maybe this would require some documentation.
AFAIU, plugin would use request headers starting with "es-" ?

If my understanding is correct, this might break some applications. But maybe I misunderstand something here.

Regards

Answer 9 · 2018-09-11T19:39:55.000Z

Hi @pmouawad !

I documented the whole thing here https://github.com/delirius325/jmeter-elasticsearch-backend-listener/wiki/Sending-JMeter-variables .

I don't think it would break any applications per say, you might just have some unwanted key/pair values in your ElasticSearch index. Otherwise we can always change the startsWith("es-") to something more unique like startsWith("es-backend-").

Au plaisir,

Answer 10 · 2018-09-11T19:46:37.000Z

Thanks for your rapid answer.
1/ Reading documentation, I don't understand where Header Manager would be put, and my notes depends on where it is located.
Maybe you could add a screenshot with a little test plan showing where it would be located.

2/ For custom fields, maybe the screenshot should show the whole config element with a red rectangle around the required values.
As you know, Variables are related to Thread , Property are global, I am a bit confused for custom fields, do you allow customization per thread or do you send properties?

Answer 11 · 2018-09-11T19:58:06.000Z

@pmouawad

I'll rework the documentation, but to answer your question:

The Header Manager can either be located globally or locally, attached to a specific request.
The custom fields include the variables loaded through the the jp@gc CSV config element, the Test Plan variables, and properties that would be specified as ${__P(myProp)} or ${__groovy(props.get("myProp"))}.

Maybe it isn't clear in the documentation but Custom Fields are one of the two ways to include custom values to results sent to ElasticSearch. The second way being with the headers.

Thanks,

Answer 12 · 2018-09-11T20:04:51.000Z

Thanks again.

In this case, it means es-XXX headers would be sent in every HTTP Request. Some network component or application might refuse such headers and return 401 or 400 status codes. Usually such headers should be prefixed with 'X-' although this seems to have been deprecated (https://stackoverflow.com/questions/3561381/custom-http-headers-naming-conventions).
IMO using Headers is too invasive.
So maybe indicate that those "variables" should not vary per Thread, so they are king of Global Variables / Properties. Because currently users might think they can send value A for thread 1, and B for thread 2.... AFAIU, it's not expected to work this way

Answer 13 · 2018-09-12T21:35:45.000Z

Hello!

I thought the idea behind headers was just to insert into ElasticSearch ALL the headers which are part of the actual request and responses as keys and values (instead of a single string we can not use in ElasticSearch/Kibana). Not to add some more "es-XXX" headers to the actual requests.
At least that's how I understood the suggestion... :)

Answer 14 · 2018-09-12T23:52:08.000Z

Hey there!

That could also be another functionality, I'd just add a boolean to do the work! I'll add this in a minor version tomorrow 😄

Cheers,

Answer 15 · 2018-09-14T16:39:55.000Z

Hi @ncasaux ,

So I just pre-released version 2.5.1. It includes the following modifications:

Renamed the method to parseHeadersAsDocumentProps(boolean).
The boolean in the method indicates whether or not the user wants to parse all headers or simply the X-es-backendones
The headers used to pass variables as key/value in ES must now be written as X-es-backendinstead of es-

@cprevot04 , @pmouawad , @ncasaux : If some of you guys can try it and give me feedback/suggestions, I'd greatly appreciate it!

Thanks,

https://github.com/delirius325/jmeter-elasticsearch-backend-listener/releases/tag/2.5.1-SNAPSHOT

Answer 16 · 2018-09-14T21:41:15.000Z

Hi @delirius325 ,

I tried the "simple" headers parsing and it works! :)

One remark though: I think that request headers and responses headers should be in 2 different arrays in the json structure.
I hope it makes sense! :)

Thanks!

Answer 17 · 2018-09-27T13:34:38.000Z

This is still in the works, I pushed out a minor version yesterday but I realize now that it comes with its set of challenges and problems. I'll rethink the whole thing soon and push a patch.

Answer 18 · 2019-07-24T12:25:47.000Z

Hello,

Having issues with this functionality. I think i'm missing something simple but I'm not sure.

I'm using JMeter 5.1.1 with ES Plugin version 2.6.7. ElasticSearch backend is version 5.6.

What I'm trying to do is within my JMeter Thread Group I add an HTTP Header Manager with the following Names and Values
X-es-backend-test-variable hola1
es-test-variable hola2

When I run JMeter I see these headers being added to all my requests. However, I do not see them appearing in ElasticSearch. The rest of my JMeter data is appearing fine in ElasticSearch.

Update One issue I had was my samplers were all in Transaction Controllers with 'Generate parent sample' checked which apparently makes all the samplers contained within not visible to listeners. Even after changing that I still could not successfully get JMeter Variables to appear in ES by using 'X-es-backend-' prefix.