Support for Webdriver Sampler

[uddirulz]

Hi,

Can we have something like the Web driver config/sampler where in JMETER GUI we can run selenium scripts for RUM/synthetic monitoring. This can help in CI integration along with the rest of the dsl test pack.

Thanks,
Uddipan

[rabelenda]

Thank you for bringing this to it's own issue :).

Anyone interested in this same issue, please vote and if possible add additional context if you have more reasons to add this support.

[wtgchkj]

This would be a huge help for our team that uses ChromeDriver (https://jmeter-plugins.org/wiki/ChromeDriverConfig/) and WebDriverSampler (https://github.com/undera/jmeter-plugins-webdriver) for our UI coverage

Best,
Andrew

[rabelenda]

Hello @wtgchkj, thank you for voting for this and giving some context.

Can you provide more information about how you use it for UI coverage? Why using JMeter DSL + Selenium instead of just Selenium for coverage? How does JMeter + Selenium improves over plain Selenium?

[rabelenda]

@uddirulz, what are the main benefits of using JMeter + Selenium in your case?

Do you actually create JMeter test plans that generate load to backend while at the same time collect execution metrics with the selenium Sampler? And having everything in JMeter is easier for analysis, opposed to triggering from a test a selenium script in a loop + jmeter DSL test.

Or are you just generating some synthetics, and need to just collect each Selenium script metrics time? If your scenario is the later, have you considered using Allure @step annotations and use Allure reports?

Thank you in advance.

All collected information will help us to properly prioritize this development, and provide insightful information in user guide for users to know when are good scenarios to use such feature.

[wtgchkj]

Some tests have a blend of backend calls and Selenium test flows where Selenium scripts act as RUM and provide sample timings on critical UI actions. Since those backend calls are cheaper to scale resource wise pairing them with a handful of selenium test flows gives a cheap way to monitor user experience under load.

We also have some tests that are strictly Selenium + JMeter that capture UI samples across applications at scale.

Greenfield web apps that iterate on backend changes quickly also have a higher effort to maintain frequent changes to backend coverage so to get quick feedback on how early builds are performing, scripting JMeter Selenium (DSL) coverage is a low effort, high return use of time.

JMeter provides an easy, standardized way to push and report out metrics and analyze them quickly too.

[rabelenda]

Great info @wtgchkj , thank you!

[rabelenda]

I initially think that might go two ways here:

Selenium Plugin Support

With something like:

testPlan(
  chromeWebDriverConfig(),
  threadGroup(1, 1,
    webDriverSampler("WDS.sampleResult.sampleStart()\n"
      + "WDS.browser.get('https://mysite')\n"
      + "WDS.sampleResult.sampleEnd())\n"
  )
)

This would be easy to run at scale (BlazeMeter, Azure Load Testing, OctoPerf) and would have easy conversion with jmx2dsl, but selenium scripting would not benefit from Java IDE features (autocompletion, type safety, etc), or existing Selenium script code (reusability).

Jsr223Sampler extension

Currently jsr223Sampler allows you to provide a Java Lambda or class that implements SamplerScript interface. You could do something like this:

public static class SeleniumScript implements SamplerScript {
  
  private WebDriver driver;

  public SeleniumTestScript() {
    driver = new ChromeDriver();
  }

  @Override
  public void runScript(SamplerVars vars) {
    driver.get('https://mysite');
  }
  
}

@Test
public void test() {
  testPlan(
    threadGroup(1, 1,
      jsr223Sampler(SeleniumScript.class)
    )
  )
}

Even though this solution has several benefits:

• only 1 element is needed to be configured in test plan (so less chances you forget to add something)
• no need to remember calling sampleStart, sampleEnd, and learn WebDriverSampler scripting particularities
• java code which can reuse existing Selenium scripts code and IDE features like autocompletion, refactors & type safety.
• promotes modularization (Selenium script logic is separated from general test plan logic).
• might be more performant (would need to run some tests to verify) than webDriverSampler alternative, since the later uses Groovy which has proven (at least in some cases, as you can see in Performance Comparison Details here) to be slower than DSL provided support for executing lambdas and custom code.

But as is, it has a big drawback: No simple way to implement for proper driver disposal (some logic could be implemented with statics and tearDown thread group + jsr223 sampler, but is not intuitive).

To improve this, we could add a onTestEnd method to ScriptSampler interface, which would allow you to have something like this:

public static class SeleniumScript implements SamplerScript {
  
  private final WebDriver driver;

  public SeleniumTestScript() {
    driver = new ChromeDriver();
  }

  @Override
  public void runScript(SamplerVars vars) {
    driver.get('https://mysite');
  }

  @Override
  public void onTestEnd() {
    driver.quit();
  }
  
}

This could be easily implemented, and is a generic solution that could prove to be useful in many other situations.

The main downsides with this approach would be, not simple to implement jmx2dsl conversion (ergo: have simple migration from existing JMeter scripts) and that if you want to run your test at scale (BlazeMeter, OctoPerf, Azure Load Testing), then you would need to follow some additional steps as detailed here.

What do you think?

I tend to prefer the later one, since has more and better benefits from my point of view.

[wtgchkj]

I think the last one would be the better option too. It's important that we're able to pass arguments to ChromeDriver (run as headless, accept insecure certs, incognito, etc.) as well as be able to maintain the driver instance between sampler loops in order to preserve state (cache, etc.)

[rabelenda]

What do you think @uddirulz and @mikeliucc?

[mikeliucc]

I like the second option too. Our use of Selenium would be fairly extensive, considering the different type of users and scenarios, each with multiple pages to navigate. Also I really like the idea of reusability and working with real code (not quoted in strings). It'll help us to manage UI element changes over releases and to collaborate amongst team members.

[rabelenda]

We have just released a new version which allows you to easily integrate Selenium scripts. Check the new user guide, section.

[mikeliucc]

will be trying it out this week... thanks for adding this!

[rabelenda]

I will close this issue. If anyone has any comments or proposals in this, feel free to re open.