[BUG] - Load test YAML gets corrupted on upload.
Opened this issue · 9 comments
Describe the bug
Load test YAML gets corrupted on upload.
Failing/aborted Load Test is started using the Azure Load Testing v1 task in Azure Devops as part of a Release Pipeline (v1.2.20).
Load test aborts after about 4 mins with this error is the test detail :
"Test run failed due to error in test run configuration. Please try again. If the issue persists, please raise a support ticket along with the test run id."
example runid
testRunId 820afe4e-0960-4641-b791-0b7976763837
Test time summary on the step shows it appears to continue to run for over 3 hours.
To Reproduce
- Start with a correctly configure YAML (Our YAML in our repo is correct)
- Start load test using the Azure Load Testing v1 task in Azure Devops as part of a Release Pipeline (v1.2.20)
- Test aborts
- Download the input files after test fails
- Review YAML in the input file zip
- Note that one of the failureCriteria has been corrupted during copy
- input file zip YAML is not a copy of the YAML in our repo.
- testRunId 820afe4e-0960-4641-b791-0b7976763837 is an example test with this issue
Expected behavior
YAML not to be changed by the AZURE run test step in azure devops.
Screenshots
REPO YAML
YAML from the downloaded Input file zip from the failed/aborted test.
Additional context
This issue maybe related to issue #136
AB#1734561
@Shaun-Regan Thanks for reaching out. We will look into this? Meanwhile can you help us with details of what are you seeing in the Azure Portal for this test run? Did it fail? If yes, what is the error message?
Load test aborts after about 4 mins with this error is the test detail :
"Test run failed due to error in test run configuration. Please try again. If the issue persists, please raise a support ticket along with the test run id."
@Shaun-Regan Just confirming my understanding, the test run failed in 4 min on portal but the pipeline ran for 3+ hours. Additionally the YAML file is corrupted.
Can you please also share what was the duration for which you configured the test?
@Shaun-Regan Just confirming my understanding, the test run failed in 4 min on portal but the pipeline ran for 3+ hours. Additionally the YAML file is corrupted.
Your understanding is correct.
Can you please also share what was the duration for which you configured the test?
The test is configured to run for 20 mins.
@Shaun-Regan We looked into this issue and found that the memory and the CPU for the test engine exceeded the capacity which made JMeter crash. Could you please reduce the number of threads per engine and configure more engines instead to achieve your target load?
Hi, This seems like 100 users would not cause the issue. Here is my thinking why:
1.) The input files YAML is not in valid format (why would dropping the number of users resolve this)
2.) I have very similar tests executing successfully with 100 concurrent users.
3.) I can execute the exact same jmx in blazemeter using UK South - (London, Azure) as the location without issue
4.) Feels like 100 concurrent users is not very many for a load engine.
5.) I was under the impression that a load test engine should be capable of around 250 users each.
6.) If a load engine has an issue then the test step should abort, report the issue in the error detail and not continue to appear to execute for over 3 hours?
Even if dropping the users works, it feels like a workaround rather than an actual bugfix.
I will try dropping the users for this one test and see if that resolves the YAML issue and the test runs successfully. I'll let you know the results and the change.
Cheers
hi, the test ran for 20 mins with only 50 users. I've looked at the YAML and it still looks incorrect. Please See screenshot below.
Cheers
@Shaun-Regan Thanks for confirming that the test ran. The YAML file that is exported is only an output file that is generated. It does not effect the test that you ran. We will fix the syntax of the YAML file. Thanks for pointing that out.
On the test run, you can monitor the engine health metrics to make sure the engines are withing acceptable memory and CPU utilization.
We will make sure that in these cases a proper error message is provided.
Hi @ninallam please help me with the below issue.
I have followed your Azure Load testing service video, to run the JMeter script through portal. I have followed exact same steps. I have a JMeter script with csv data config and added necessary plugins , but still im getting the same Failure error. The test seems to be executing for long time and later it shows as failed. "Test run failed due to error in test run configuration. Please try again. If the issue persists, please raise a support ticket along with the test run id."
Whereas if i run a sample script without any CSV data files, it is running and the test is passed. Why these differences?
do we have configure anything else while running with CSV data file? I'm trying to create a framework for my organization using Azure Load testing service, is it reliable to create one? Please help
