microsoft/azure-load-testing

[BUG] Pipeline task returns InvalidPassFailCriteria when yaml has more than 5 failureCriteria

Opened this issue · 2 comments

Describe the bug
AzureLoadTest@1 - Azure Load Testing v1 task fails pipeline with error code: 'InvalidPassFailCriteria' when failureCriteria contains more than 5 items.

To Reproduce
Steps to reproduce the behavior:

pipeline.yml:

 # variables are set before this, other details removed for brevity
stages:
    - stage: LoadTest
      displayName: Load test
      jobs:
          - job: Run
            displayName: Runs a fixed load test in azure
            steps:
              - task: AzureLoadTest@1
                displayName: Azure load test
                inputs:
                  azureSubscription: $(serviceConnection) 
                  loadTestResource: $(loadTestResource)    
                  resourceGroup: $(loadTestResourceGroup) 
                  loadTestConfigFile: src/HomePage/HomePage.yml   

src/HomePage/HomePage.yml:

version: v0.1
testName: BHB_Home_Page
testPlan: HomePage.jmx
description: BHB Home page
engineInstances: 1
splitAllCSVs: False
failureCriteria:
  - AppointmentType: percentage(error) > 1 
  - AppointmentType: p90(response_time_ms) > 1000 
  - PatientLocationById: percentage(error) > 1 
  - PatientLocationById: p90(response_time_ms) > 1000 
  - Provider: percentage(error) > 1
  - Provider: p90(response_time_ms) > 1000 
env:
  - name: JM_RampTime
    value: 30
  - name: JM_EndRpsA
    value: 100
  - name: JM_EndRpsB
    value: 100
  - name: JM_HoldFor
    value: 300
  - name: JM_CoolDown
    value: 10
  - name: JM_PauseFor
    value: 5

result from DevOps pipelines

Starting: Azure load test
==============================================================================
Task         : Azure Load Testing
Description  : Automate performance regression testing with Azure Load Testing
Version      : 1.2.9
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/load-testing/tutorial-cicd-azure-pipelines#azure-load-testing-task
==============================================================================
{ error:
   { code: 'InvalidPassFailCriteria',
     message: 'Invalid pass fail criteria "Invalid PassFailCriteria" .',
     target: null,
     details: null } }
##[error]Error in creating test: bhb_home_page
Finishing: Azure load test

Expected behavior

Azure load testing task should allow create/update test with the same amount of test criteria as the tests created by portal.

Screenshots

Error from pipeline:
image

Additional context
Add any other context about the problem here.

AB#1642808

@leandrodotec ... Noted. we will update once we have rectified the issue.

We are having the same issue