Publish Test Plan Results

This custom Azure Pipeline task makes it possible to update your Azure DevOps Test Plan using the results of your test automation suite, regardless of which test framework you're using (Cucumber, JUnit, xUnit, others).

- task: PublishTestPlanResults@1
  inputs:
    testPlan: 'Test Plan 01'
    testResultFormat: xUnit
    testResultFiles: $(Build.StagingDirectory)/testResults.xml

Track Progress

Results from your Test Automation are available in Test Plans / Runs:

Progress is reflected in your Test Cases:

And overall progress is reflected in the Test Plan report:

A quick primer on Azure Test Plans

It's helpful to understand the following about Azure Test Plans:

You can have one or more Test Plans for a given Azure DevOps Project. Some teams have only a single Test Plan, others create a Test Plan per sprint, and some may create ad-hoc or fit for purpose plans.
Each Test Plan has a name and optional start and end date.
Each Test Plan has at least one Test Configuration that represents a platform or environment. For example, "Internet Explorer on Windows 11".
Test Plans are comprised of one or more Test Suites.
Test Suites are comprised of one or more Test Cases.
Each Test Case is associated with at least one or more Test Configuration.

The combination of Test Case + Test Configuration is referred to as a Test Point.

When we publish our test results from our test framework to the Test Plan, we are interested in creating a Test Run that contains an outcome for each Test Point. There is a one-to-one mapping between each individual automated test and a Test Point.

Mapping Test Results to Test Cases

The PublishTestPlanResults task supports several different mapping strategies to help you link your test framework results to Test Cases (Test Points). The default auto strategy will attempt to find the appropriate strategy for you, or you can customize the configuration to meet your needs.

Test Case Mapping Strategies

The following strategies are used to match the test automation result to individual Test Point items:

TestName: the name of the automated test must match the title of the Test Case.

Hyphens, spaces and underscores are removed from both the automated test name and the Test Case title to simplify case-insensitive comparisons.

Note:

This approach might be the easiest to configure, but is prone to mismatch errors if either the name of the Test Case or test method change. We recommend using the RegEx or TestProperty strategies to avoid mismatch issues.

RegEx: the id of the Test Case appears in the name of the test automation name.

The regex to find the test case identifier can be customized by setting the testCaseRegex.

Most test frameworks should allow you to specify the Test Case identifier somewhere in the name of the test automation.

// xunit
[Fact]
public void TestCase1234_ShouldAddTwoNumbers()
{
}

// junit
@Test
public void TestCase1234_shouldAddTwoNumbers() {
}

// cucumber
Scenario: TestCase1234 Can add two numbers

// javascript
it('Should add two numbers TestCase1234', () => {
});

// pester
it 'Should add two numbers TestCase1234' {
}

TestProperty: the id of the Test Case appears in the meta-data for the test.

The meta-data element to use defaults to TestCase, but this can be customized by setting the testCaseProperty setting.

Each test framework supports mechanisms to attach property meta-data in the output, some examples:

// xunit
[Fact]
[Trait("TestCase", "1234")]
public void ShouldAddTwoNumbers() 
{
}

// nunit
[Test]
[Property("TestCase","1234")]
public void ShouldAddTwoNumbers()
{
}

// cucumber
@TestCase=1234
Scenario: Add Two Numbers

// javascript
it('Add Two Numbers @TestCase=1234', () => {})
it('Add Two Numbers #TestCase=1234', () => {})

Test Config Mapping Strategies

If your Test Plan supports multiple Test Configurations, you'll need a mechanism to map the test results to the appropriate Test Configuration.

If you've specified a testConfigFilter then only TestPoints for that Test Configuration will be updated. However, if your test-automation contains the outputs from multiple configurations, additional configuration settings are needed:

Config Aliases: the testConfigAliases setting is used to configure one or more simple name aliases for the Test Configuration in your Test Plan. For example, "x86_chrome" can be used in the tests instead of "Chrome on Windows 11".
ConfigProperty: the configuration name or alias is defined in the meta-data for the test automation result. The testConfigProperty setting is used to define one or more test-meta properties to inspect.

Examples

Basic Usage (xUnit)

In this example, we're using a trait named TestId to hold the id of the TestCase in our TestPlan.

[Fact]
[Trait("TestId","12345")] // 12345 is a TestCase identifier in our TestPlan
public void AddItemToShoppingCart() {
  // ... test automation
}

In our Azure Pipeline, after running your tests, you can update your Azure Test Plan Test Plan 01 using the following syntax:

- task: PublishTestPlanResults@1
  inputs:
    testPlan: 'Test Plan 01'
    testResultFormat: xUnit
    testResultFiles: $(Build.StagingDirectory)/testResults.xml
    testCaseMatchingStrategy: property
    testCaseProperty: TestId

bryanbcook/azdevops-testplan-extension