Test results

Semaphore collects XML test reports and uses them to provide insight into your pipelines.

With test reports, you enable your team to get an effective and consistent view of your CI/CD test suite across different test frameworks and stages in a CI/CD workflow. You get a clear failure report for each executed pipeline. Failures are extracted and highlighted, while the rest of the suite is available for analysis.

The test-results command-line interface (CLI) is a tool that helps you compile and process JUnit XML files. The output of the test results CLI is a report in JSON format.

This CLI is distributed as a part of the Semaphore toolbox, and it is available in all Semaphore jobs.

The main purpose of the CLI is to:

compile and publish JUnit XML files into a JSON report
merge multiple JSON reports into a single summary report

Note

Starting from version 0.7.0, test reports are compressed using gzip to optimize storage and handling. To maintain backward compatibility with existing systems and processes, the compressed files will continue to use the .json file extension. This approach ensures seamless integration with tools and workflows that expect files in this format.

However, please be aware that while these files retain the .json extension, they are in a compressed format and will need to be decompressed using gzip-compatible tools before they can be read as standard JSON.

For users who prefer to work with uncompressed reports or for systems that require non-compressed files, we've introduced a --no-compress option. This can be used to generate and upload test reports in the traditional, uncompressed JSON format.

Compiling and publishing JUnit XML files

Given your JUnit XML report is named results.xml you can run the following command to generate a report:

test-results publish results.xml

The above command parses the content of the results.xml file, and publishes the results to Semaphore.

While parsing the content, the CLI tries to find the best parser for your result type. The following test runners have a dedicated parser:

exunit
golang
mocha
rspec
phpunit

If a dedicated parser is not found, the CLI will parse the file using a generic parser. The generic parser uses JUnit XML Schema definition to extract data from the report.

The parser can be selected manually by using the --parser option.

test-results publish --parser exunit results.xml

The name of the generated report is based on the selected parser. If you want to overwrite this you can use the --name option:

test-results publish --name "Unit Tests" results.xml

The generated tests in a report will sometimes contain a prefix in the name. For example Elixir.MyTest. If you want to remove the Elixir prefix from the test names you can use --suite-prefix option:

test-results publish --suite-prefix "Elixir." results.xml

Multiple reports from one job

If your job generates multiple reports: integration.xml, unit.xml you can use this command to merge and publish them

test-results publish integration.xml unit.xml

In addition, each report is published separately to artifact storage as a junit-<index>.xml. <index> is a number starting from 0 that corresponds to the order of the report passed to the command line.

Merging multiple JSON reports into a single summary report

If you have multiple jobs in your pipeline that generate test results, you can merge them into a single report with the following command

test-results gen-pipeline-report

The above command assumes you are running it in a semaphore pipeline. As it uses SEMAPHORE_PIPELINE_ID environment variable to identify the pipeline and fetch the job level reports.

Where are test reports stored?

The test results CLI uses the Semaphore Artifact Storage to store the test reports:

the test-results publish command stores the report in the test-results/junit.json file on a job level
the test-results gen-pipeline-report command stores the report in the test-results/${SEMAPHORE_PIPELINE_ID}.json file on a workflow level

Expiration date of the artifacts can be controlled via Retention Policies.

Skip uploading raw JUnit XML files

By default, test-results publish will upload the raw JUnit XML file alongside the JSON report to the artifact storage. This can be disabled with the --no-raw option:

test-results publish --no-raw results.xml

Overwrite existing reports

By default test-results publish and test-results gen-pipeline-report will fail if the report is already present in the artifact storage. This behaviour can be disabled with the --force option:

# Publish the report
test-results publish results.xml

#...

# other-results will overwrite results
test-results publish --force other-results.xml

Using the CLI on a local machine

Latest CLI binaries are available here.

semaphoreci/test-results