testplanner

A tool for parsing testplans written in Hjson format into a data structure that can be used for:

Expanding the testplan inline within the DV document as a table;
Annotating the simulation results with testplan entries for a document driven DV execution

It is based on OpenTitan's testplanner tool, extracted as a standalone module.

Installation

To install the tool, run:

pip3 install git+https://github.com/antmicro/testplanner.git

After this, the tool is available as testplanner, e.g.:

testplanner --help

The --help flag will provide all available flags for the tool.

Testplan files

Testplan is defined in a single file in HJSON format. It consists of a list of planned tests (testpoints) and coverage items (covergroups).

Testpoint

Testpoint is an entry in the testplan representing a planned test or tests.

It consists of following fields:

name - a name of the testpoint,
desc - a description of the testpoint, see Testpoint description for more details
stage - verification stage the testpoint is assigned to
tests - list of tests implementing the testpoint
tags - list of arbitrary tags used e.g. to represent specific conditions to run testpoint tests

Covergroup

Covergroup is an entry in the testplan representing functional coverage plan for the design.

It consists of:

name - name of the covergroup
desc - description of the covergroup

Testpoint description

While the testpoint description does not have a specific structure, it is recommended to define following sections in the documentation:

Testbench - describes the architecture of the test itself,
Intent - outlines the desired effect and area of coverage,
Stimulus - enumerates the steps that are going to be taken to start and run the testbench, and describes the inputs to the test,
Check - describes the tested output and final state in which the device under test should be in.

The description can look as follows:

Testplan example

The following code represents a testplan named Example testplan with single testpoint:

{
  "name":"Example testplan",
  "testpoints": [
    {
      "name": "Example testpoint",
      "desc": '''
              Testbench:
              * Two modules connected to each other output-to-input
              Intent:
              * Test to prove that the modules are stackable.

              Stimulus:
              * Input a clock signal to the clock input of both modules.
              * Test should feed the first module a randomized sequence of data.
                * Edge cases should be taken into consideration, eg. an empty data lane.

              Check:
              * The output of the first module should be checked for correctness of calculation.
              * The output of the second module should be checked for correctness of calculation based on the results from the first module.
              '''
      "stage": "tests",
      "tests": ["test1", "test2"],
      "tags": [""]
    }
  ]
}

Including tests' results

Testplanner can parse results of tests and use them to generate summaries for testpoints and testplans, as well as associate tests and testpoints with their implementations. It expects results to be in HJSON format, one file per testplan.

Each file with results consists of:

timestamp - when tests were executed
test_results - list of tests' results

Each entry in test_results consists of:

passing - number of passed tests
total - total number of tests
job_runtime - how long did the execution of the test take
simulated_time - how long did the test take in simulation time
name - name of the test. It needs to match the name of the test in tests list in testpoints to properly associate the test's result with its definition in testplans.
file (optional) - path to the implementation of the test
lineno (optional) - line number within file where test is implemented
passing_logs (optional) - list of URLs to logs holding passed tests
failing_logs (optional) - list of URLs to logs holding passed tests

Test results can be provided with -os flag of testplanner.

Examples

To generate verification plan in Markdown, provide HJSON like in the following example:

testplanner verification_plan.hjson

Additionally you can choose output directory explicitly by providing -o flag:

testplanner verification_plan.hjson -o generated

Adding comments to testplan

Testplanner supports adding comments to summary, stages, testpoints and tests. You can do so by passing comments.hjson file using --comments-file option. The comments file consists of:

summary_comment - comment for the summary,
link_regexes - list of regexes that for creating links in comments,
<testplan> - entry for each testplan, it should be a name of testplan file without extension.

Link regexes

Each entry in link_regexes consist of:

regex - an actual regex for finding substring from which link can be created,
text - text that will be visible in the comment,
link - URL to which link will navigate.

You can use capture groups from regex in text and link properties. To do so add \\x where x should be the index of the capture group that you want to insert.

Testplan comments

Each testplan entry can consist of:

general_comment - comment for the whole testplan,
stage_comments - comments for stages,
testpoint_comments - comments for testpoints,
test_comments - comments for tests.

Withing each of these entries (except general_comment) the structure is the same:

name: comment

where name is the name of the entity (stage, testpoint, test).

Example

{
    summary_comment:
    '''
    [My project](https://link-to-my-project.com)
    '''

    link_regexes:
    [
       {
         regex: "!([0-9]+)"
         text: '!\\1'
         link: 'https://github.com/my-org/my-project/pulls/\\1'
       }
    ]

    testplan:
    {
        test_comments:
        {
            example_test: "Currently fails. Will be fixed in !123."
        }
    }
}

Using cocotb tests' results in testplanner

cocotb provides results in XML format conforming to xUnit definition of tests' results. It is possible to convert XML files from cocotb using cocotbxml-to-hjson tool.

It can be executed like so:

cocotbxml-to-hjson -i <path-to-cocotb-xml-1> <path-to-cocotb-xml-2> ... -t <testplan-hjson-1> <testplan-hjson-2> <testplan-hjson-3> ... -o <output-dir>

Where:

-i <path-to-cocotb-xml-1> <path-to-cocotb-xml-2> - list of XML results from cocotb (does not need to match list of testplans)
-t <testplan-hjson-1> <testplan-hjson-2> <testplan-hjson-3> - list of testplans in the project
-o <output-dir> - path to the directory where output HJSON files associated to testplans will be created

For more options run cocotbxml-to-hjson --help.

Interlinking tests, documentations and sources for tests

Testplanner:

Creates documentations for planned tests
Parses results of tests and creates a summary of testplans' progress
Creates general overview of testplans
Can trace implementations of tests

It is possible to interconnect all of the above for easier exploration using --testplan-file-map <file-map-yaml> flag in testplanner. This flag takes a YAML file with regex-like rules allowing to associate tests to their sources, documentations, and more.

The file passed with --testplan-file-map is managed by ResourceMap class (which can be also used in other Python scripts using testplanner to obtain testplan-related files).

The YAML is required to have testplans key in the root of the YAML (other fields are ignored, which can be used in third-party tools). The testplans is a list of rules and associated assets, such as documentation links, sources, logs and more.

Rules are in form of regex rules that may be matched against names of testplans/testpoints/tests, or testplans' file names. At each level (testplan, testpoint or test) it is possible to assign a specific resource (source, docs_html, custom resource).

testplans consists of entries with:

name - regex rule for testplan's name field
filename - regex rule for testplan's
optional resources specific to testplan, e.g. source
testpoints - list of testpoint-specific entries

testpoints is an array of entries associated with testplans' testpoints:

name - regex rule for testpoint's name field
optional resources specific to testpoint, e.g. source
tests - list of test-specific entries

tests is an array of entries associated with testpoints' tests:

name - regex rule for test's name field
resources specific to test, e.g. source

Resources supported by testplanner are:

source - path to the implementation of tests
docs_html - path or URL to the documentation regarding testplan, testpoint, test

It is possible to add other resources, which can be used by tools associating testplanner.

The first matching entry in file map will be used.

The example of file map looks as follows:

testplans:
  - filename: ".*"
    docs_html: "design-verification/{{testplan_file}}.html"
  - filename: ".*testplan_another_module.hjson"
    testpoints:
      - name: ".*"
        tests:
          - name: ".*"
            source: "**/tb_another_module_top.sv"
            vcs_logs: "**/tb_another_module_top_{{test}}.hjson"
  - name: "^.*$"
    testpoints:
      - name: "^.*$"
        tests:
          - name: "^reg_(.*)$"
            source: "**/{{testplan}}/reg/test_{{regex_groups['test'][1]}}.sv"
          - name: "^.*$"
            source: "**/{{testplan}}/test_{{test}}.sv"

Values for each resource (here docs_html and source) are template-based strings that can access special variables by using {{variable_name}} syntax.

Available variables are:

testplan - name of the testplan
testpoint - name of the testpoint
test - name of the test
testplan_file - path to the testplan file
test_source - special access to the value of the source resource, if defined earlier
regex_groups - a dictionary providing access to arrays with regex groups defined for current testplan, testpoint, test.

For example, for reg_reset_check test name in the sample_testplan testplan, the source resource will be evaluated to **/sample_testplan/reg/test_reset_check.sv. For smoke, it will evaluate to **/sample_testplan/test_smoke.sv.

Providing links to sources

The source field allows to define the glob-like search for test implementation. It can be defined on testplan, testpoint and test level.

NOTE: If the results of tests provide coordinates to the test (filename and lineno), they override path provided here. It is to allow introducing more precise location of test implementation coming from tools like cocotb.

NOTE: In documentation and simulation results, it is crucial to extinguish implemented tests from unimplemented tests. To address that, the resource mapping for individual tests works only if the resource is given at tests level.

To create a full URL path to the source, testplanner tool needs --source-url-prefix flag with URL base.

Providing links to documentation

For testplans, links to relevant documentation can be defined with docs_html. The generated string acts as a URL suffix. The prefix for the URL can be provided with testplanner --docs-url-prefix flag.