WeTest

Tests automation utility for EPICS.

WeTest reads YAML files describing tests targeting EPICS PV, run these tests in CLI with or without a GUI, and generates a report in PDF.

Installation

Python and tkinter

WeTest is implemented in Python 3, which needs to be installed.

The tkinter module should be included in your python installation. If not, you need to also install the corresponding package. This module is needed for the graphical interface.

For example in Ubuntu:

sudo apt-get install python-tk

For example in CentOS:

sudo yum install tkinter

WeTest is currently successfully running on CentOS 7 and Ubuntu 18.04 (support for Windows is planned but not yet available).

Build using Nix

First, install Nix. Refer to the Download Nix page for instructions.

Then, run:

nix-build

This builds WeTest and its dependencies, and puts the compilation result under ./result.

After running the build, you can run WeTest by running:

./result/bin/wetest

If you want WeTest available in your global user environment, then you can run:

nix-env -i -f default.nix

Build using Poetry

First, install Poetry. Refer to the Poetry documentation for instructions.

To build the project, you can run:

poetry install
poetry build

Running WeTest can be done with:

poetry run wetest [other arguments]

Set up pyepics

Setting up pyepics is not needed for the Nix build.

WeTest relies on the pyepics module, and depending on your EPICS installation you might need to set up the PYEPICS_LIBCA environment variable manually.

For instance:

export PYEPICS_LIBCA=/opt/epics/bases/base-3.14.12.5/lib/centos7-x86_64/libca.so

You can find full explanation in the pyepics installation page.

Run WeTest

Running tests

To run WeTest with a scenario file, use either:

wetest <scenario.yaml>
wetest -s <scenario.yaml>
wetest --scenario <scenario.yaml>

You can provide several scenarios files in the CLI, or you may use the include block within a scenario file.

PV connection and naming

You can run WeTest without a scenario file if you provide an EPICS DB directory or DB files. In that case no test will run and WeTest will monitor the PV connection status.

wetest -d <epics_db_folder>
wetest --db <epics_db_folder> --naming ESS

By default WeTest will only the PVs found in the tests. Using the --db option adds the new PVs to the PV list extracted from the tests.

The --naming or -n option checks the conformity of the PV name to the naming convention, and sort the PVs by sections and subsections.

change PDF report name

At the end of the test runs, WeTest generates a PDF report in the current directory under the name wetest-results.pdf. You can change this name using the --pdf-output option.

wetest <scenario.yaml> -o <another_name.pdf>
wetest <scenario.yaml> --pdf-output  <another_location/another_name.pdf>

Writing scenarios files

Scenario files are YAML files.

As of today the only documentation about the content of scenario files are:

the PowerPoint presentation in the doc folder
the YAML schema validation file also in the doc folder

A user manual is planned but not yet available.

One may also find interesting example in the test directory.

Pre-made scenarios are also available in the generic folder. These files are meant to be included in another scenario with macros to activate and customize tests.

Here is another summary.

Content expected in a scenario file:

version: checks the compatibility of the file against the running version of WeTest
config: to define a scenario name, PV prefix, default values, and whether the tests should be shuffled (unit) or run in the order defined (functional)
tests: a list of test, see below for the expected content of a test
macros: macros can be defined here and will be substituted throughout the file
include: a list of scenario to run, macros can be provided to each included scenario, tests are run last by default, but this can be changed by adding test somewhere in the include list.
name: a name for the suite (report and GUI title), only the topmost file suitename will be used, included files suitename will be ignored. This parameter is also ignored when providing multiple files in the command line.

If a tests block is defined, then a config block is expected too, and conversely.

Content expected in a test:

name: test title, displayed CLI in GUI and report
prefix: string appended after the configuration prefix and before the PV name
use_prefix: whether to use append prefix from config
delay: wait time between setting and getting the PVs
message: free length description for the test, displayed in GUI and report
setter: name of the PV where to write a value
getter: name of the PV from where to read a value
margin: lets you to set a percentage tolerance when checking the value
delta: lets you to set a constant tolerance when checking the value
finally: put back to a known configuration
ignore: whether the test shouldn't be read
skip: whether the test shouldn't be run
on_failure: determines whether to continue, pause, or abort if the test fails
retry: number of time to try again the test if it failed
range: generate a list of values to test, available sub-fields are:
- start: starting value
- stop: end value
- step: space between values from start to stop
- lin: number of values linearly spaced between start and stop
- geom: number of values geometrically spaced between start and stop
- include_start: whether to test the start value
- include_stop: whether to test the stop value
- sort: whether the values should shuffled or ordered
values: custom list of value to test
commands: a list of highly customizable tests, available sub-fields are:
- name: command title, displayed CLI in GUI and report
- message: free length description for the command, displayed in GUI and report
- margin: lets to set a percentage tolerance when checking the value
- delta: lets to set a constant tolerance when checking the value
- setter: overrides the test setter PV name
- getter: overrides the test getter PV name
- get_value: value expected in the getter PV
- set_value: value to put in the setter PV
- value: same value for setter and getter PV
- delay: overrides the test delay
- ignore: ignore this command even if test isn't ignored
- skip: whether the command shouldn't be run
- on_failure: continue, pause, or abort, if the command fails

A test should have one of the three "test kind": range, values, or commands.

Development

This section is under works.

Pull requests, feature requests, and bug reports are welcome.

Aside from a proper user manual, proper non-regression tests are today lacking, so development might be a bit slow.

epics-extensions/WeTest