etf-validator/governance

Test Run Templates

Closed this issue · 3 comments

jonherrmann commented

ETF Improvement Proposal (EIP)

Background and Motivation:

This EIP supplements EIP #26 . In addition to the creation of test objects,
it should also be possible to pre-select a set of ETS and parameters and save
these as one element to execute all ETSs easily together. Furthermore a
test object should also be presettable.

Proposed change

Add a new model item Test Run Template item that possesses the following
properties:

  • an EID
  • a label
  • a description
  • a version
  • the language for reports
  • a parameter list to parameterize the ETS
  • a set of Executable Test Suites
  • a set of Test Objects (optional)

An example instance which would run all INSPIRE Metadata ETS could look like this:

<TestRunTemplate id="EID...">
  <label>Metadata (Technical Guidance 1.3)</label>
  <description>Run INSPIRE Metadata (TG 1.3) tests against the files on network drive X</description>
  <language>de-DE</language>
  <ParameterList>
    <parameter name="Files_to_test">.*</parameter>
  </ParameterList>
  <ExecutableTestSuite>
    <!-- Conformance Class 'INSPIRE Profile based on EN ISO 19115 and EN ISO 19119' -->
    <executableTestSuite ref="EID..."/>
    <!-- Conformance class 'Metadata for interoperability' -->
    <executableTestSuite ref="EID..."/>
  </ExecutableTestSuite>
  <TestObjects>
    <!-- Test Object with all test data on network drive X -->
    <testObject ref="EID..."/>
  </TestObjects>
</TestRunTemplate>

TestRunTemplates are Metadata items and can be stored like Tags as XML in the Metadata directory. They will be read and loaded like Tags during startup.

The user should be able to:

  • create a Test Run Template, by selecting one or multiple ETSs and optionally
    a Test Object
  • delete an existing Test Run Template
  • clone an existing Test Run Template and modify it
  • Select and execute one Test Run Template

Cloning an existing Test Run Template would create a clone with a new EID.
Changing an existing instance of a test run template should not be possible.
The cloning feature could be configured through a property to activate/deactivate it:

etf.testruntemplates.cloning.allow = true/false

The default would be true for organisation-internal workflows and false for public workflows.

The REST interface endpoint would be accessible via /v2/TestRunTemplates .

If the user selects Test Run Template in the user interface and no Test Object
has been pre-selected in the template, the user interface should ask which
Test Object should be taken.

A draft for the workflow and the web interface is described in EIP #27 .

This EIP is about the new REST interface which should be implemented before
the changes in the web interface and the implementation of
the complete workflow.

Alternatives

none considered

Funding

Full funding available.

Additional information

  • This EIP is backwards compatible with existing ETSs
  • For the sake of simplicity, only a single test object would initially
    be supported in a Test Run Template.
  • Since this is an addition to the existing interface, the main API version
    number v2 does not need to be increased, and existing external clients do not
    need to be modified
  • This EIP requires changes in the ETF core, the data storage and the web
    application
jonherrmann commented

Todo:

  • describe configuration option to deactivate local Test Run Template cloning
jonherrmann commented

Configuration description updated inline, see 'etf.testruntemplates.cloning.allow'

jonherrmann commented

Implemented in Version 2.1.0