Plugin Test Matrix

This repository contains regularly scheduled continuous-integration for the official PennyLane plugins.

The continuous integration is performed by GitHub actions. Tests are executed on every push to master, and are also run on a schedule of once per day.

Finally, tests can be triggered manually via the 'Actions' tab.

Testing matrix

All entries in the matrix are tested against PennyLane latest (GitHub master).

	PyPI version	Stable plugin/stable PennyLane	Stable plugin/latest PennyLane	Latest plugin/ stable PennyLane	Latest plugin/latest PennyLane
Qiskit
Cirq
Qulacs
AQT
IonQ
Rigetti
Lightning
Braket
Quantum Inspire

Notes:

The device integration tests currently do not support devices with a fixed number of wires; as a result, rigetti.qvm is not tested.
The Braket plugin device integration tests are run with -k “not Sample and not no_0_shots”, see #6
The Qiskit tests above are run with -k 'not test_ibmq.py and not test_runtime.py', that is, without using devices that access the IBMQ backend. The IBMQ backend is tested via test_ibmp.qpy and test_runtime.py tests on the latest version of Pennylane and the plugin twice a week. Their status is shown below:

	Status
Qiskit-IBMQ

QML repo

	Status
`build-branch-dev`
`build-branch-master`

Lightning Docker builds

	Status
Docker build `master`
Docker build `latest`

Catalyst compiler

The following table shows the current Catalyst compatibility with the Lightning and PennyLane packages, where each of the 3 packages is tested as "latest" and "stable". Latest versions are built directly from the repository's main or master branch, while stable is built from the repository at the most recent release tag. The reason this is needed rather than installing PyPI packages is that the Lightning dependency in Catalyst needs to be built from source and cannot be obtained from the PyPI package. PennyLane stable is installed from PyPI.

	PennyLane stable		PennyLane latest
	Lightning stable	Lightning latest	Lightning stable	Lightning latest
Catalyst stable
Catalyst latest

Interpreting the test matrix

Stable/stable: This reflects a scenario where PennyLane and the plugin are both installed from PyPI. This column should always pass. If this test fails, it indicates that either (a) a bug exists, or (b) an upstream dependency (such as Qiskit or Qulacs) were released with breaking changes. In both cases, a bugfix release of the plugin should be made.
Stable/latest passing: A new plugin release is not essential; the current PyPI release will continue to work once the new PennyLane version is released.
- Latest/latest passing: Nothing to be done! However, it is still worth checking the plugin repository to see if new unreleased changes have been added that would be worth releasing.
- Latest/latest failing: Indicates that recent changes have been made to the plugin repository post-release; further updates must be made to the plugin to ensure GitHub master passes.
Stable/latest failing: A new plugin release is required for compatibility with the upcoming PennyLane release.
- Latest/latest failing: Updates must be made to the plugin to ensure GitHub master passes.
- Latest/stable failing: The requirement of PennyLane in the plugin will have to be increased.
- Latest/latest passing: Nothing to be done! The plugin is ready to be released with the upcoming PennyLane release.

Feature freeze testing matrix

All entries in the matrix are tested against the PennyLane release candidate branch (GitHub vX.XX.X-rc0 branch).

	Latest plugin/ release candidate PennyLane
Qiskit
Cirq
Qulacs
AQT
IonQ
Rigetti
Lightning
Catalyst
Braket
Quantum Inspire

Adding a plugin

Two Jinja2 workflow templates are provided, that makes it easier to add a new plugin to the test matrix:

workflow-template-latest.yml: for testing plugins against PennyLane latest
workflow-template-stable.yml: for testing plugins against PennyLane stable

Simply add a new plugin to the workflows list in compile.py, with the following dictionary keys:

plugin (required): the name of the plugin not including the PennyLane prefix. The plugin repository and PyPI project are inferred from the plugin name. E.g., 'plugin': 'qiskit' will correspond to the GitHub repo PennyLaneAI/pennylane-qiskit and the PyPI project pennylane-qiskit.
gh_user (required): the GitHub user or organization which hosts the plugin repository.
which (required): a set specifying whether to generate a workflow to test the {"latest"} (GitHub master) version of the plugin, {"stable"} (PyPI) version of the plugin, or both {"latest", "stable"}.
requirements (optional): a list of Python packages that should be installed prior to plugin installation. You may use pip syntax, e.g., pyscf==1.7.2.
requirements_latest (optional): additional list of Python packages that should be installed prior to latest/GitHub master plugin installation. For example, if the latest version of the plugin depends on a development version of a particular framework.

device_tests (optional): a list of command line arguments to pass to the PennyLane device integration tests. Each list element corresponds to a single test run, e.g.,

"device_tests":
    [
        "--device=cirq.simulator --tb=short --skip-ops --analytic=True",
        "--device=cirq.simulator --tb=short --skip-ops --analytic=False --shots=8000"
    ]

additional_setup (optional): multiline string containing additional GitHub actions for execution before the installment of plugins.
token (optional): sets an environment variable with the same name as the token. The value is loaded from the repository secrets. The token is required by some plugins for backend authentication.
additional_env_vars (optional): string containing additional environment variables you'd like to be set for the CI action. Double-check the whitespace when using this - variables should be separated with "\n "
runs_on (optional): string to override the workflow's default runs-on attribute of ubuntu-latest.
test_kwargs (optional): additional arguments to pass to pytest for the given plugin.
no_deprecation_error (optional): set to True to not raise PL deprecation warnings as errors when testing the latest version of the plugin. By default, PL deprecation warnings are raised as errors.

Once you have added your plugin, run

$ python compile.py

This will autogenerate up to five workflow files, depending on the variable which:

.github/workflows/plugin-stable-stable.yml
.github/workflows/plugin-stable-latest.yml
.github/workflows/plugin-latest-stable.yml
.github/workflows/plugin-latest-latest.yml
.github/workflows/plugin-latest-rc.yml

Finally, make sure to add a row to the testing matrix above!

PennyLaneAI/plugin-test-matrix