pokt-network/pocket-core

Automated e2e Testing

Closed this issue · 0 comments

iajrz commented

This proposal suggests adding an in-repo automated suite of e2e tests that can be expanded upon and help reduce the effort needed to increase the confidence of a specific release.

Goals

  • Support the tests from the functional tests repo.
  • Enable the creation of new tests
  • Reduce the time necessary to run tests in the release cycle

Non-goals

  • Increase coverage from that of existing tests
  • Fix any defects revealed by the tests
  • Mimic the tests' exact behavior if it would require changing pocket-core code to enable interfacing properly

Components:

  • Network Launcher
    • Pocket Client
    • Pocket Server
    • Network Configurations
  • Tests
    • cucumber style feature files, with scenarios written in gherkin
    • step implementation for scenario creation and execution
    • new external dependency: godog, to parse feature files and run scenarios

Details

The code to be introduced would depend on code from pocket-core, but would not affect any builds.
The complexity introduced would be self-contained in its packages, which would live under the e2e folder at the top level.

The Network Launcher compiles pocket-core, establishes datadirs for all the defined nodes, and starts the network. It contains in runtime the information defined in the network configuration.

Network configurations allow defining a genesis file, node configurations, and hold the private keys for the actors of the network.

The Pocket Client and Pocket Server can be run with any command-line arguments. We get to read their full stdout and stderr, which allows for inspecting state transitions as evidenced by, for example, the debug log.

The tests are defined in a twofold process:

  • Design and implementation of the scenarios, choosing or defining the "steps" required to verify that the system is behaving the way we want. This should be biased towards reusing existing steps. This is done in gherkin.
  • Design and implementation of steps so they work together properly. This includes designing the state that should be kept from one step to the other and how it will be made available to a future step. This is done in go.