/booster-testing-utilities

Patterns, utilities, and code examples for testing Booster.cloud applications.

Primary LanguageTypeScript

Booster Testing Utilities

⭐️ Heads Up ⭐️

This is currently a work in progress - and opinionated.

I'm sharing it mid-stream in case it can help others work through similar questions for testing Booster apps, or better understand Booster's currently undocumented built-in testing tool.

(I think) I'm fairly skilled in creating event-driven applications, working with Booster, and creating tools in JS/Typescript, but I'm relatively new to writing tests.

Most of this work is based on a common-sense approach but if there something critical you think I'm missing in my thinking or approach, please let me know.

Currently, I'm sorting the details for -

  • the levels and types of testing that make sense for a Booster/EDA app
  • the helper utilities that can help support, standardize, and simplify tests
  • better documentation of everything going on here

I'm doing this next across some real-world projects. My hope is to update this repo as I learn what works best in practice.

In the mean-time, please see the notes below for a high-level overview and dig into the code examples for specifics.

Updates

TL;DR

  • Goal is to provide patterns and utilities to more quickly and reliably test Booster apps.
  • This repo example is using Vitest for testing, but this can be updated to another testing framework.
  • Using Booster's application-tester package across all examples to simplify testing.
  • Includes custom updates to Booster's local-infrastructure 'test-helper' package to allow the Booster application-tester to be used for both local and deployed tests. (The current Booster package functionality for local testing is incomplete).
  • Includes opinionated code-examples and helper utilities to test opinionated ideas of the expected behavior of an application and the code hygiene of Booster elements (e.g. commands, entities, etc.).
  • Also includes helper utilities that abstract test creation substantially, inferring as much data and intention as possible from the tested files themselves via code or special @ notation. (The jury is still out what level of this is useful (e.g. don't want to accidentally create self-fulfilling tests).

Test Relevant Files in Repo

The things you'll want to bring into your own project to replicate this testing approach.

  • dependencies and 'test:*' scripts: /package.json
  • Booster config setup: /src/config/config.ts (updates to 'local' & 'test' environments)
  • testing framework config: /vitest.config.ts
  • testing constant references: /test/constant.ts
  • testing global startup/tear-down: /test/globalSetup.ts (backup/restore datastores for local testing)
  • testing helper utilities: /test/test-helpers/* (utilities to simplify/standardize tests)
  • testing JWT key: /keys/testing-public.key (to decode JWTs created by application-tester)
  • 'framework-provider-local-infrastructure' custom 'test-helper': /test/test-helpers/custom-local-test-helper/* (in-app 'package' override until Booster package completes functionality for local testing)

Current Rough & Dirty Notes

Purpose

  • Pull together useful patterns, utilities, and code examples to quickly and confidently test Booster applications.

Getting Started

  • Clone repo
  • Install dependencies
  • Perform steps in Running Tests: Testing Local Environment below
  • Take a look at -
    • example command
    • example command test #1, with inline methods that use application-tester
    • example command test #2, that does same work using helper utilities
    • example command test #3, that uses helper utilities and also infers test data from source file
    • example command test #4, that fully automates test creation via source file @comments and helper utilities

Application

This application is purposely simple, but complete with common elements like roles, validation, handler processing, event-handlers, and projected read-models to test.

Sorely lacking more comprehensive documentation on the details yet, but take a peek at src/commands/order-snack.ts and src/commands/order-cocktail.ts to get a feel for how the application works. (The funny '@syntax' comments included within these are part of an idea to possibly help automate some tests — jury is still out on whether this makes sense or not).

Running Tests

Testing Local Environment

  • in 1st terminal:
    • run npm run start to start app server
    • in this terminal, will see output from app server
  • once app server running, in 2nd terminal:
    • run npm run test:local (or npm run test:local+ui to open test UI in browser)
    • in this terminal, will see output from Vitest test runner
    • within this terminal, press a to rerun all tests, if wanted

Local Testing Datastores

  • local testing attempts to backup and restore the event and read model datastores within the /.booster directory
    • currently, this is a simple backup/restore mechanism that can encounter issues (see below)
      • if it is critical to preserve local datastores' state before testing, also back these up manually
    • backup and restore is done via setup and teardown methods in /test/globalSetup.ts
  • backup/restore lifecycle
    • when testing is started it will backup all files within the /.booster directory (normally, there should only be 2 files)
    • while testing is running you can inspect and edit how testing is affecting the datastores (event.json and read_models.json files)
    • when testing is stopped it will delete the testing datastores and restore the original ones
  • when testing crashes (e.g. due to a test error), the teardown methods do not run and both the testing and backup datastores are left in place
    • if you start testing again, the setup methods will reproduce all of these files, creating more backups and not being able to successfully restore the original datastores
    • in these cases you'll need to eventually clean up, and optionally restore, the local datastores manually

Testing Deployed Environment (AWS)

  • run npm run test:live
    • will deploy test stack to AWS
  • if make changes to code
    • in new terminal run boost deploy -e test
    • in original terminal (where tests running), press a to re-run tests
  • when cancel process in original terminal, should nuke deployed test stack

Source File Requirements

There are a few things tests currently need or can use in Command source files:

  • an tid input parameter
    • this is currently required for tests to find the entries they created in the datastores
    • it should be of type 'string'
    • it can be an optional input in the Command's constructor
    • example
  • optional @syntax comments that can inform testing automation

Testing Tools

  • This repo uses Vitest to run tests - but that is not required. Vitest has a syntax very similar to Jest or Mocha+Chai, but runs tests more quickly and offers features like HMR. If you'd prefer to use Jest or Mocha+Chai, you should be able to utilize most of the patterns and utilities with minimal revisions.

  • It also utilizes Booster's application-tester package to simplify testing and normalize test code for different environments (e.g. local & AWS).

  • It includes a locally linked, customized version of the 'test-helper' from the framework-provider-local-infrastructure package with updates to put in in parity with the 'test-helper' methods afforded by other provider packages (e.g. AWS).

Testing Scope

I'm still trying to determine what level of testing is useful/necessary in a Booster application. E.g True unit testing via executing imported classes? Pseudo-unit testing via testing effects? Specific integrations testing?

My understanding and appreciation are certainly limited by my current testing experience but I'm operating on a few assumptions at the moment:

  1. the Booster framework appears to have good unit/integration testing coverage of its core functionality
  2. if we can rely on these to ensure the framework is working as intended (e.g. commands processing successfully, events reducing into entities, entities projecting into read models, etc.) than the job at hand for us is to test:
  • the intended behavior of the application and its processes
  • the code hygiene of the application's parts (i.e. are we following a desired practice like input validation?)
  1. we can test intended behavior/processes via e2e/functional tests
  • if e2e/functional tests can be created more quickly and reliably, we may not need to create (as many) unit tests to confirm the functionality of the individual parts
  1. we can test code hygiene via pseudo-unit tests

Again, this is definitely an area I'm operating on some shallow knowledge and big assumptions, but hopeful there is leverage to be found that can help make testing quick and part of the creation process, while still being reliable.

Tested Patterns

For testing, I'm trying to think backward from the responsibility and/or expected effect(s) of each element in the system to determine what assertions and expectations are necessary. The goal is to sort out what's the least we can get away with testing (above and beyond the framework testing that's already occurring) to confirm expected functionality and behavior.

Legend

  • ✅ = example and utility added
  • ⏩ = will likely be derivative of an existing example/utility
  • 🚧 = working on this currently
  • 🤔 = not certain yet if this is needed; may be accounted for in Booster framework infrastructure tests

A Process

Note: 'process' in this context is something that can be initiated by an actor (command) or on a schedule (scheduled-command), may have 0 or more event-handlers to perform its work, and results in some expected change in the system (i.e. a request and activity that may span more than a single handler).

  • 🚧 should perform correct authorization OR correct schedule
  • 🚧 should accept specific parameter(s)
  • 🚧 should perform certain work
  • 🚧 should save specific data to entity(ies)
  • 🚧 may make specific data visible from entity(ies)

A Command

Note: finished building out items below and currently re-evaluating which of these are relevant to asserting/testing expected behavior or code hygiene vs. testing the reliability of framework.

  • ✅ should perform correct authorization
  • ✅ 🤔 should accept specific parameter(s)
  • ✅ should fail when required parameters are missing
  • ✅ should succeed if submitting only required parameter(s)
  • ✅ should fail if parameters values are empty ('')
  • ✅ 🤔 should fail if parameters are invalid type
  • ✅ should perform certain work
  • ✅ should register specific event(s)
  • ? should be able to receive same command repeatedly or specific number of times

A Scheduled Command

An Event Handler

  • should be called when a specific event is emitted
  • ⏩ should accept specific parameter(s)
  • ⏩ should fail when required parameters are missing
  • ⏩ should succeed if submitting only required parameter(s)
  • ⏩ should fail if parameters values are empty ('')
  • ⏩ should fail if parameters are invalid type
  • ⏩ should perform certain work
  • ⏩ should register specific event(s)
  • should be able to receive same event without duplicating work
  • should be able to process events out of order

An Event

  • should have specific parameter(s)
  • ? should update specific entity(ies) (not sure yet if knowledge of this should really be an event concern)

An Entity

A Read Model

Testing Utilities

Included are a number of helper utilities to help simplify and standardize tests. These are still maturing, and range from generic utilities to test-specific utilities.

Testing Automation

Note: some helper utilities abstract the test creation substantially (e.g. within helpers-command.ts) and attempt to infer as much data as possible from the tested files themselves. Jury is still out what level of this is useful (e.g. don't want to accidentally create self-fulfilling tests).

@syntax comments are optional for test automation.

Command: Registered Events @comments

  • must be first arguments in event constructor
  • must be in this order:
    • @requiredInputs: input variables required to trigger event
    • @aReducingEntity: an entity that will be updated by event (required to lookup event in datastore)

Command: Work to be Done @comments

  • can be any where in document
  • can include any unique (within document) two-digit key
  • require:
    • @work00: brief description of work to be done
    • @work00-inputs: input name and value (e.g. { fruit: 'apple', drink: string })
      • can use explicit value or generic (e.g. string, number, boolean, id) to generate fake values
    • @work01-entity: an entity that will be updated by event (required to lookup event snapshot in datastore)
    • @work01-shouldHave: value to test if work was done
      • can be true or false if want to test yes/no of work done
      • can be an array of one or more strings and/or numbers to test if all values exist within result
      • example

Repo Updates

Pseudo-changelog of updates to repo since initial publish - helpful if you cloned an earlier version and want to check what's new.

04-25-22

  • Updated test-helpers/helper-readmodels
    • returned list of items instead of boolean test
    • removed fixed inline filter for tid value within id field

    • have only tested these with local provider thus far

  • Updated test-helpers/helper-misc
    • add utility to test if string is JSON or not
  • Updated test-helpers/helper-commands
    • updated wasWorkDone method to account for stringified JSON values
    • added 'shouldNotHave' option to wasWorkDone method

04-24-22

  • Added test-helpers/helper-readmodels
    • stubbed out initial methods for testing data projected into read models

    • have only tested these with local provider thus far

04-21-22

  • Updated testing id input parameter
    • to use string instead of UUID to make it more flexible for user input in testing
    • renamed from id to tid to help avoid future possible conflicts
  • Updated notes
    • added "Source File Requirements" section with notes about currently required tid in commands
    • added notes to "Testing Local Environment" about methods that backup/restore local datastores in /test/globalSetup.ts
  • Updated test-helpers/helper-commands
    • fix wasWorkDone and wasEventRegistered methods to utilize a tid value manually set in command request
    • allow multiple values to be used for Work to be Done shouldHave values (true|false or array of 1+ string/number values)

04-20-22

  • Added start:testing script and comment to package.json to utilize MODE=test variable for Vitest tests.
  • Changed @work00-result syntax to @work00-shouldHave
  • Normalized input format/syntax for command inputs and work-to-be-done inputs
  • Updated test-helpers/helper-commands
    • update getRegisteredEvents utility regex to better match various code patterns.
    • update getWorkToBeDone utility to use '-shouldHave' syntax
    • add getWorkToBeDone errors if a @work comment set is incomplete
    • update wasWorkDone utility to search across nested record data for '-shouldHave' value
    • relabel 'parameter' to 'input' across utilities and automation @syntax
  • Updated custom-local-test-helper/local-queries
    • filter results by kind ('event' v 'snapshot')