Automatically generated end-to-end tests for apps
olemartinorg opened this issue · 2 comments
Background
As the library of deployed apps grows larger, this also increases the damage potential from any code changes (in frontend, app-lib) breaking existing functionality in deployed apps. As of right now, most apps use the latest v3 of the frontend app, and soon we'll want to migrate towards an app template were app-lib dependencies are managed automatically so that simple apps never need to be manually updated to keep getting updates.
We've had a few cases this spring/summer where seemingly innocuous changes (by me) in the frontend application have caused existing apps to crash unexpectedly. (If you're curious, one of these happened because a regex included syntax not supported in safari - a browser not routinely tested right now, and the other happened because a change to fix a warning caused a crash when the paragraph text was null.)
Proposed solution
As we're moving to make most simple apps into a collection of configuration files, we can also parse the configuration and generate automatic tests to be run in a browser. These tests could open the app, fill out the form, navigate though tracks, summaries, confirmations and receipts, and test the validity of data models produced in the end. Automatic tests could (or should) include a bit of randomness as it adds rows to repeating groups, uploads dummy attachments, possibly going back to correct mistakes on previous pages, try to submit before filling out all required fields (or adding too few rows in a repeating group), in order to get validation errors. In general though, we should be testing the happy-path where most users end up, as it's most important for us to catch show-stopping problems before we roll out new code to the public.
I would suggest implementing this by reading the page/layout/etc configuration files in the app, start the app in a closed/contained environment, and attempt to generate tests to run in a test framework like Playwright. With broad browser support, we should make sure to test in all major browsers every new release (for frontend, at least) along with mobile browsers.
Custom code in apps
Some apps have limitations on who can create instances, custom server-side validations on some fields, custom options, etc. We should make it easy for app developers to extend our tests in order to:
- Suggest valid example inputs for some fields
- Provide valid parties/roles/claims needed to successfully use/test the app via LocalTest
- Implement mocks for APIs, etc, that we do not want to execute in a test-mode
Other benefits
- With a test-mode for an app that allows us to automatically authenticate a user and instantiate an app, we are one step closer to getting a preview mode in Studio working.
- With some tooling to get the test-mode running locally, there should be less fiddling for everyone when debugging a problem reported to us, because reproducing more often can be boiled down to 'clone this repo, start the test-mode and go to this page in our form'. (Today we often also have to fiddle with an app, comment out code, etc, to get the app barely working locally.)
- Developer happiness (making changes should feel less risky), app developer happiness (better tooling, less surprises) user happiness (less frequent show-stopping bugs in production, and we can include regression tests to all apps when it happens)
Pitfalls
- If we generate code and commit to the app repos (like the additional files generated for a data model), we'll invite app developers to make changes to these files directly, thereby also limiting our ability to improve the test generation.
- If these tests are not run automatically, or if we only implement this to support the simplest of apps, this functionality could fade into obscurity and just become a burden to maintain (or worse, it will be shut down entirely). If it becomes a focus during app development, it could lead to less firefighting for everyone involved.
Potential future improvements
- In order to test a new version of the frontend on existing apps before a wide-spread release, we need to intercept the request to the frontend application and replace it. However, if we were to move the responsibility of serving the index file to app-lib instead, we could gradually roll out new frontend versions in production and add automatic rollbacks if the new frontend version crashes often. Doing this requires adding configuration options for extra styles/scripts (or other modifications app developers make to their index files).
- The ability to run the test directly from Studio, adjusting the speed of inputs. Possibly also using a time-travelling from redux-devtools to allow for pausing a test and going back.
Next steps
Implementing this is most likely quite an undertaking. We should try our hands at making a Proof of Concept, and proceed from there.
Related issues
These are both simpler solutions to the same problem (of crashing app-frontend-react bringing down the house), but will be much easier to implement (but won't give us all the same benefits, and both of these would be reactive instead of preventative):
🤩🤩🤩🤩
This sounds like a great suggestion!