Explore feasibility of integration tests in Github Actions
Closed this issue · 8 comments
- Test viability of launching FoundryVTT in a github action
- Verify github actions are not overly public on public repositories
- Determine method for caching downloaded FoundryVTT server for launch
- Fix checkout to actually use the pull request's code
- Determine how to mount the active codebase into the module (probably via a volume mount)
- Automate the initialization of foundry and the installation of the star wars module
Initial tests were successful for running the felddy/foundryvtt container as a github action service using github action secrets to store my foundry username/password. Additionally, reviewing both the documentation and the live interface, it doesn't look like there's a way to download job caches outside of a github action. That means we should be able to safely store the downloaded foundry server in the cache.
My current issue is I don't think services can actually utilize the cache, which might make the above implementation infeasible.
Unfortunately, I don't think there is a way to restore the cache prior to launching a service.
Fortunately, the way github actions are designed, it seems that I can use the parent host without a container, and execute all the docker steps manually. It won't be nearly as clean, but it should work.
Okay, I have a PoC of a single Cypress test with FoundryVTT via Github actions:
- Without cache: https://github.com/bheiskell/StarWarsFFG-Enhancements/actions/runs/4098645009/jobs/7067907707
- With cache: https://github.com/bheiskell/StarWarsFFG-Enhancements/actions/runs/4098645009/jobs/7067916362
The first job has a step called Launch FoundryVTT and run Cypress Test, which you can see in the log downloads the server.
Entrypoint | 2023-02-05 21:58:01 | [info] Using CONTAINER_CACHE: /data/container_cache
Entrypoint | 2023-02-05 21:58:01 | [info] Downloading Foundry Virtual Tabletop release.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
3 191M 3 6935k 0 0 6142k 0 0:00:31 0:00:01 0:00:30 6137k
32 191M 32 61.4M 0 0 29.1M 0 0:00:06 0:00:02 0:00:04 29.1M
61 191M 61 118M 0 0 37.8M 0 0:00:05 0:00:03 0:00:02 37.8M
90 191M 90 173M 0 0 42.1M 0 0:00:04 0:00:04 --:--:-- 42.1M
100 191M 100 191M 0 0 43.2M 0 0:00:04 0:00:04 --:--:-- 44.3M
Entrypoint | 2023-02-05 21:58:06 | [info] Installing Foundry Virtual Tabletop 10.291
The second job has a step called Launch FoundryVTT and run Cypress Test (container cached).
Entrypoint | 2023-02-05 22:00:08 | [info] Using CONTAINER_CACHE: /data/container_cache
Entrypoint | 2023-02-05 22:00:08 | [info] Installing Foundry Virtual Tabletop 10.291
Two very cool things.
-
The default cypress actions have a nice table of results attached as an artifact
-
It was trivial to attach the cypress video recordings of the runs, which should really help with troubleshooting / verifying changes. Here's an example: https://user-images.githubusercontent.com/96167/216848549-63c56552-9596-4b6b-a270-8ad5562d0814.mp4
My code to achieve this setup still needs a lot of love.
Additionally, there are two major tasks still needed to have a full PoC.
- I still need to figure out how to download or mount the module code into the environment.
- I need to automate, likely via cypress, the basic setup of the system in foundry
After demoing my progress to @wrycu we paired researching how to lock down access to the secrets. I had a plan for how to do this, but it turns out Github's restrictions didn't work exactly as I expected.
Using the event pull_request grants access to secrets to changes pushed to branches on the upstream repository (i.e., contributors). To grant access to secrets from pull requests on forks, the pull_request_target event can be used, which executes in the context of the target branch on the main repository. The latter event however can expose our foundry credentials, since it runs unsafe code (like npm install) that the fork can tampered with. Github by default now requires approval for the first time a new contributor creates a pull request, but that's not sufficient since folks may emulate this pattern. To work around that, an environment that requires an explicit approver can be added to the workflow. That creates a step where an approver has to manually run the job (after verifying the PR doesn't do anything nefarious).
However, the github actions yaml configuration doesn't make it easy to only add that approval for forks. I extracted the core of the workflow to a "reusable workflow" and used it in two workflows: one for branches coming from the upstream repository that don't require the approvals, and one that comes from forks that does.
I've made some progress here. The core problem issue I'm running into now is around dismissing popups.
home.cy.js.mp4
Cypress does well when interacting with a known state. However the pop-ups/notifications aren't always repeatable. If I hard code them, it will end up being brittle in another way. I need to think through a different strategy that can track the pop-up dismissal in a way that doesn't require an arbitrary timeout.
Edit: Hah, well this is what I forget for working on this until 3am. I forgot to mount the code base into the docker container in the github action. I'll tackle that next.
I've made pretty significant progress here, but I'm hitting a wall.
This is a video of the tests running locally via the commandline:
home.cy.js-1.mp4
This is a video of the same test running in github actions:
home.cy.js.mp4
You can see that the video recording is extremely choppy. Ultimately, the test in CI fails. I strongly suspect that the CI workers are just demanding too much from the 2 core/7 gig runners.
Theoretically, this should be as simple as extending timeouts to really large numbers to give the headless browser time to load. In practice though, I think it's bumping into conditions that show up when the browser is crushed under load. I haven't given up on the free runners yet. The errors are at mostly consistent in CI, and they seem to trigger on specific interactions that perhaps trick cypress into thinking the element is ready when it's not.
Okay, I spent some more time working through the why's for each arbitrary wait. I figured out a better (but not perfect) way to wait for the page to be initialized. The game engine exposes game.ready and game.canvas.ready. The side bar buttons explicitly check for the canvas to be ready and I noticed a race there. The star wars engine itself adds the destiny pool tracker close to last in initialization, so that's the best ready state I have for it at the moment.
I also removed as many force: trues as possible, as those I think also are a source of brittleness, clicking a button before it might be ready.
Lastly, I didn't notice it before, but there's a page reload triggered by closing a dialog complaining about a plugin not being enabled (because it disables the setting upon close, which triggers an event we have configured to reload the page). Adding the page initialization hooks there has got the build working!
https://github.com/bheiskell/StarWarsFFG-Enhancements/actions/runs/4158875529/jobs/7194392961
home.cy.js.mp4
Done!