Onboard to Appium for Windows UI unit testing
Closed this issue · 5 comments
We should unit test our UI, probably. Quite a few issues arise from UI changes that would probably be easier caught were we to unit test out stuff.
There's a few options for doing so; namely, just testing Wpf and using one of several available Wpf unit test libraries, or exploring something like appium [1] which provides testing drivers for multiple platforms.
Putting this in the v0.3 milestone as I'd like to have it before we upgrade to Eto latest so we can check for any UI regressions.
Renamed the issue break it down from actually writing the UI unit tests, which is a separate issue.
I think I might take a crack at this and see if I can get a basic test working.
I have some basic Appium tests working locally. The biggest problem we're going to have is getting them working on Azure Pipelines in PRs. Some problems that I'm currently trying to solve:
- On Windows, we want to be able to test Docker-based ASM patching. However, on the Azure Pipelines Windows hosted machines, Docker runs in Windows-container mode by default and there doesn't seem to be a way to switch it to WSL-backed containers (and according to this, the containers might only have WSLv1 enabled anyway).
- There's also the problem of getting devkitARM onto the machines. On Linux, we could try creating a Docker image based off the official devkitARM Docker image which now runs on Debian Bullseye (they updated it finally). For OSX and Windows, this is potentially a bit more difficult. There are OSX Docker images that we could potentially pull and create our own devkitARM/Appium macOS Docker images. However, according to the docs, those containers require bare minimum 20GB of space and the macOS hosted machines have 14GB of space. This leaves us with two options:
- Run on the hosted macOS machines and bootstrap devkitARM onto the machine from a portable zip that we copy into /opt
- The containers apparently can run on Linux, so we could create a self-hosted Linux agent (as described below) with more disk space and then run on top of that
A solution to several of these problems would be to use a customized self-hosted agent. This would allow us to prep the machines with whatever software we need in advance. It would limit our parallelism, but we're already limited to 10 parallel jobs on the public hosted machines anyway, so I don't see that as a significant issue. The main issue is cost -- I currently use about $111 of my $150 allotted monthly Azure credits, but I am going to take steps to attempt to reduce that number. If we then use relatively cheap VMs for this (burst VMs with spot allocation would work great -- a B2s v2 on spot would run us $15.18/month. Add 64GB OS disks onto each of them would be ~$6 each meaning (after networking and whatnot) we could potentially run three agents (one Win, one Linux with dockerize macOS, and one native Linux). I'm going to try setting up custom hosted agents and see how that works.
Scoping this issue further bc Linux & Mac UI testing, while important, are a lot harder.