Speed up automated testing (`main.yaml`) workflow runs
Closed this issue · 12 comments
Brief outline of the enhancement
Although l3build check is set to run per config and for the largest testfiles* directory per engine, main.yaml still takes 17--22min to complete now, if triggered by a Free plan GitHub user like me. For Pro plan users, it takes ~15min, because Pro plan users can run 40 concurrent jobs, twice as Free plan can. See https://github.com/latex3/latex2e/actions?query=is%3Asuccess.
Using run 5072420381 as an example. It takes 18min 4s to complete. The two slowest jobs in l3build matrix are
- "Tests: base - build - luatex", runs
l3build check -q --show-log-on-error -c build -e luatexinbasedirectory, 17min 13s - "Tests: required/latex-lab - config-block-tagging", runs
l3build check -q --show-log-on-error -c config-block-tagginginrequired/latex-labdirectory, 11min 3s
I'm not satisfied with the time duration reaching 20min. In my experiments by hand done last week,
- simply splitting the slowest job into two smaller jobs by using
l3buildoption--first <name>and--last <name>will speed up the whole workflow run to take only 10--11min. See run 4997761890. - splitting above two slowest jobs into 3 and 2 smaller jobs respectively (so each of them would finish in ~6min) won't reduce total time duration (due to 20-concurrent-job limits), but might make the total time duration more stable.
Picking marker test names and using --first <name> and --last <name> is surely fragile hence non-ideal. Other solutions:
- Do nothing until it reaches for example 30min. (
latex3tests already takes 21--27min to complete, though it's not making use of job matrix / run per config yet ...) - Pick marker test names wisely (by taking naming conventions of future test names into consideration), and get ready for adjusting marker names periodically.
- Assuming each test finishes in equal time, extend
l3buildto add support for running first m-nth portion of tests. - Split large
testfiles*directories info smaller directories, or use several configs for a sametestfiles*directory.
Which one do you think is better, for at least both latex2e and latex3 repositories?
- "Tests: required/latex-lab - config-block-tagging", runs
l3build check -q --show-log-on-error -c config-block-tagginginrequired/latex-labdirectory, 11min 3s
We could split this based on engines first, currently this isn't done since it doesn't really help given that the other job is significantly slower anyway and we are running into the job limit already.
Last I looked into --first and --last they looked too fragile to be useful here since they make it very easy to accidentally miss tests (or force you to duplicate tests). We might want to extend l3build to add a variant of --last which doesn't specify the last test but the first no longer run test (maybe called --before?). The we could use --first and the --before with the same test name and wouldn't have to worry about changing tests (we could still adjust the cutoff later to re-balance, but if we don't do that the workflow just gets slower and isn't silently missing tests)
I think I'd look at the latex2e repo separately from others (including latex3). There was a particular desire by the 'major users' of the repo to have the tests suite run faster. In contrast, for other repos, I think most of the 'regulars' are happy (enough) with the idea that the tests run exactly as they would locally, with just the doc target separate as that really is a different concept. (Indeed, it was for a long time not a test at all.)
Last I looked into
--firstand--lastthey looked too fragile to be useful here since they make it very easy to accidentally miss tests (or force you to duplicate tests). We might want to extendl3buildto add a variant of--lastwhich doesn't specify the last test but the first no longer run test (maybe called--before?). The we could use--firstand the--beforewith the same test name and wouldn't have to worry about changing tests (we could still adjust the cutoff later to re-balance, but if we don't do that the workflow just gets slower and isn't silently missing tests)
Reasonable idea: the planning behind --first and --last was really for small batches of tests to be targetted when doing some clearly-defined change.
I think there's also a mindset question here: I don't see 20 minutes as particularly long for a 'background' task that is there to back-up local testing, etc., before commits - it's still faster than running everything on my own laptop, for example, and is a lot faster than one might expect for a significant binary building process.
What are the options to skip some tests or all tests for a commit? When a branch is "under development" I don't need to run base or build all the documentation but e.g. only the latex-lab tests or none at all.
(I remember that I tried a few times [ci skip] when I knew that the tests fail, but that seemed to have no effect, I got failure messages anyway.)
I think latex2e is special and for most repos there is no point in promoting a split or some methods to run only some tests. But I agree with Ulrike that it would be nice on 2e to be able to run only "latex-lab" or "tools" through a simple mechanism. During development it is seldom necessary to run everything (except once in a while or at the very end).
This issue has been automatically marked as stale because it has not had recent activity.
@zauguin I don't know if you want to implement something that enables testing only tools or latex-lab in an easy way (if that is possible), if not then I suggest you close this one.
This issue has been automatically marked as stale because it has not had recent activity.
After commit 46f836e (Split LuaTeX main test executions in GH Actions, 2023-08-07), I think this issue is already partially done.
This issue has been automatically marked as stale because it has not had recent activity.
I think we are 'good enough' here at present: closing