Run tests against installed rgbds, not on built binaries
Closed this issue · 9 comments
In the CI we're setting up to package for Debian, we need to run the tests against the installed package, not on the built executable, so "run-tests.sh" fails because it checks for the compiled binaries in the parent directory (
Line 38 in 727c1f5
We could add a flag that runs "run-tests" expecting the executables to be found in the path, without checking for the built binaries..
test_downstream() in tests/run-tests.sh passes a Make variable RGBDS to the repositories that it builds. This points to the prefix of the RGBDS build under test. So we'd need a variable that defaults to "$(realpath ..)" but which a caller can override. For example, to test installed RGBDS, the caller would override it with something like "$(dirname "$(command -v rgbasm)")". Should this variable be passed to tests/run-tests.sh as a command-line option or as an environment variable?
I'd go with a CLI option: run-tests.sh --rgbds ../path/.
I believe neither of those is necessary.
The makefiles default to the rgbds installation in PATH, meaning a simple switch to disable the RGBDS variable assignment should do the trick.
I was going to suggest "just do cp $(which rgb{asm,link,fix,gfx}) . and then run the tests normally", but then saw that your rgbds-tests script has a line very similar to that already commented out (#cp build/src/rgbasm build/src/rgblink build/src/rgbfix build/src/rgbgfx .) Why, did it not work, or did the Debian maintainers say not to?
This was part of an older test script (and I should honestly remove it).
I'd like to test against the system installation and not copied binaries for the most accurate results.
FWIW, its normal for the testsuite to test the built local binaries and not the system binaries. This is especially important for distros where tests may be run when rgbds is not yet installed and when you want to be sure to test the newly built binaries and not the preexisting installed package.
FWIW, its normal for the testsuite to test the built local binaries and not the system binaries. This is especially important for distros where tests may be run when rgbds is not yet installed and when you want to be sure to test the newly built binaries and not the preexisting installed package.
Is that so? Then if the Debian maintainers are fine with it, I'd agree, it would avoid the need for all these changes. (And if you need to get the binaries that were installed to the system for some reason, cp $(command -v rgb{asm,link,fix,gfx}) can put them in the right directory.)
I just had a chat with someone on the #debian-mentors IRC channel and they confirmed that the autopkgtests test against the installed package by definition.
Whether we should use the rgbds tests for this is debatable, but I think they work nicely to confirm everything works as it should.
(Though we could restrict it to the testing of external projects I suppose, and run the internal tests against the locally built rgbds in a previous stage?)
Actually, that might be the easiest way to do it:
The tests already contain a --only-internal flag. Adding a --only-external flag would allow splitting the test suite in two, running the first half locally and the second half in autopkgtests.