Some RGBFIX tests fail on BSD
Closed this issue · 32 comments
Hello,
I'm maintaining FreeBSD port and considering to switch to cmake build and also use ctest to include testing in port.
Testing fails with around 8 .gb mismathes.
Bellow is a partial log and a full log is available to download.
Any help and hints are welcome.
FreeBSD 15
cmake version 3.31.2
Clang 19.1.5
GNU bash 5.2.37
xxd 2024-12-07
git 2.47.1
<snip>
1: < 7f:7fe0: 4545 4545 4545 4545 4545 4545 4545 45 EEEEEEEEEEEEEEE
1: ---
1: 7f:7fe0
1: > 7f:7fe0: 4545 4545 4545 4545 4545 4545 4545 4545 EEEEEEEEEEEEEEEE
1: > 7f:7ff0: 4545 4545 4545 4545 4545 4545 4545 4545 EEEEEEEEEEEEEEEE
1: verify-pad.gb mismatch!
1: verify-trash...
1: /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/verify-trash.gb out.gb differ: char 335, line 3
1: 00:0140
1: < 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 809f e92c ...............,
1: ---
1: 00:0140
1: > 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 809f ea09 ................
1: verify-trash.gb mismatch!
1: verify...
1: /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/verify.gb out.gb differ: char 335, line 3
1: 00:0140
1: < 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 8060 1150 .............`.P
1: ---
1: 00:0140
1: > 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 8060 1073 .............`.s
1: verify.gb mismatch!
1: version...
1: version piped...
1: Checking padding...
1: 9...
1: 182...
1: 249...
1: 40...
1: 42...
1: 209...
1: 212...
1: 236...
1: 164...
1: 101...
1: Done checking padding!
1: 8 of the tests failed!
Full log:
LastTest.log
I see that all the asm and link tests pass, but eight fix tests fail (which prevents the gfx tests from running at all):
global-larger:/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/global-larger.gb out.gb differ: char 336, line 1 00:0140 < 00:0140: 0c29 b9c1 f8ae 9958 6790 a35d 05ac 1ca8 .).....Xg..].... --- 00:0140 > 00:0140: 0c29 b9c1 f8ae 9958 6790 a35d 05ac 1c8e .).....Xg..]....padding-bigperfect:cmp: EOF on /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/padding-bigperfect.gb 03:7ff0 < 03:7ff0: 38b9 6b28 5254 70d4 546a 5d70 42f0 7b 8.k(RTp.Tj]pB.{ --- 03:7ff0 > 03:7ff0: 38b9 6b28 5254 70d4 546a 5d70 42f0 7bff 8.k(RTp.Tj]pB.{.padding-imperfect:cmp: EOF on /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/padding-imperfect.gb 03:7ff0 < 03:7ff0: ffff ffff ffff ffff ffff ffff ffff ff ............... --- 03:7ff0 > 03:7ff0: ffff ffff ffff ffff ffff ffff ffff ffff ................padding-perfect:cmp: EOF on /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/padding-perfect.gb 01:7ff0 < 01:7ff0: faf0 5b29 7e18 59d0 6bfe 8d2f 3c73 19 ..[)~.Y.k../<s. --- 01:7ff0 > 01:7ff0: faf0 5b29 7e18 59d0 6bfe 8d2f 3c73 19ff ..[)~.Y.k../<s..title-trunc:/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/title-trunc.gb out.gb differ: char 308, line 3 00:0130 < 00:0130: d03a de30 3132 3334 3536 3738 3941 4243 .:.0123456789ABC < 00:0140: 4445 46e9 9c0a ef86 96a3 742a 5e36 6ecd DEF.......t*^6n. --- 00:0130 > 00:0130: d03a deef 3031 3233 3435 3637 3839 4142 .:..0123456789AB > 00:0140: 4344 4546 9c0a ef86 96a3 742a 5e36 6ecd CDEF......t*^6n.verify-pad:/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/verify-pad.gb out.gb differ: char 335, line 3 00:0140 < 00:0140: 9d9d 8cb9 b62e c6da 06d3 96ff 8041 f522 .............A." --- 00:0140 > 00:0140: 9d9d 8cb9 b62e c6da 06d3 96ff 8041 f8da .............A.. 7f:7fe0 < 7f:7fe0: 4545 4545 4545 4545 4545 4545 4545 45 EEEEEEEEEEEEEEE --- 7f:7fe0 > 7f:7fe0: 4545 4545 4545 4545 4545 4545 4545 4545 EEEEEEEEEEEEEEEE > 7f:7ff0: 4545 4545 4545 4545 4545 4545 4545 4545 EEEEEEEEEEEEEEEEverify-trash:/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/verify-trash.gb out.gb differ: char 335, line 3 00:0140 < 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 809f e92c ..............., --- 00:0140 > 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 809f ea09 ................verify:/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/verify.gb out.gb differ: char 335, line 3 00:0140 < 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 8060 1150 .............`.P --- 00:0140 > 00:0140: 9d9d 8cb9 b62e c6da e7d3 96ff 8060 1073 .............`.s
I don't have a FreeBSD environment or expertise available to debug this directly. Maybe you can try narrowing down the problem(s)? E.g. git bisect and just run test/fix/test.sh.
I'm new to this tests and running test/fix/test.sh gives same errors with additional console warns:
cmp: EOF on /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/padding-bigperfect.gb
cmp: EOF on /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/padding-imperfect.gb
cmp: EOF on /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/fix/padding-perfect.gb
Should we investigate if there are some incompatibiliies between BSD and Gnu in programs used by the bash scripts?
Yes, I had quoted those from your log.
If the tests were running and passing with RGBDS 0.8.0 but not with 0.9.0, please try narrowing down the commits where they start to fail, e.g. with git bisect.
It's possible the cause is related to a change in the test shell scripts themselves, yes.
If the tests were running and passing with RGBDS 0.8.0 but not with 0.9.0, please try narrowing down the
commits where they start to fail, e.g. with git bisect
I never run tests on 0.8.0 before. I will try on that version and check.
Unable to test with ctest as 0.8.0 isn't ready for it.
Just a question so I can proceed:
Is cmake build mature to use in production enviroments?
If positive, I can upgrade to 0.9.0 with cmake build and put tests as unsafe.
Providing test units with ports gives us extra testing but it doesn't affect end users.
Oh I see, the tests are running now only because 0.9.0 added ctest support?
In that case, yes, cmake build support has been good for years, so if you want to disable/unsafe the tests and release 0.9.0 that should be fine.
Maybe one of your end users could investigate why the tests are failing, as well?
Sure, there is why its ok to have tests in ports even if tests fails, it always can lead to further investigation.
The only consumer that I can test is sameboy and build/run-test is OK so I've committed update:
https://cgit.freebsd.org/ports/commit/?id=7b5bb5efdad1ca269649861a3b8e38a778c67e54
If upstream want to do some testing on FreeBSD, it can be done on a VM and just go to ports and test:
https://www.freebsd.org/where/
cd /usr/ports/devel/rgbds && make test
You can run the tests like this too even without cmake, might make it easier to test old versions to see if there is a good version.
for dir in asm link fix gfx; do
(
cd "test/${dir}" >/dev/null
echo "Running ${dir} tests."
./test.sh || break
)
done
Hello @orbea ,
I did a quick revert port to 0.8.0 and tests fail same way:
Including full log:
0.8.0_test.log
@nunotexbsd @orbea I've opened #1586, which might fix these failing tests. Please try building https://github.com/Rangi42/rgbds/tree/bsd-bug and run the tests.
@Rangi42 Unfortunately I only have linux machines, hopefully @nunotexbsd or another BSD user can try. Would also be interesting to see how many of the tests pass on OpenBSD.
I thought you were the maintainer for https://www.freshports.org/devel/rgbds/.
yes I am. I thought that there will be interest in doing tests in openbsd and maybe netbsd too
hopefully @nunotexbsd or another BSD user can try.
Would also be interesting to see how many of the tests pass on OpenBSD.
Same 8 mismatches errors on .gb with b6c520c applied to 0.9.0
Aww. Thank you for testing that out though!
Can you try it on a FreeBSD vm?
I can try, but I was really hoping that the existence of this BSD port meant there was at least one actual BSD RGBDS user out there who could help debug/test their own issue. (After all, it's not an OS chosen by people who can't deal with make and gdb and fixing their own build issues and so on.)
I've just sent mail to NetBSD and OpenBSD rgbds port maintainers. Lets see is something comes up.
@nunotexbsd You might be able to find the issue with one of the various sanitizers, I would try with both gcc and clang.
Which sanitizers do we not already run to build in CI?
It's possible the errors are simply not triggered outside of BSD, so that's worth testing for.
@nunotexbsd You might be able to find the issue with one of the various sanitizers, I would try with both gcc and clang.
Just tested with gcc 13 and still fails.
Maybe is worth to take a look on logs to check if something is different:
tests log:
test.log
build log:
rgbds-0.9.0.log
EDIT: Think I found the culprit, port was using a dos2unix converter and I disabled that. It seems that that conversion used in older ports was doing something wrong on tests:
tests OK with clang and gcc.
Just some errors on test/pokecrystal/Makefile maybe due to diffs between BSD and GNU make. I will work on it now.
<snip>
1: Testing: ../../rgbgfx @none_round_trip.flags -o none_round_trip.2bpp -r 1 result.png && ../../rgbgfx @none_round_trip.flags -o result.2bpp result.png
1: Testing: ../../rgbgfx @reverse_1bit.flags -d 1 -o reverse_1bit.1bpp -r 1 result.png && ../../rgbgfx @reverse_1bit.flags -d 1 -o result.2bpp result.png
1: Testing: ../../rgbgfx @reverse_curve.flags -o reverse_curve.2bpp -r 1 result.png && ../../rgbgfx @reverse_curve.flags -o result.2bpp result.png
1: All 174 tests passed!
1: ~/rgbds/test
1: ~/rgbds/test/pokecrystal ~/rgbds/test
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 38: Invalid line 'ifeq (,$(shell which sha1sum))', expanded to 'ifeq '
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 40: Invalid line 'else'
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 42: Invalid line 'endif'
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 109: Invalid line 'ifeq ($(DEBUG),1)', expanded to 'ifeq '
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 111: Invalid line 'endif'
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 128: Invalid line 'ifeq (,$(filter clean tidy tools,$(MAKECMDGOALS)))', expanded to 'ifeq '
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 130: Invalid line '$(info $(shell $(MAKE) -C tools))', expanded to ''
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 136: Invalid line 'define DEP'
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 139: Invalid line 'endef'
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 142: Invalid line '$(foreach obj, $(pokecrystal_obj), $(eval $(call DEP,$(obj),$(obj:.o=.asm))))', expanded to ''
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 143: Invalid line '$(foreach obj, $(pokecrystal11_obj), $(eval $(call DEP,$(obj),$(obj:11.o=.asm))))', expanded to ''
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 144: Invalid line '$(foreach obj, $(pokecrystal_au_obj), $(eval $(call DEP,$(obj),$(obj:_au.o=.asm))))', expanded to ''
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 145: Invalid line '$(foreach obj, $(pokecrystal_debug_obj), $(eval $(call DEP,$(obj),$(obj:_debug.o=.asm))))', expanded to ''
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 146: Invalid line '$(foreach obj, $(pokecrystal11_debug_obj), $(eval $(call DEP,$(obj),$(obj:11_debug.o=.asm))))', expanded to ''
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 147: Invalid line '$(foreach obj, $(pokecrystal11_vc_obj), $(eval $(call DEP,$(obj),$(obj:11_vc.o=.asm))))', expanded to ''
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 153: Invalid line 'endif'
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 230: while evaluating variable "eval " with value "": Unknown modifier " $(png"
1: make: "/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal/Makefile" line 230: Invalid line '$(foreach png, $(unown_pngs), $(eval $(png:.png=.2bpp): $(png) gfx/pokemon/unown/normal.gbcpal))', expanded to ')'
1: make: Fatal errors encountered -- cannot continue
1: make: stopped making "clean" in /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/pokecrystal
1/1 Test #1: all ..............................***Failed 30.66 sec
0% tests passed, 1 tests failed out of 1
(...)
It seems tests is trying to build pokecrystal.
I don't have nonfree set.
if "$nonfree"; then
test_downstream pret pokecrystal compare pokecrystal.gbc f4cd194bdee0d04ca4eac29e09b8e4e9d818c133
test_downstream pret pokered compare pokered.gbc ea9bcae617fdf159b045185467ae58b2e4a48b9a
test_downstream zladx LADX-Disassembly default azle.gbc d90ac17e9bf17b6c61624ad9f05447bdb5efc01a
fi
And comment those lines above or ./run-test.sh --only-free or cmake USE_NONFREE_TESTS OFF
<snip>
All 174 tests passed!
/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test
/wrkdirs/usr/ports/devel/rgbds/work/rgbds/test/ucity /wrkdirs/usr/ports/devel/rgbds/work/rgbds/test
rm ucity.gbc ucity_compat.gbc ucity.sym ucity.map
--- ucity.gbc ---
rgblink ucity.gbc
FATAL: Please specify an input file (pass `-` to read from standard input)
Usage: rgblink [-dMtVvwx] [-l script] [-m map_file] [-n sym_file]
[-O overlay_file] [-o out_file] [-p pad_value]
[-S spec] <file> ...
Useful options:
-l, --linkerscript <path> set the input linker script
-m, --map <path> set the output map file
-n, --sym <path> set the output symbol list file
-o, --output <path> set the output file
-p, --pad <value> set the value to pad between sections with
-x, --nopad disable padding of output binary
-V, --version print RGBLINK version and exits
In a rude way I have tests OK without any complaint commenting last tests.
1: All 174 tests passed!
1: ~/rgbds/test
1/1 Test #1: all .............................. Passed 30.87 sec
100% tests passed, 0 tests failed out of 1
--- work/rgbds/test/run-tests.sh 2024-12-30 12:15:49.953051000 +0000
+++ run-tests.sh 2024-12-30 12:15:30.440317000 +0000
@@ -69,11 +69,11 @@
popd
}
-if "$nonfree"; then
- test_downstream pret pokecrystal compare pokecrystal.gbc f4cd194bdee0d04ca4eac29e09b8e4e9d818c133
- test_downstream pret pokered compare pokered.gbc ea9bcae617fdf159b045185467ae58b2e4a48b9a
- test_downstream zladx LADX-Disassembly default azle.gbc d90ac17e9bf17b6c61624ad9f05447bdb5efc01a
-fi
-test_downstream AntonioND ucity all ucity.gbc 67e67a4eaefb12ddc98d3caa8959102b885c0405
-test_downstream pinobatch libbet all libbet.gb 7001af0f0664ba4e941310cde2c4c8392423abd2
-test_downstream LIJI32 SameBoy bootroms build/bin/BootROMs/cgb_boot.bin 113903775a9d34b798c2f8076672da6626815a91
+#if "$nonfree"; then
+# test_downstream pret pokecrystal compare pokecrystal.gbc f4cd194bdee0d04ca4eac29e09b8e4e9d818c133
+# test_downstream pret pokered compare pokered.gbc ea9bcae617fdf159b045185467ae58b2e4a48b9a
+# test_downstream zladx LADX-Disassembly default azle.gbc d90ac17e9bf17b6c61624ad9f05447bdb5efc01a
+#fi
+#test_downstream AntonioND ucity all ucity.gbc 67e67a4eaefb12ddc98d3caa8959102b885c0405
+#test_downstream pinobatch libbet all libbet.gb 7001af0f0664ba4e941310cde2c4c8392423abd2
+#test_downstream LIJI32 SameBoy bootroms build/bin/BootROMs/cgb_boot.bin 113903775a9d34b798c2f8076672da6626815a91Which sanitizers do we not already run to build in CI?
Excuse me if I missing something, but I don't see -fsanitize=leak, -fsanitize=thread or -fsanitize=undefined.
EDIT: Think I found the culprit, port was using a dos2unix converter and I disabled that.
All 174 tests passed!
Alright! I'm glad you found the root cause of those rgbfix failures!
Just some errors on test/pokecrystal/Makefile maybe due to diffs between BSD and GNU make.
Yes, rgbds stays compatible with BSD make, but pret projects take advantage of GNU features. I believe this is common enough that BSD users are accustomed to having gmake installed? Anyway, you're probably fine just skipping those tests.
@nunotexbsd Please confirm once you have the tests passing/skipping that this issue can be closed. And thank you for your patience in resolving it!
Excuse me if I missing something, but I don't see
-fsanitize=leak,-fsanitize=threador-fsanitize=undefined.
Thank you, good catch! I wonder if we had left those out because an older gcc didn't support them, or because C-era rgbds unavoidably triggered them, or (for thread) because it seemed unnecessary, or they were just an oversight. Anyway, I'll do a separate PR for those.
Edit: @orbea ...or, this is why :P
cc1plus: error: ‘-fsanitize=thread’ is incompatible with ‘-fsanitize=address’
cc1plus: error: ‘-fsanitize=leak’ is incompatible with ‘-fsanitize=thread’
Yea, some of these would have to be run separately, but clang's ubsan did help with this issue so it might be worthwhile.
Tests OK for me using USE_NONFREE_TESTS OFF and skiping last 3 tests.
It is ok to close PR as fixed.
Thanks very much and sorry for so much noise since the problem was an port issue.