How to use `mdx test` in makefiles ?
Closed this issue · 9 comments
I just found mdx and it does precisely what I want (cram tests for shell commands). But I don't quite see how to use it in a Makefile ? If I run mdx test foo.md, and the test is incorrect, it will generate foo.md.corrected, but still exits with code 0, which make interprets as "success".
Is there something I'm missing as a way of checking for errors ? I guess I can check for the existence of the .corrected file, but this seems like a deviation from the standard behaviour of test commands ?
Normally mdx is used through dune which basically runs mdx and then checks against the diff (and thus also allows seemless promotion of the corrected files).
I don't know if there is a reason why a test mismatch still returns 0. It could be due to compatibility with some tools that expect mdx to return 0 unless the error is indeed an error with the input (as mdx currently does). Maybe an option can be added to return 0 on test failures and maybe we can explore returning non-zero on failed tests.
If this something you'd be open to accepting, I can explore how to make the change. I figured I should ask first, since it seemed possible it was desired behaviour.
The best way would be to use -o to ensure that Mdx will generate something and to diff the file explicitly:
foo.md.corrected: foo.md
ocaml-mdx test -o $@ $^
testmdx: foo.md.corrected foo.md
diff foo.md.corrected foo.md@Julow It's kind of wasteful to create all these files and run diff if ocaml-mdx already knows whether the test is successful or failed so I can see @chetmurthy's point about returning with a non-zero error code on test failures.
I think we can definitely accept a patch to make such behavior optional for now and consider making it default in some future MDX version (unless that will create issues with other tools).
Actually, I'm convinced by @Julow 's explanation. It's a perfectly cromulent idiom, and I intend to adopt it in my Makefiles. Thank you! I mean, we're talking about Make here: it create enormous numbers of extra files! grin What was needed is just a straightforward rule that could be implemented cleanly, and now I have that.
Thank you!
I have anything against a non-zero error code (thought it's a breaking change), that could be useful in many cases. But that wouldn't help implementing promotion, a failing command is hard to recover from in a makefile. The error code should always be 0 when -o is passed.
I was thinking that --error-code-on-test-failure/--nonzero-on-failure (or however you want to call the option) could be a decent option to add to avoid having to write one (.PHONY) rule per file tested just to get Make to fail when the tests fail. That way there is no breaking change.
Although the main consumer of MDX, dune, doesn't even use ocaml-mdx on the mdx 0.2+ stanzas, so for Dune the CLI is pretty much irrelevant.
I'm still waking up, so haven't coded this up, but I think a pattern rule should suffice, so it wouldn't be a lot of code. I'll report back when I get it working, with a snippet.
Here's a snippet that does the trick for my Makefile. I just list my asciidoc files as prereqs of mdx-test and they get the desired treatment. I haven't written promote, mostly b/c I prefer to do that manually for now.
Thank you for writing MDX! It's great stuff!
mdx-test:: README.asciidoc.TEST
.PRECIOUS: %.asciidoc.corrected
%.asciidoc.corrected: %.asciidoc
ocaml-mdx test -o $@ $^
%.asciidoc.TEST: %.asciidoc.corrected %.asciidoc
diff -Bwiu $^