:file vs :test-file issue
slyrus opened this issue · 11 comments
If I declare my file to be a :file in the .asd file, everything seems fines (although the reporting says All 0 files passed, which was a clue that maybe something wasn't right). If I declare the file as a :test-file, it doesn't get tested when I try to test the system via (asdf:test-system 'cl-bio-align). If, OTOH, I try to run the tests a bit more manually by doing: (prove-asdf:run (asdf:find-system "cl-bio-align-test")) then things seem to work. Any ideas?
Apparently things work fine (as a :test-file) as long as the hierarchy is:
...
:components
((:module :test :components ((:test-file "cl-bio-align-test"))))
...
But if I have another level of nesting, e.g.:
((:module :test :components ((:module :align :components ((:test-file "cl-bio-align-test"))))))
then things break. It seems as though the :test-file is being ignored if it's not at the top-level of the module.
One more thought... After running things a bunch of times, I get a very polluted prove-asdf::system-test-files hashtable. Might this be part of the problem?
Not sure I follow. I have one of those: https://github.com/slyrus/cl-bio/blob/master/cl-bio-align-test.asd#L18 but the problem persists.
Try (asdf:test-system :cl-bio-align-test).
I'm having exactly the same problem described at the start of this issue. "All 0 files passed".
Slyrus, by chance are you running CCL? I've tried this a few ways, but have not got it work. The file with the test forms just doesn't seem to run.
No, I'm using SBCL.
Slyrus, did you manage to fix the issue? Here is a summary of what I'm seeing. First, the .asd:
(defsystem "lisp-stat-tests"
:description "Unit tests for CLS"
:depends-on ("lisp-stat" "prove")
:defsystem-depends-on (:prove-asdf)
:pathname "tests/"
:components ((:test-file "variables-test"))
:perform (test-op :after (op c)
; (symbol-call :lisp-stat-tests :run-test-system c)
; (funcall (intern #.(string :run-test-system) :prove-asdf) c)
(funcall (intern #.(string :run) :prove) c)
))
As you can see, I've been experimenting with various test-ops; all produce the same result:
CL-USER> (asdf:test-system :lisp-stat-tests)
To load "lisp-stat-tests":
Load 1 ASDF system:
lisp-stat-tests
; Loading "lisp-stat-tests"
Summary:
All 0 files passed.
It is clear that the results of variables-test.lisp is not being executed, because if I put trivial failure tests in there, I get the same result. If I rename it, asdf complains, so the file location seems correct. However if I call prove on the test file directly, it works:
CL-USER> (prove:run #P"s:/src.sct/lisp-stat/tests/variables-test.lisp" :reporter :fiveam)
...
Did 3 checks.
Pass: 3 (100%)
Fail: 0 ( 0%)
T
(#P"s:/src.sct/lisp-stat/tests/variables-test.lisp")
Any way to trace what's going on in ASDF or Prove to find out where & why this isn't being executed?
A bit more data. Taking a cue from Slyrus's initial description of the problem, I changed from keyword :test-file to :file in -tests.asd. With this (asdf:test-system :main-system) worked, once, but not in a subsequent run. Here is the sequence:
(asdf:test-system :main-system) -> Worked once after changing from :test-file to :file. No idea why.
(asdf:test-system :main-system-tests) evaluated right after the above, failed
(asdf:test-system :main-system), run right after the above, failed, despite working less than 30 seconds earlier.
It does look like something is being corrupted somewhere.
Has anyone got this working reliably?
A little more data: making a change in the -test.lisp file makes it work, once. Subsequent runs fail to execute the tests until another change is made in the file.
Even in the case where the tests are run, the system still reports:
CL-USER> (asdf:test-system :lisp-stat)
1..3
✓ 4 is not in the list (1 2 3)
✓ 4 is expected to be 4
✓ 1 is not expected to be #\1
✓ 3 tests completed (0ms)
To load "lisp-stat-tests":
Load 1 ASDF system:
lisp-stat-tests
; Loading "lisp-stat-tests"
Summary:
All 0 files passed.
T
CL-USER>
That "All 0 files passed." can't be right.
A final comment: this bug appears to be ASDF not loading the test file, probably because of the belief that it is already loaded. If any change is made to the test file, the test file is executed, once. Subsequent runs fail to run/load the test file.
This code has a learning curve I cannot afford to climb right now; hopefully these comments will point someone who can fix it in the right direction.