No useful output in case of failing specs
Closed this issue · 5 comments
I ran my spec suite with rspectre (0.0.1 or 0.0.2, doesn’t matter) for 30 minutes, and all output I got was:
Running the specs failed. Either your tests do not pass normally or this is a bug in RSpectre.
Would it be possible to output
- the regular rspec output about which spec failed, and
- the rspectre output, which may be useful no matter if the specs passed or not.
@bquorning Yeah, this is something that clearly needs to be improved. I normally swallow the rspec output because in the happy path it's very noisy and would interfere with rspectre's output. The only reason it's slightly tricky is that I believe I just get a stream of rspec's output, not well-schematized objects that I can selectively output. As at least an improvement i can just dump the entire rspec output on failure (which I think I may have originally intended to do). I should definitely do something about 1.
2. Is interesting because the results may be tainted if there is some kind of bug / set of failing tests since that affects what gets executed and what gets counted (and may be indicative of an rspectre bug). Maybe it could be an option to force possibly-invalid output but I'm not sure that's the right move for the tool or not.
Hopefully I can look into this in the near future.
Thanks for the report!
@bquorning When you get a chance, can you give 0.0.3 a try?
Thanks @dgollahon, it works fine.
Running the specs failed. Either your tests do not pass normally or this is a bug in RSpectre.
RSpec Output:
---
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Turns out I have one spec failing with
SystemStackError:
stack level too deep
I will try to investigate and see if this is related to rspectre or not. Does rspectre support rspec’s --bisect flag?
Update: I’m trying with rspectre --rspec '--bisect' now.
Does rspectre support rspec’s --bisect flag?
I would suspect that it does, at least to some extent. I pass through whatever you give rspectre though you won't see output if things succeed.
If you find a reproduction case, let me know! I may have some recursive code or something that is contributing to blowing the stack or something like that--not sure. I should probably hunt for more medium to large OSS project corpuses I can make sure the tool doesn't crash on.