Allow RSpecTracer to ignore/exclude certain spec files

Is your feature request related to a problem? Please describe.

When RSpecTracer can determine the group examples for a single spec, it "gives up" and outputs the "DO NOT USE RSPEC TRACER" message. I still want to get value from having it on other files, so I want to be able to exclude them until they are fixed. I assumed that this configuration existed when using add_filter but RSpecTracer still runs to that file even when I include them in the filter.

Describe the solution you'd like

Have some sort of:

RSpecTracer.start do
    ignore ['spec/to/ignore_spec.rb']
end

Describe alternatives you've considered

I tried using add_filter as following:

RSpecTracer.start do
    add_filter 'spec/to/ignore_spec.rb'
end

But when I ran rspec spec/to/ignore_spec.rb it would still try to run for that file.

@theomelo The purpose of filters on rspec-tracer is to exclude certain files from the dependency analysis and coverage reporting. If you don't want to run certain specs, you can use exclude-pattern in RSpec.

@avmnu-sng SeamphoreCI will fail the whole job because rspec-tracer "could not uniquely identify" the examples of this single spec. Therefore, I still want Rspec to run it, and I'm ok with Codecov running analysis for this rspec, but not with rspec-tracer, if that makes sense?

I think the real issue is that this method that returns false:

rspec-tracer/lib/rspec_tracer.rb

Line 77 in 6da4539

::Kernel.exit(1) if runner.incorrect_analysis?

If passing a list of specs to exclude is the problem. I wonder if we could set an environment variable to send it to an external file reporting something like: "These files were resulted in an incorrect analysis, please fix them" kinda thing and exclude them from the final report? I think there's still a lot of value in having rspec-tracer running for the other files.

@theomelo It makes sense to report the duplicate description specs but not fail and most importantly not includ these specs in reporting. I will work on this feature and try to release it faster.

I am not sure whether I should be concerned about these duplicates getting fixed in the future or not. Because, once I start excluding such specs, there is no way to validate if these are fixed or not.

I see what you mean there. In my opinion (and you can completely ignore it), it makes more sense to leave it as a user responsibility, meaning that they can explicitly pass the specs they want to leave out based on your duplicate description report. For example, they would run once. It would fail and report X and Y files were incorrect due to duplicate descriptions. Then they pass these as a list of specs to exclude (the more accessible would be via ENV). Then, they fix them on their time, and once they are ready to include them, they do so by removing them from the explicit exclusion. What do you think? I can volunteer to help with developing it if you think this approach makes sense.

@theomelo It is better for the gem to implicitly take care of this case because the only option it has is ignoring these duplicate examples. I am still keeping the exit(1). It can be controlled by the env RSPEC_TRACER_FAIL_ON_DUPLICATES which defaults to true.

Thank you for doing this! 🙏

Let me know how it goes.