AnalyzerTest`1.cs AnalyzerTest<T>.IsSubjectToExclusion() returns incorrect result

[Serenyrddraig]

I've created a C# Analyzer/CodeFix project using Visual Studio (17.4.3). When I run the associated tests, I get an error:
Assert.AreEqual failed. Expected:<0>. Actual:<4>. Context: Diagnostics of test state
Context: Verifying exclusions in code
Mismatch between number of diagnostics returned, expected "0" actual "4"

However, the test supplies the correct expected of DiagnosticResult, all properly configured, as shown:

        [TestMethod]
        public async Task TestMethod1()
        {
            var test = @"
                using System;
                using System.Collections.Generic;
                using System.Linq;
                using System.Text;
                using System.Threading.Tasks;
                using System.Diagnostics;

                namespace ConsoleApplication1
                {
                    class TYPENAME
                    {   
                    }
                }";
            DiagnosticResult[] expectedResults = new DiagnosticResult[4]
            {
                VerifyCS.Diagnostic(UsingsAnalyzer.DiagnosticId).WithSpan(3, 17, 3, 50).WithArguments("/0/Test0.cs"),
                VerifyCS.Diagnostic(UsingsAnalyzer.DiagnosticId).WithSpan(4, 17, 4, 35).WithArguments("/0/Test0.cs"),
                VerifyCS.Diagnostic(UsingsAnalyzer.DiagnosticId).WithSpan(5, 17, 5, 35).WithArguments("/0/Test0.cs"),
                VerifyCS.Diagnostic(UsingsAnalyzer.DiagnosticId).WithSpan(7, 17, 7, 42).WithArguments("/0/Test0.cs")
            };
            await VerifyCS.VerifyAnalyzerAsync(test, expectedResults);

        }

My analyzer is identfying Using Directives that are not in a specified sort order, in this case 4 of them as shown by the expectedResults.

After a bunch of frustrating hours I traced what I think is the problem:

AnalyzerTest.VerifyGeneratedCodeDiagnosticsAsync() filters the expected results (line 294) using IsSubjectToExclusion(). However, the last check in that method (line 832) appears to be incorrect.

            if (!analyzers.Any(analyzer => analyzer.SupportedDiagnostics.Any(supported => supported.Id == result.Id)))
            {
                // This diagnostic is **not** reported by an active analyzer
                return false;
            }

should be

            if (analyzers.Any(analyzer => analyzer.SupportedDiagnostics.Any(supported => supported.Id == result.Id)))
            {
                // This diagnostic is reported by an active analyzer
                return false;
            }

By using the debugger to return false from this routine, my tests complete successfully. Otherwise, the expected results array is filtered to an empty array by the code above, resulting in my error.

Note the default tests provide by the Visual Studio project template do not suffer from this issue because they do not return multiple diagnostics from the analyzer.

I can supply the sample project it needed.

[sharwell]

Note that this code shouldn't be running at all in properly-configured analyzers due to #569. It only runs in cases where one or more analyzers didn't call ConfigureGeneratedCodeAnalysis. The check in IsSubjectToExclusion appears to be correct, for the following reason:

1. The analyzer did not call ConfigureGeneratedCodeAnalysis. It is therefore the analyzer's responsibility to detect the presence of generated code (in this case, files with // <auto-generated/> at the top) and avoid reporting diagnostics in those files.
2. To verify the previous point, the test constructed a file with // <auto-generated/> at the top of the file, and attempted to verify that all diagnostics reported by analyzers no longer appear.
3. During the validation, 4 unexpected diagnostics were reported, triggering a test failure.

This default test behavior was created to help analyzer authors adhere to common behaviors users expect, since it's extremely easy to miss this detail.