crank compare throws an exception when new benchmark is added to the benchmarks project

[elizalcodes]

We use DotNetBenchmarks to run benchmarks on the main-branch (and save results in main.json file) and the feature-branch (and save results in branch.json file).
Then we use crank controller to compare the results with
$crank compare main.json feature.json
The output is correct as long as main and feature branches have exactly the same set of benchmarks.

We use this to keep track of the performance of our library as we keep developing it. That means new benchmarks for new methods will be added.

Unfortunately, when we add a new Benchmark on the branch, run benchmarks for main and branch, then run the crank compare, we get an exception:

PS ... $  crank compare main.json branch.json
Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key 'BenchmarkLibrary.Benchmarks.BenchmarkNew(Number: 4)' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.Crank.Controller.ResultComparer.DisplayDiff(IEnumerable`1 allBenchmarks, IEnumerable`1 allNames) in /_/src/Microsoft.Crank.Controller/ResultComparer.cs:line 209
   at Microsoft.Crank.Controller.ResultComparer.Compare(IEnumerable`1 filenames, JobResults jobResults, Benchmark[] benchmarks, String jobName) in /_/src/Microsoft.Crank.Controller/ResultComparer.cs:line 64
   at Microsoft.Crank.Controller.Program.<>c__DisplayClass52_1.<Main>b__3() in /_/src/Microsoft.Crank.Controller/Program.cs:line 221
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass144_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Microsoft.Crank.Controller.Program.Main(String[] args) in /_/src/Microsoft.Crank.Controller/Program.cs:line 678

Is there any configuration option which would allow the compare command to omit the benchmarks that don't have an equivalent on one of the branches?

here's a sample project with instructions for repro steps in the README.md file
https://github.com/elizalcodes/ClassLibraryCrank

[catcherwong]

crank compare use the first job's benchmarks as the reference, and there is no configuration option to omit it.

crank/src/Microsoft.Crank.Controller/ResultComparer.cs

Lines 209 to 220 in 20fd95e

	summaries[benchmark.FullName].Add(new BenchmarkSummary()
	{
	Name = benchmark.FullName,
	MeanNanoseconds = benchmark.Statistics.Mean,
	StandardErrorNanoseconds = benchmark.Statistics.StandardError,
	StandardDeviationNanoseconds = benchmark.Statistics.StandardDeviation,
	MedianNanoseconds = benchmark.Statistics.Median,
	Gen0 = benchmark.Memory?.Gen0Collections ?? 0,
	Gen1 = benchmark.Memory?.Gen1Collections ?? 0,
	Gen2 = benchmark.Memory?.Gen2Collections ?? 0,
	AllocatedBytes = benchmark.Memory?.BytesAllocatedPerOperation ?? 0
	});

I think modify summaries[benchmark.FullName] to summaries.TryGetValue(benchmark.FullName, out var summary) is more better here. It can omit the benchmarks that don't have an equivalent on one of the branches.

a test for the change

[elizalcodes]

hi, thank you for a super fast reply!
I cloned crank repo and debugged it locally (by adding a consoleApp and calling ResultComparer.Compare(filenames) directly).
I had to also add an if statement. then I got the correct output:

here is my code:

foreach (var benchmark in benchmarks)
                {
                    if (summaries.TryGetValue(benchmark.FullName, out var summary))
                    {
                        summary.Add(new BenchmarkSummary()
                        {
                            Name = benchmark.FullName,
                            ....
                        });
                    }
                }

Shall I make a contribution to your repo and a PR (happy to do it), or maybe you're expecting to add the fix with your team any time soon? Please let me know!