dotnet/BenchmarkDotNet

Custom Config breaks Benchmark detection.

swtrse opened this issue · 4 comments

swtrse commented

Hi, I noticed something strange.
As basis I copied the implementation of DefaultConfig
If I run my benchmakrs this is the starting log

// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 16 benchmark(s) in total *****
// ***** Building 2 exe(s) in Parallel: Start   *****
// ***** Done, took 00:00:06 (6.9 sec)   *****
// Found 8 benchmarks:
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary_AddItem: ShortRun-.NET 8.0(Runtime=.NET 8.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedImmutableDictionary_AddItem: ShortRun-.NET 8.0(Runtime=.NET 8.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedList_AddItem: ShortRun-.NET 8.0(Runtime=.NET 8.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary: ShortRun-.NET 8.0(Runtime=.NET 8.0, IterationCount=3, LaunchCount=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary_AddItem: ShortRun-.NET 9.0(Runtime=.NET 9.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedImmutableDictionary_AddItem: ShortRun-.NET 9.0(Runtime=.NET 9.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedList_AddItem: ShortRun-.NET 9.0(Runtime=.NET 9.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary: ShortRun-.NET 9.0(Runtime=.NET 9.0, IterationCount=3, LaunchCount=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]

however I do have some special needs, therefore I need to run the benchmarks in process. So I changed

public IEnumerable<Job> GetJobs()
{
	return [];
}

to (based on DebugInProcessConfig)

public IEnumerable<Job> GetJobs()
{
	return [Job.Default.WithToolchain(new InProcessEmitToolchain(TimeSpan.FromHours(24D), true))];
}

However now the logs look like

// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 24 benchmark(s) in total *****
// ***** Building 3 exe(s) in Parallel: Start   *****
// ***** Done, took 00:00:07 (7.49 sec)   *****
// Found 12 benchmarks:
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary_AddItem: ShortRun-.NET 8.0(Runtime=.NET 8.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary_AddItem: Job-IOTDSQ(Toolchain=InProcessEmitToolchain, InvocationCount=1, UnrollFactor=1) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedImmutableDictionary_AddItem: ShortRun-.NET 8.0(Runtime=.NET 8.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedImmutableDictionary_AddItem: Job-IOTDSQ(Toolchain=InProcessEmitToolchain, InvocationCount=1, UnrollFactor=1) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedList_AddItem: ShortRun-.NET 8.0(Runtime=.NET 8.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedList_AddItem: Job-IOTDSQ(Toolchain=InProcessEmitToolchain, InvocationCount=1, UnrollFactor=1) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary: ShortRun-.NET 8.0(Runtime=.NET 8.0, IterationCount=3, LaunchCount=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary: Job-YVSATQ(Toolchain=InProcessEmitToolchain) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary_AddItem: ShortRun-.NET 9.0(Runtime=.NET 9.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedImmutableDictionary_AddItem: ShortRun-.NET 9.0(Runtime=.NET 9.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedList_AddItem: ShortRun-.NET 9.0(Runtime=.NET 9.0, InvocationCount=1, IterationCount=3, LaunchCount=1, UnrollFactor=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]
//   AddItemBenchmarkFlatPrefill0<BigInteger>.SortedDictionary: ShortRun-.NET 9.0(Runtime=.NET 9.0, IterationCount=3, LaunchCount=1, WarmupCount=3) [Prefill=0, PriceRange=1000, DataType=BigInteger]

This does not look right. Even the benchmark count is different.
So what is the right way to run the benchmarks in process?

timcassell commented

Do you have job attributes elsewhere in your code? All configs get conglomerated, and that includes from job attributes.

swtrse commented

Ah yes. I get it. I have to remove the ShortJobAttributes and instead add the different platforms to the jobs.

thx

swtrse commented

@timcassell Ok me again. I changed the Method too

public IEnumerable<Job> GetJobs()
{
	Runtime[] runtimes = [CoreRuntime.Core80, CoreRuntime.Core90];
	foreach (var runtime in runtimes)
	{
		var sourceJob = Job.ShortRun;
		var baseJob = sourceJob.WithRuntime(runtime).WithId($"{sourceJob.Id}-{runtime.Name}").WithToolchain(new InProcessEmitToolchain(TimeSpan.FromHours(24D), true));
		yield return baseJob;
	}
}

that looks better now. But it seams this ignores my
[GenericTypeArguments(typeof(BigInteger))]
[GenericTypeArguments(typeof(UInt256))]
Attributes as well.
What I do have to change so they take effect?

swtrse commented

ok it looks like it is working.
The difference is that the other benchmarks are only shown when the first ones are finished.

  • 👍1