Http filter / enrich delegates are not invoked when service makes Grpc requests

[BlakeWills]

Bug Report

OpenTelemetry 1.2.0-beta2.1
OpenTelemetry.Instrumentation.Http 1.2.0-beta2.1

Tested on .NET5.0, looks like it will affect all runtimes.

Symptom

If a service with http instrumentation makes a Grpc request before a plain old Http request, the Filter and Enrich delegates within the HttpClientInstrumentation will not get invoked for any subsequent http requests.

This is particularly painful when dealing with Azure Isolated functions, which always makes a Grpc request on start-up, which due to this issue breaks all Http instrumentation. (You still get spans exported, you just can't filter or enrich them).

What is the expected behavior?

Filter & Enrich methods should be invoked for all Http requests.

What is the actual behavior?

Filter & Enrich methods are not exported for any Http requests.

This is because:

The first time startRequestFetcher.TryFetch is invoked, innerFetcher is null, and so it calls PropertyFetch.FetcherForProperty which creates a TypedPropertyFetch

When the first request the application makes is a Grpc request, obj is of type Grpc.Shared.TelemetryHeaderHandler+ActivityStartData. This means the innerFetcher now always looks for Grpc ActivityStartData instances, and when any subsequent http request is invoked, TypedPropertyFetch.TryFetch fails because obj is now of type System.Net.Http.DiagnosticsHandler+ActivityStartData, so OnStartActivity exits prematurely.

Reproduce

Sample Repo - OtelTestHarness is an Azure isolated function which executes a Http request via a timer trigger every 15 seconds. The Filter and Enrich methods within Program.cs will not be executed for any of the http requests, as the isolated function SDK makes a Grpc request during start-up.

[cijothomas]

Could you clarify if the issue is only with the Azure Functions case?
In other words, does this (https://github.com/BlakeWills/OtelTestHarness/tree/master/OtelTestHarnessApp) as expected?

[BlakeWills]

@cijothomas Yeah that's correct, I built two apps to compare one that worked to one that didn't whilst debugging.

[cijothomas]

#1803 Functions has some known issues with DiagnosticListener callbacks. This issue has links to corresponding issues in Azure functions repos as well.

[BlakeWills]

@cijothomas Sorry, I misread. This breaks regardless of whether or not you're using an Azure function. I've updated the sample console app (https://github.com/BlakeWills/OtelTestHarness/tree/master/OtelTestHarnessApp) to make a grpc request before any http requests, this breaks the http instrumentation.

If you comment out line #41 within program.cs which makes the grpc request, the instumentation works as expected (I.E. The filter and enrich delegates are invoked).

[BlakeWills]

FYI a workaround is to set SuppressDownstreamInstrumentation:

using (Sdk.CreateTracerProviderBuilder()
	.AddSource("test-source")
	.AddGrpcClientInstrumentation(o =>
	{
		o.SuppressDownstreamInstrumentation = false;
	})

[CodeBlanch]

Thanks for the sample repo @BlakeWills! I opened a PR that I think will take care of the problem without users needing to add in AddGrpcClientInstrumentation to their tracer config.