microsoft/ApplicationInsights-Kubernetes

Cannot get pod info because of proxy configuration

henols02-sr opened this issue · 9 comments

Describe the bug
My ASP.NET Core 7 application logs the following warnings:

[Warning] [2023-05-08T12:59:05.7419250Z] Query exception while trying to get pod info: The proxy tunnel request to proxy 'http://proxy:8080/' failed with status code '502'."
[Warning] [2023-05-08T12:59:36.9033456Z] Query exception while trying to get pod info: The proxy tunnel request to proxy 'http://proxy:8080/' failed with status code '502'."
[Warning] [2023-05-08T13:00:07.9161911Z] Query exception while trying to get pod info: The proxy tunnel request to proxy 'http://proxy:8080/' failed with status code '502'."
[Warning] [2023-05-08T13:00:34.4382518Z] Query exception while trying to get pod info: A task was canceled.
[Critical] [2023-05-08T13:00:34.4409067Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.K8sEnvironmentFactory.CreateAsync(CancellationToken cancellationToken)

The reason for this is that I have set the HTTP_PROXY and HTTPS_PROXY environment variables.
What URL (or rather host name) is this library using for getting pod info? I want to set the NO_PROXY environment variable because the proxy is only for connections to the Internet, and we have an on-premise installation of Kubernetes. Currently NO_PROXY is set to localhost,127.0.0.1,.local and a couple of more.

Package Versions
Application Insights Kubernetes Version: 6.1.0
Application Insights SDK Version: 2.21.0

Hey @henols02-sr, thank you for the reporting. Let me check the code and get back to you.

Here's the environment variable for the host name, could you please look it up in your environment?

KUBERNETES_SERVICE_HOST
KUBERNETES_SERVICE_PORT

And the default values are:
Default host: kubernetes.default.svc
Default port: 443

Here's the related source code:

https://github.com/kubernetes-client/csharp/blob/1342531db153b48d315921bdb1d1821174d36ebb/src/KubernetesClient/KubernetesClientConfiguration.InCluster.cs#LL51C1-L52C1

/app # env | grep KUBERNETES
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_SERVICE_HOST=10.43.0.1

So the address I'm looking for seems to be 10.43.0.1. I'll add that to NO_PROXY and see how it goes.

@henols02-sr, please try it and let us know.

Now I don't get the bit about proxy but it still doesn't work:

[Warning] [2023-05-11T12:05:49.4480166Z] Query exception while trying to get pod info: A task was canceled.
[Warning] [2023-05-11T12:06:09.0392950Z] Query exception while trying to get pod info: A task was canceled.
[Critical] [2023-05-11T12:06:09.0419462Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.K8sEnvironmentFactory.CreateAsync(CancellationToken cancellationToken)

I will talk to our Kubernetes team.

@henols02-sr, if you turn on trace logging by following this wiki, you shall be able to see the issue:
https://github.com/microsoft/ApplicationInsights-Kubernetes/blob/develop/docs/SelfDiagnostics.MD#with-204-beta1-or-above

Please give it a shot and let me know if there's anything unexpected.

I turned on tracing and here is the result:

[Trace] [2023-05-15T14:13:18.4686126Z] Registering bootstrap and hosted service.
[Information] [2023-05-15T14:13:18.5642140Z] Skip registering K8sInfoBackgroundService by user configuration.
[Trace] [2023-05-15T14:13:18.5647911Z] Registered bootstrap and hosted service.
[Debug] [2023-05-15T14:13:18.5649481Z] Application Insights Kubernetes injected the service successfully.
[Trace] [2023-05-15T14:13:20.5597187Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:37.5685781Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:39.6616908Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:39.6646638Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:39.9642832Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.0642293Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.0664903Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.0677772Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.1670951Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.3598338Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.3776381Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.3777975Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.4712580Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.2625747Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.5603021Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.6706579Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.6716769Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:42.3633168Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:42.5609480Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Debug] [2023-05-15T14:13:43.6646719Z] Starting update K8sEnvironment
[Debug] [2023-05-15T14:13:44.8601395Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T14:13:44.8604740Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result: 
[Debug] [2023-05-15T14:13:44.8605481Z] Getting pod name by provider: HostNamePodNameProvider
[Debug] [2023-05-15T14:13:44.9629228Z] Try getting pod name by environment variable HOSTNAME. Result: abc-backend-74d4fcc97c-7km4w
[Debug] [2023-05-15T14:13:44.9629830Z] Pod name by providers: abc-backend-74d4fcc97c-7km4w
[Trace] [2023-05-15T14:13:45.7629870Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.0608595Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.1595579Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.1602106Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.1604096Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:47.8641744Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:47.9634433Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:49.8661289Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:50.0629243Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.3613144Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.3624510Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.3632592Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.8597378Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.8603084Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.0611344Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.0622455Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.3604841Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.4607701Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0662887Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0667259Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0668279Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0668768Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669170Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669484Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669733Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669964Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0670244Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0670494Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:03.8100580Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:03.8100577Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:13.8081906Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:13.8081906Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:21.3627769Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:21.3662496Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:21.3682503Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:23.8080337Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:23.8081450Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:33.8077301Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:33.8078547Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:40.8655909Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:43.8075426Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:43.8076482Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:53.8073731Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:53.8073535Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:00.9995484Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:00.9996393Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:03.8076598Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:03.8076672Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:13.8082821Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:13.8082871Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:21.3587711Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:21.3591438Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:21.3593114Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:23.8071272Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:23.8071744Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:25.0646879Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:25.0658901Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Warning] [2023-05-15T14:15:25.0686069Z] Query exception while trying to get pod info: A task was canceled.
[Trace] [2023-05-15T14:15:25.0759046Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICoreV1Operations.ReadNamespacedPodWithHttpMessagesAsync(String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.CoreV1OperationsExtensions.ReadNamespacedPodAsync(ICoreV1Operations operations, String name, String namespaceParameter, Nullable`1 pretty, CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.GetMyPodAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
[Debug] [2023-05-15T14:15:25.5754421Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T14:15:25.5755023Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result: 
[Debug] [2023-05-15T14:15:25.5755072Z] Getting pod name by provider: HostNamePodNameProvider
[Debug] [2023-05-15T14:15:25.5755110Z] Try getting pod name by environment variable HOSTNAME. Result: abc-backend-74d4fcc97c-7km4w
[Debug] [2023-05-15T14:15:25.5755131Z] Pod name by providers: abc-backend-74d4fcc97c-7km4w
[Trace] [2023-05-15T14:15:33.8073760Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:33.8073946Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:40.8658556Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:43.8073109Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:43.8073109Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:44.6588611Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:44.6589131Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Warning] [2023-05-15T14:15:44.6596068Z] Query exception while trying to get pod info: A task was canceled.
[Trace] [2023-05-15T14:15:44.6605610Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICoreV1Operations.ReadNamespacedPodWithHttpMessagesAsync(String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.CoreV1OperationsExtensions.ReadNamespacedPodAsync(ICoreV1Operations operations, String name, String namespaceParameter, Nullable`1 pretty, CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.GetMyPodAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
[Critical] [2023-05-15T14:15:44.6618284Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.K8sEnvironmentFactory.CreateAsync(CancellationToken cancellationToken)
[Debug] [2023-05-15T14:15:44.6640928Z] Finished update K8sEnvironment, next iteration will happen at 05/15/2023 14:15:46 (UTC) by interval settings of 00:00:02
[Debug] [2023-05-15T14:15:46.6628421Z] Starting update K8sEnvironment
[Debug] [2023-05-15T14:15:46.6643397Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T14:15:46.6643686Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result: 
[Debug] [2023-05-15T14:15:46.6643716Z] Getting pod name by provider: HostNamePodNameProvider
[Debug] [2023-05-15T14:15:46.6643754Z] Try getting pod name by environment variable HOSTNAME. Result: abc-backend-74d4fcc97c-7km4w
[Debug] [2023-05-15T14:15:46.6643778Z] Pod name by providers: abc-backend-74d4fcc97c-7km4w
[Trace] [2023-05-15T14:15:53.8076062Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:53.8076068Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9996989Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9997396Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9997495Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9997613Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:03.8078271Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:03.8078272Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:13.8074145Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:13.8077361Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:21.3587918Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:21.3592904Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:21.3596254Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:23.8075679Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:23.8080065Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:33.8079620Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:33.8079621Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:40.8657275Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:43.8077016Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:43.8077015Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:53.8076268Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:53.8076265Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:01.0000398Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:01.0000963Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:03.8067795Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:03.8067793Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:13.8070421Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:13.8070420Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.

Hi @henols02-sr, the logs showed that the requests timed out when querying the cluster. There are no details revealing why. Can you search your application insights logs for exceptions and see if you could correlate it to requests and find out why it timeout? Maybe, do you have a firewall to prevent requests into the cluster (from within the container)?

On my side, I did a simple experiment and I think the configuration should work. That is why I think there might be another security mechanism like a firewall or so.

Here are the details:

  1. Setup proxy by:
ENV HTTP_PROXY=http://proxy:8080/
ENV HTTPS_PROXY=http://proxy:8080/

And I got the following expected error since I don't have a real proxy server

[Warning] [2023-05-15T18:35:15.6404868Z] Query exception while trying to get pod info: Resource temporarily unavailable (proxy:8080)  # I don't have a real proxy
[Debug] [2023-05-15T18:35:16.1785030Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T18:35:16.1786398Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result:
...```

And then, I exclude my cluster:

```shell
ENV NO_PROXY=10.96.0.1

And after that, it worked again for me.

Here's my cluster environment:

PS D:\Repos> kubectl exec -it $pod -- /bin/bash
root@ai-k8s-f5webapi-deployment-77dc74c9dc-j4qqf:/App# printenv
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
HOSTNAME=ai-k8s-f5webapi-deployment-77dc74c9dc-j4qqf
DOTNET_VERSION=6.0.16
ASPNETCORE_URLS=http://+:80
PWD=/App
HOME=/root
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
APPLICATIONINSIGHTS__CONNECTIONSTRING=InstrumentationKey=5d8258e7-abb2-4066-89a5-8c73071b74ff;IngestionEndpoint=https://westus2-0.in.applicationinsights.azure.com/;LiveEndpoint=https://westus2.livediagnostics.monitor.azure.com/
TERM=xterm
SHLVL=1
ASPNET_VERSION=6.0.16
HTTPS_PROXY=http://proxy:8080/
HTTP_PROXY=http://proxy:8080/
KUBERNETES_PORT_443_TCP_PROTO=tcp
DOTNET_RUNNING_IN_CONTAINER=true
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1 # I added this to the no_proxy list
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/printenv

Yes, there is probably some network policy blocking the request. I will work the Kubernetes team to get to bottom if it.