Threw an exception: when collecting telemetry data using the Dashboard Docker Image (Preview 5)
Closed this issue · 10 comments
I run
> docker run --rm -it -p 18888:18888 -p 4317:18889 --name aspire-dashboard -e DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS='true' mcr.microsoft.com/dotnet/nightly/aspire-dashboard:8.0.0-preview.5Run the application:
> $env:OTEL_EXPORTER_OTLP_ENDPOINT= 'http://localhost:4317'; $env:DOTNET_RESOURCE_SERVICE_ENDPOINT_URL = "service-a"; spin up
Then on the application (http://localhost:3000), I submit the request:
> curl http://localhost:3000
Handling request to Some(HeaderValue { inner: String("http://127.0.0.1:3000/") })
2024-04-13T05:23:26.711395Z ERROR spin_telemetry: There has been an error with the OpenTelemetry system, traces and metrics are likely failing to export
2024-04-13T05:23:26.711758Z ERROR spin_telemetry: Further OpenTelemetry errors will be logged at DEBUG level
When I check the logs of Aspire Dashboard container:
info: Aspire.Dashboard.DashboardWebApplication[0]
Aspire version: 8.0.0-preview.5.24201.12+1b627b7d5d399d4f9118366e7611e11e56de4554
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
Storing keys in a directory '/home/app/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
No XML encryptor configured. Key {6936c4f3-bcd8-4488-baca-573ec9eff46b} may be persisted to storage in unencrypted form.
info: Aspire.Dashboard.DashboardWebApplication[0]
Now listening on: http://0.0.0.0:18888
info: Aspire.Dashboard.DashboardWebApplication[0]
OTLP server running at: http://0.0.0.0:18889
fail: Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery[7]
An exception was thrown while deserializing the token.
Microsoft.AspNetCore.Antiforgery.AntiforgeryValidationException: The antiforgery token could not be decrypted.
---> System.Security.Cryptography.CryptographicException: The key {d4adf3f2-4f9c-431f-83ea-be2fe6010347} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken)
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken)
at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.GetCookieTokenDoesNotThrow(HttpContext httpContext)
info: Aspire.Dashboard.Model.BrowserTimeProvider[0]
Browser time zone set to 'Asia/Bangkok' with UTC offset 07:00:00.I checked to run with Jeager with the same setup and it works as below
> docker run -d -p16686:16686 -p4317:4317 -p4318:4318 -e COLLECTOR_OTLP_ENABLED=true jaegertracing/all-in-one:latestst
This exception happened with Aspire Dashboard Preview 5 on both Windows 11 and Ubuntu 22 OSes.
I think this related to the issue at #2248
/cc @davidfowl @JamesNK
This looks bogus:
$env:DOTNET_RESOURCE_SERVICE_ENDPOINT_URL = "service-a"; spin up
Did you implement a resource server? If not, you should remove it.
> curl http://localhost:3000
Handling request to Some(HeaderValue { inner: String("http://127.0.0.1:3000/") })
2024-04-13T05:23:26.711395Z ERROR spin_telemetry: There has been an error with the OpenTelemetry system, traces and metrics are likely failing to export
2024-04-13T05:23:26.711758Z ERROR spin_telemetry: Further OpenTelemetry errors will be logged at DEBUG level
Are you using grpc or http? The dashboard only supports grpc ingestion. Can you share what this application is doing?
Are you using grpc or http? The dashboard only supports grpc ingestion. Can you share what this application is doing?
The app uses
http. And source code I created at https://github.com/thangchung/spin-o11y
$env:DOTNET_RESOURCE_SERVICE_ENDPOINT_URL = "service-a"; spin up
Yeah. this look bogus
Ask the spin team if they support grpc endpoints for otlp. Their docs are a little thin.
Seems like yes. no.
In the future, set OTEL_EXPORTER_OTLP_PROTOCOL=grpc and it should work.
cc @JamesNK lets update our standalone dashboard docs to mention this.
Found it fermyon/spin@
cbb2cb9/crates/telemetry/src/traces.rs#L40C23-L40C50.Seems like
yes.no.In the future, set
OTEL_EXPORTER_OTLP_PROTOCOL=grpcand it should work.cc @JamesNK lets update our standalone dashboard docs to mention this.
Seems like Aspire Dashboard only supports grpc,
And the Spin App I used only support http: https://github.com/fermyon/spin/blob/cbb2cb91acd651c0098e7415c6503daaa3a02f5b/crates/telemetry/src/traces.rs#L42
Then looks like it will never work unless Spin supports grpc, right? @davidfowl
Correct!
Closing this as by design, I don't think we're adding otlp http support anytime soon. @JamesNK ?
Maybe we should just have a tracking issues so others can upvote.
Closing this as by design, I don't think we're adding otlp http support anytime soon. @JamesNK ?
I don't think it will be terribly difficult to at least partially support. OTLP HTTP can be either JSON or binary (Protobuf over HTTP). Protobuf over HTTP won't be too difficult. JSON will be more work because we'll need to create new DTOs and mappings for them.
Also, it will be a new port on Aspire. (OTLP grpc and OTLP http have different default ports)
https://opentelemetry.io/docs/specs/otlp/#otlphttp
It really depends on how many people want it.
@davidfowl It worked in the new spin version
spin --version
spin 2.5.0 (83eb68d 2024-05-08)After running the Spin App, then Aspire can handle the trace in grpc protocol now.
