GoogleCloudPlatform/opentelemetry-operations-js

Cloud monitoring export reports "invalid location or region" in GAE nodejs20

Closed this issue · 2 comments

fpoppinga commented

Please answer these questions before submitting a bug report.

What version of OpenTelemetry are you using?

Excerpt from my package.json:

    "@google-cloud/opentelemetry-cloud-monitoring-exporter": "^0.17.0",
    "@google-cloud/opentelemetry-cloud-trace-exporter": "^2.1.0",
    "@google-cloud/opentelemetry-resource-util": "^2.1.0",
    "@opentelemetry/api": "^1.6.0",
    "@opentelemetry/core": "^1.17.1",
    "@opentelemetry/instrumentation": "^0.44.0",
    "@opentelemetry/instrumentation-express": "^0.33.2",
    "@opentelemetry/instrumentation-http": "^0.44.0",
    "@opentelemetry/instrumentation-pg": "^0.36.2",
    "@opentelemetry/resources": "^1.17.1",
    "@opentelemetry/sdk-metrics": "^1.17.1",
    "@opentelemetry/sdk-trace-base": "^1.17.1",
    "@opentelemetry/sdk-trace-node": "^1.17.1",
    "@opentelemetry/semantic-conventions": "^1.17.1",

What version of Node are you using?

The app is running on the GAE app engine standard environment nodejs20.

What did you do?

  • Setup the cloud monitoring according to the documentation in the README.md
  • observe, no metrics are reported when running in GAE standard environment
  • enable diag logging
  • observe the error message below

What did you expect to see?

Metrics being reported to cloud monitoring.

What did you see instead?

PeriodicExportingMetricReader: metrics export failed (error Error: Send TimeSeries failed: One or more TimeSeries could not be written: Unrecognized region or location.: gae_instance{instance_id:0037d<redacted>da5b0,version_id:20231024t093347,module_id:default,location:projects/1<redacted>5/zones/eu2} timeSeries[0]: workload.googleapis.com/order.erp_order_change_upsert{orgId:hmd})`

Additional context

  • Reporting metrics worked fine with a prehistoric version of opentelemetry before, but in an effort to upgrade all dependencies, metric reporting stopped working.
  • I assume that this might have to do with the resource detection. We're using the import { GcpDetectorSync } from "@google-cloud/opentelemetry-resource-util";.
fpoppinga commented

After speculating about the root cause lies in the way the information from GcpDetectorSync is consumed, I disabled the resource detection by commenting out the .merge line in the example given in the docs.

Now I receive metrics, however, obviously the resource information is not complete. What would be your recommendation on how to proceed?

aabmass commented

Sorry for not following taking a look at this sooner.

We have some integration tests that try the resource detector on GAE flex environment. It seems like you're running into a bug for GAE standard environment, which we should be supporting.

After speculating about the root cause lies in the way the information from GcpDetectorSync is consumed, I disabled the resource detection by commenting out the .merge line in the example given in the docs.

As a workaround, you can definitely populate the OTel resource attributes that you need manually (i.e. without the resource detector). Something like this should work:

const meterProvider = new MeterProvider({
  resource: new Resource({
    "service.name": "example-metric-service",
    "service.namespace": "samples",
    "service.instance.id": "12345",
    "cloud.region": "europe-west",
    "faas.name": "<gae service name>",
    "faas.version": "<gae service version>",
    "faas.instance": "<gae instance>",
  }),
});
  • 👍1