Java Agent ignoring environment variables and java parameters for service name

Question

Java Agent ignoring environment variables and java parameters for service name

decimalst opened this issue a month ago · 13 comments

Describe the bug

Hi folks, I am trying to instrument my service with the Otel java agent. For some reason however, the agent appears to be ignoring specific parameters or environment variables, such as OTEL_SERVICE_NAME.

Steps to reproduce

This happens when invoking the app with

OTEL_SERVICE_NAME=validator_service java -javaagent:/path/to/otel-2.10.0.jar -DapplicationName=ValidatorService -Ddebug=true -Dlogback.debug=true -DworkingDir=/path/to/app/linux64/version -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:+UseG1GC -XX:+PrintGCDetails -Xloggc:log/gc-2024-11-26.14.41.23.log -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -Dotel.javaagent.enabled=true -Dotel.traces.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider -Dotel.java.disabled.resource.providers=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider,io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider -Dotel.javaagent.logging=application -Dotel.instrumentation.common.default-enabled=true -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true -Dotel.instrumentation.jetty.enabled=true -Dotel.instrumentation.jetty-httpclient.enabled=true -Dotel.instrumentation.vertx-http-client.enabled=true -Dotel.instrumentation.vertx-redis-client.enabled=false -Dotel.instrumentation.vertx-rx-java.enabled=true -Dotel.instrumentation.vertx-sql-client.enabled=true -Dotel.instrumentation.vertx-web.enabled=true -Dotel.instrumentation.jdbc.enabled=true -Dotel.instrumentation.http-url-connection.enabled=true -Dotel.instrumentation.jms.enabled=true -Dotel.instrumentation.apache-httpclient.enabled=false -classpath [class path libraries] path.to.my.ValidationService

Expected behavior

The otel service name to be validator_service in Datadog, and traces being produced

Actual behavior

Traces are produced, but only for unknown_service:java in Datadog. I can see the configuration parameters reflected in the telemetry as process args.

When I enable debug logs, I see the log line:

..., process.pid=, service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}

So it seems the autoconfigured SDK builder is being created and picking up some attributes, but not all

Javaagent or library instrumentation version

2.10.0

Environment

JDK: jdk1.8.0_173
OS: RHEL7

Additional context

No response

Answer 1 · 2024-11-29T19:33:04.000Z

I am able to observe the agent picking up the environment variable OTEL_SERVICE_NAME as expected when running an application in the format you provided.

I don't see you passing a javaagent in your command, how are you using the instrumentation?

Answer 2 · 2024-11-29T20:09:18.000Z

Hey @jaydeluca - Thanks for your response. I accidentally stripped the -javaagent:path/to/jar command because it had a username in the path, but edited it for clarity now. I did double check that the -javaagent param is the first param after the java command.
Here's another sample log line: I am trying to provide configuration via the arg -Dotel.javaagent.configuration.file=/home/username/code/servicename/otel.properties

Inside the otel.properties files I have otel.service.name defined:

# OpenTelemetry Java Agent Configuration

# Core Settings
otel.javaagent.enabled=true
otel.service.name=test_servicename

# Exporters Configuration
otel.traces.exporter=otlp
otel.logs.exporter=otlp
otel.metrics.exporter=otlp
otel.exporter.otlp.endpoint=http://localhost:4318

In the app logs

figuredOpenTelemetrySdkBuilder Global OpenTelemetry set to OpenTelemetrySdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="hostname", process.command_line="/path/to/my/jdk1.8.0_173/jre/bin/java -javaagent:/home/username/otel.jar -DapplicationName=Validator -Dotel.javaagent.configuration.file=/home/username/code/servicename/otel.properties
...
service.name="unknown_service:java"

Or, if I try -Dotel.service.name=some-service-name

2024-11-29T13:59:21.630 DEBUG [main                          ] figuredOpenTelemetrySdkBuilder Global OpenTelemetry set to OpenTelemetrySdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="ch12ldvdi457", process.command_line="/java/linux64/jdk1.8.0_173/jre/bin/java -javaagent:/home/username/otel.jar -Dotel.service.name=my_service_name -DapplicationName=FitValidator  -Ddebug=true -Dlogback.debug=true -DworkingDir=/app/linux64/someversion -Dcom.sun.management.jmxremote.ssl=false -Dotel.javaagent.enabled=true -Dotel.traces.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider -Dotel.java.disabled.resource.providers=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider,io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider -Dotel.javaagent.logging=application -Dotel.instrumentation.common.default-enabled=true -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true -Dotel.instrumentation.jetty.enabled=true -Dotel.instrumentation.jetty-httpclient.enabled=true -Dotel.instrumentation.vertx-http-client.enabled=true -Dotel.instrumentation.vertx-redis-client.enabled=false -Dotel.instrumentation.vertx-rx-java.enabled=true -Dotel.instrumentation.vertx-sql-client.enabled=true -Dotel.instrumentation.vertx-web.enabled=true -Dotel.instrumentation.jdbc.enabled=true -Dotel.instrumentation.http-url-connection.enabled=true -Dotel.instrumentation.jms.enabled=true -Dotel.instrumentation.apache-httpclient.enabled=false com.chicagotrading.fod.fitting.validation.ValidationService",  service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}, s, process.pid=1646, service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}, metricReaders=[PeriodicMetricReader{exporter=OtlpHttpMetricExporter{exporterName=otlp, type=metric, endpoint=http://localhost:4318/v1/metrics, timeoutNanos=10000000000, proxyOptions=null, compressorEncoding=null, connectTimeoutNanos=10000000000, exportAsJson=false, headers=Headers{User-Agent=OBFUSCATED}, retryPolicy=RetryPolicy{maxAttempts=5, initialBackoff=PT1S, maxBackoff=PT5S, backoffMultiplier=1.5}, aggregationTemporalitySelector=AggregationTemporalitySelector{COUNTER=CUMULATIVE, UP_DOWN_COUNTER=CUMULATIVE, HISTOGRAM=CUMULATIVE, OBSERVABLE_COUNTER=CUMULATIVE, OBSERVABLE_UP_DOWN_COUNTER=CUMULATIVE, OBSERVABLE_GAUGE=CUMULATIVE, GAUGE=CUMULATIVE}, defaultAggregationSelector=DefaultAggregationSelector{COUNTER=default, UP_DOWN_COUNTER=default, HISTOGRAM=default, OBSERVABLE_COUNTER=default, OBSERVABLE_UP_DOWN_COUNTER=default, OBSERVABLE_GAUGE=default, GAUGE=default}, memoryMode=REUSABLE_DATA}, intervalNanos=60000000000}], metricProducers=[], views=[]}, loggerProvider=SdkLoggerProvider{clock=SystemClock{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="", process.command_line="/java/linux64/jdk1.8.0_173/jre/bin/java -javaagent:/home/username/otel.jar -Dotel.service.name=my_service_name -DapplicationName=Validator -Ddebug=true -Dlogback.debug=true -DworkingDir=/app/linux64/someversion -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:+UseG1GC -XX:+PrintGCDetails -Xloggc:log/gc-2024-11-26.14.41.23.log -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -Dotel.javaagent.enabled=true -Dotel.traces.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider -Dotel.java.disabled.resource.providers=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider,io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider -Dotel.javaagent.logging=application -Dotel.instrumentation.common.default-enabled=true -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true -Dotel.instrumentation.jetty.enabled=true -Dotel.instrumentation.jetty-httpclient.enabled=true -Dotel.instrumentation.vertx-http-client.enabled=true -Dotel.instrumentation.vertx-redis-client.enabled=false -Dotel.instrumentation.vertx-rx-java.enabled=true -Dotel.instrumentation.vertx-sql-client.enabled=true -Dotel.instrumentation.vertx-web.enabled=true -Dotel.instrumentation.jdbc.enabled=true -Dotel.instrumentation.http-url-connection.enabled=true -Dotel.instrumentation.jms.enabled=true -Dotel.instrumentation.apache-httpclient.enabled=false com.chicagotrading.fod.fitting.validation.ValidationService", process.executable.path="/java/linux64/jdk1.8.0_173/jre/bin/java", process.pid=1646, service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}, logLimits=LogLimits{maxNumberOfAttributes=128, maxAttributeValueLength=2147483647}, logRecordProcessor=BatchLogRecordProcessor{logRecordExporter=OtlpHttpLogRecordExporter{exporterName=otlp, type=log, endpoint=http://localhost:4318/v1/logs, timeoutNanos=10000000000, proxyOptions=null, compressorEncoding=null, connectTimeoutNanos=10000000000, exportAsJson=false, headers=Headers{User-Agent=OBFUSCATED}, retryPolicy=RetryPolicy{maxAttempts=5, initialBackoff=PT1S, maxBackoff=PT5S, backoffMultiplier=1.5}, memoryMode=REUSABLE_DATA}, scheduleDelayNanos=1000000000, maxExportBatchSize=512, exporterTimeoutNanos=30000000000}}, propagators=DefaultContextPropagators{textMapPropagator=MultiTextMapPropagator{textMapPropagators=[W3CTraceContextPropagator, W3CBaggagePropagator]}}} by autoconfiguration

I see that the process arg is there, and even being capture as a process arg, but I still see service.name="unknown_service:java"

Answer 3 · 2024-11-29T20:21:54.000Z

It feels like this issue may be specific to my environment due to our jdk version?
E.g. I do see in some of the debug logs:

2024-11-29T13:59:21.629 DEBUG [main                          ] i.o.j.t.SafeServiceLoader      Unable to load instrumentation class: io/opentelemetry/javaagent/shaded/instrumentation/javaagent/runtimemetrics/java17/Java17RuntimeMetricsInstaller has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 52.0

But I assumed I would see that same error for the autoconfiguration class if that's why it wasn't catching specific parameters or env vars e.g.

Answer 4 · 2024-11-29T20:23:54.000Z

the property for the javaagent config file should be -Dotel.javaagent.configuration-file instead of -Dotel.javaagent.configuration.file

You could also try using the new declarative-config approach

Answer 5 · 2024-11-29T20:37:51.000Z

Ah, thanks - tried again with -Dotel.javaagent.configuration-file=/home/username/code/service/otel.properties but no change in behavior, it still seems to be ignoring the otel.service.name specified in the properties file.

Answer 6 · 2024-11-30T22:32:45.000Z

would you be able to provide a link to a github repo with a minimal app that reproduces the issue? I'm able to use the same properties file you provided and I see the service name as expected

Answer 7 · 2024-12-02T07:27:20.000Z

service.name is set by a resource provider https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/internal/EnvironmentResourceProvider.java. Using -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider disables all other resource providers except what you have listed, including the provider that sets the service.name. cc @jack-berg

Answer 8 · 2024-12-02T14:22:15.000Z

@laurit Ah - thank you, that appears to have been the fix, as after removing that line I do see service names in my traces!
So basically I would need to add -Dotel.java.enabled.resource.providers=**io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider**,others ?
Edit: Is it worth adding that to the docs here https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/docs/languages/java/configuration.md ? Happy to raise a PR to do so

Answer 9 · 2024-12-02T15:37:27.000Z

So basically I would need to add -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider,others ?

Edit: Is it worth adding that to the docs here https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/docs/languages/java/configuration.md ? Happy to raise a PR to do so

There is some description of otel.java.enabled.resource.providers here. Perhaps it could use some clarification, but I'm curious what your interpretation was.

Going back the config from the original issue, you had (formatted as a list for readability):

Dotel.java.enabled.resource.providers
- io.opentelemetry.instrumentation.resources.HostResourceProvider
- io.opentelemetry.instrumentation.resources.ProcessResourceProvider
Dotel.java.disabled.resource.providers
- io.opentelemetry.contrib.gcp.resource.GCPResourceProvider
- io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider

The way this worked:

Setting otel.java.enabled.resource.providers causes only specified HostResourceProvider and ProcessResourceProvider to be active
Setting otel.java.disabled.resource.providers overrides otel.java.enabled.resource.providers, but does nothing in this case because neither GCPResourceProvider or Ec2ResourceProvider are enabled

Answer 10 · 2024-12-02T16:12:47.000Z

Going back the config from the original issue, you had (formatted as a list for readability):

Dotel.java.enabled.resource.providers

io.opentelemetry.instrumentation.resources.HostResourceProvider

io.opentelemetry.instrumentation.resources.ProcessResourceProvider

Dotel.java.disabled.resource.providers

io.opentelemetry.contrib.gcp.resource.GCPResourceProvider

io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider

The way this worked:

Setting otel.java.enabled.resource.providers causes only specified HostResourceProvider and ProcessResourceProvider to be active

Setting otel.java.disabled.resource.providers overrides otel.java.enabled.resource.providers, but does nothing in this case because neither GCPResourceProvider or Ec2ResourceProvider are enabled

Hey @jack-berg thanks for the response. I originally went down this path because I noticed traces published at start up indicating the agent was attempting to call AWS/GCP metadata services. I picked those two providers from the list here: https://opentelemetry.io/docs/languages/java/configuration/#resourceprovider because I was trying to pick the minimal resource providers for a faster startup time, and figured I would add more if what they had wasn't sufficient.

Since enabled.resource.providers is a strict allow list, this behavior makes sense/isn't a bug - but I read the note and list here as being a comprehensive list of all possible resource providers rather than only those in -contrib, so I didn't realize I was disabling built in providers

Answer 11 · 2024-12-04T21:35:13.000Z

@laurit Ah - thank you, that appears to have been the fix, as after removing that line I do see service names in my traces! So basically I would need to add -Dotel.java.enabled.resource.providers=**io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider**,others ? Edit: Is it worth adding that to the docs here https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/docs/languages/java/configuration.md ? Happy to raise a PR to do so

Just ran into the same thing today, never would have occurred to me a resource provider would prevent the OTEL service name attribute from being set and thus mangling under which service my traces/metrics would be routed to.

Answer 12 · 2024-12-04T22:02:59.000Z

I agree this is confusing. Luckily it looks like the (still experimental) declarative configuration is much clearer when it comes to including and excluding resource attributes: https://github.com/open-telemetry/opentelemetry-configuration/blob/60a2d91aa03b94b1bd34b85cc349a83ed9f35527/examples/kitchen-sink.yaml#L439-L455

you can start using declarative configuration already if you would like, here's an example with the java agent: https://github.com/open-telemetry/opentelemetry-java-examples/blob/main/javaagent/sdk-config.yaml

Answer 13 · 2024-12-04T22:31:59.000Z

Just tried this (as per the solution mentioned above) and it still didn't work, maybe there is another one I need for it to work with -D params?

-Dotel.java.enabled.resource.providers=io.opentelemetry.contrib.aws.resource.EcsResourceProvider,io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider
-Dotel.service.name=alexei-test

I think I might just go for the exclude list now and remove the ones that add all the attributes I don't want (not ideal as I want to tightly control what attributes get added and don't want new ones appearing without us opting in)