newrelic/elixir_agent

DistributedTrace.generate_sampling timeouts.

Opened this issue · 1 comments

noizu commented

Describe the bug
Sampling timeouts occur under load on my IOT facing API nodes.

Need a way to increase timeouts or expose other configuration settings to avoid this exception.

image

Environment

  • Elixir & Erlang version (elixir -v):
    Erlang/OTP 22 [erts-10.7.2.16] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

  • Agent version (mix deps | grep new_relic_agent):
    new_relic_agent (Hex package)
    (mix) locked at 1.27.7 (new_relic_agent) a0aa7208

noizu commented

I was eventually able to resolve this by tweaking the following settings.

config :new_relic_agent,
analytic_event_per_minute: 250, # Reducing caps to avoid overwhelming api node telemetry
custom_event_per_minute: 250,
error_event_per_minute: 100,
span_event_per_minute: 100,
sampling_target: 5

Should we prepare a doc update to cover options for tweaking sampling rates and caps for high load environments to safe the next person some time?