DistributedTrace.generate_sampling timeouts.
Opened this issue · 1 comments
Describe the bug
Sampling timeouts occur under load on my IOT facing API nodes.
Need a way to increase timeouts or expose other configuration settings to avoid this exception.
Environment
-
Elixir & Erlang version (
elixir -v
):
Erlang/OTP 22 [erts-10.7.2.16] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] -
Agent version (
mix deps | grep new_relic_agent
):
new_relic_agent (Hex package)
(mix) locked at 1.27.7 (new_relic_agent) a0aa7208
I was eventually able to resolve this by tweaking the following settings.
config :new_relic_agent,
analytic_event_per_minute: 250, # Reducing caps to avoid overwhelming api node telemetry
custom_event_per_minute: 250,
error_event_per_minute: 100,
span_event_per_minute: 100,
sampling_target: 5
Should we prepare a doc update to cover options for tweaking sampling rates and caps for high load environments to safe the next person some time?