Getting timeouts when calling from Node Lambda

Question

Getting timeouts when calling from Node Lambda

Closed this issue 7 years ago · 2 comments

We have an AWS Lambda function that uses the SignalFx node client to post metrics to our account. Lately I've been seeing an increased number of these errors in our Lambda's CloudWatch logs:

error: Failed to send datapoint: undefined Error: ETIMEDOUT
 at null._onTimeout (/var/task/node_modules/signalfx/node_modules/request/request.js:846:19)
 at Timer.listOnTimeout (timers.js:92:15)

The incidence of these errors relative to overall load is not high, but we have seen a spike in these errors recently.

Answer 1 · 2018-04-11T22:43:19.000Z

Btw, does SignalFx expose any sort of status information either via web UI or API? We saw the behavior described above persist for a few hours and then seem to go away, so I'm wondering if SignalFx's intake APIs experience occasional latency spikes.

Answer 2 · 2018-04-12T22:25:21.000Z

I had a chat with SignalFx support today, and learned that the default timeout in the version of the Node client that we're using is 1 second, and that this value was recently raised to 5 seconds. Given that our Lambda is configured to time out at 10 seconds, it would seem that any SignalFx timeouts we experience are not likely to be contributing to or causing the Lambda timeouts that we're seeing.