connect ETIMEDOUT 54.90.52.143:443

[baileywall]

Issue Summary

In our production environment we send thousands of emails per day via sendMultiple. Over the past week or so we have run into this error intermittently: connect ETIMEDOUT 54.90.52.143:443

I've reached out to sendgrid support and was initially told that this is a rate limit issue, but our volume per second doesn't even come close to what we've been told the max rate is (10k a second) and we aren't seeing the 429 errors that the documentation says are returned in the case of exceeding the limit. I was then told it's probably just a connection issue, and to wait a few seconds and re-issue the request. Sporadic connection issues are worrying enough that I think some investigation is warranted, even though at our volume simply re-issuing the request will likely mitigate the issue for us (for now).

Steps to Reproduce

1. Send many emails via the sendMultiple method and wait for one of them to timeout due to a connection issue

Code Snippet

import { MailData } from '@sendgrid/helpers/classes/mail';
import mail from '@sendgrid/mail';
    
mail.setApiKey(apiKey);

const emailDataObject: MailData = {...};

return await mail.sendMultiple(emailDataObject);

Exception/Log

Error: connect ETIMEDOUT 54.90.52.143:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
    at TCPConnectWrap.callbackTrampoline (internal/async_hooks.js:126:14)
 => awaited here:
    at Function.Promise.await (/app/.meteor/heroku_build/app/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/external-apis/emailClient.ts:142:5
    at /app/.meteor/heroku_build/app/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/.meteor/heroku_build/app/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/modules/email/email-notifications/emailDigest.ts:884:3
    at /app/.meteor/heroku_build/app/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40

Technical details:

• sendgrid-nodejs version: 6.4.0
• node version: 14.17.5

[baileywall]

noticed this issue that seems like the same issue from February 2019 (that is now closed, which is why I've opened a new issue): #896

[childish-sambino]

I think setting a default request timeout and utilizing the keepAlive parameter in the axios client should prevent this.

This issue has been added to our internal backlog to be prioritized. Pull requests and +1s on the issue summary will help it move up the backlog.

[yiyao8932]

I'm also experiencing the same issue but I see a different IP address 54.248.129.79:443 in the error message. It just started a few weeks ago. Our production environment sends at most 1000 emails per hour.

Here's our sample code

import { MailDataRequired } from "@sendgrid/helpers/classes/mail";
import sendgrid from "@sendgrid/mail";

sendgrid.setApiKey(SENDGRID_API_KEY);
sendgrid.setTimeout(30000);

const sendGridMailArguments: MailDataRequired = {...};
await sendgrid.send(sendGridMailArguments); 

We use for loop to send out emails one by one but we have also added a wait for 500ms between each email.
I also notice that each timeout waits for 2 minutes before it proceeds to next email sending.

Technical details:

• sendgrid/mail version: 7.1.1

• node version: 10.19