Sentry: Error uploading sourcemap file - Error: connect EADDRNOTAVAIL 35.188.42.15:443 - Local

Question

Sentry: Error uploading sourcemap file - Error: connect EADDRNOTAVAIL 35.188.42.15:443 - Local

igorescobar opened this issue 3 years ago · 27 comments

Hi,

A few days from now we started to get this error on our pipelines:

Error ---------------------------------------------------
 
  Error: Sentry: Error uploading sourcemap file - Error: connect EADDRNOTAVAIL 35.188.42.15:443 - Local (172.17.0.2:0)
      at SentryPlugin.<anonymous> (/builds/services/node_modules/serverless-sentry/dist/index.js:603:31)
      at step (/builds/services/node_modules/serverless-sentry/dist/index.js:44:23)
      at Object.throw (/builds/services/node_modules/serverless-sentry/dist/index.js:25:53)
      at rejected (/builds/services/node_modules/serverless-sentry/dist/index.js:17:65)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              14.19.3
     Framework Version:         2.72.3 (local)
     Plugin Version:            5.5.4
     SDK Version:               4.3.2
     Components Version:        3.18.2

This is what our dependencies look like at the moment:

    "@sentry/integrations": "^7.2.0",
    "@sentry/node": "^7.2.0",
    "serverless-sentry-lib": "^2.5.0",
    "serverless-sentry": "^2.5.0",

We even tried to disable this functionality by adding the environment variable SENTRY_SOURCEMAPS or adding the custom config:

sentry:
    sourceMaps: false

But it runs regardless and ignores the flag. Do you have any idea what is happening and how to work around this?

igorescobar commented 3 years ago

😄

Answer 1 · 2022-06-22T11:15:54.000Z

Maybe is this related?
https://stackoverflow.com/questions/21859537/connect-eaddrnotavail-in-nodejs-under-high-load-how-to-faster-free-or-reuse-tc

Answer 2 · 2022-06-22T11:23:58.000Z

Maybe we are being too aggressive here:
https://github.com/arabold/serverless-sentry-plugin/blob/master/src/index.ts#L476 and we are running out of TCP ports/sockets or something because of the large number of files being sent?

Answer 3 · 2022-06-22T12:41:37.000Z

Thanks for reporting @igorescobar, I'm sorry this is causing you problems. I did see that error myself but it went away on its own so I assumed it was just some sort of transient network issue.

It still uploading with sourceMaps: false is a bug, we should add a check to prevent that from happening.

As a temporary workaround you could remove your authToken config, that will disable releases entirely which will skip sourcemap upload.

My understanding is that there are around 30K ports available so I wouldn't have thought we'd be hitting that. Your setup is a bit different than mine though, I'm using serverless-webpack to bundle my files so there are only a few that get uploaded. If your setup is pulling in all of node_modules there will be many more and it's plausible you're hitting the limit. We should probably be ignoring that folder since most libraries don't have sourcemaps.

Answer 4 · 2022-06-22T12:51:55.000Z

Yeah, I also noticed that after upgrading to there is no way to disable the source maps upload (besides doing what you described). the only way is rolling it back.

Answer 5 · 2022-06-22T12:56:07.000Z

I'm literally on a plane right now but should have a quick fix in a bit.

Answer 6 · 2022-06-22T13:09:51.000Z

No risk, no fun. Just published a fully untested 2.5.1 that (hopefully) fixes two problems:

Fix #63: Upload source maps serially to avoid running out of sockets.
Correctly disable uploading source maps if the config setting is false or unset.

🤞

Answer 7 · 2022-06-22T13:17:21.000Z

Fix #63: Upload source maps serially to avoid running out of sockets.

Maybe we should do this in batches instead of calling all of them in parallel or in series.... creating batches of 50-100 requests might fix the issue. In our case it would take ages to submit all files I suppose.

Answer 8 · 2022-06-22T13:27:00.000Z

The best would be to upload like 3 to 5 simultaneously for best performance. 50-100 parallel connections are a lot. But I currently lack the time and ability to test this properly, so I went for the quickest fix. It's slower but should work regardless of how large the project is and how many artifacts it has to upload.

Answer 9 · 2022-06-22T14:35:14.000Z

The best would be to upload like 3 to 5 simultaneously for best performance. 50-100 parallel connections are a lot. But I currently lack the time and ability to test this properly, so I went for the quickest fix. It's slower but should work regardless of how large the project is and how many artifacts it has to upload.

I would also create a property to configure it if necessary. It will be hard to come up with a config that will be nice for everyone so be conservative but also allow people to be more aggressive if they have the computing power to do so 👍

Answer 10 · 2022-06-22T14:56:16.000Z

I have another update 2.5.2. Can you give it a try? It will upload 5 in parallel which should not impact your performance anymore. 10, 20, 50 really shouldn't make a noticeable difference, but feel free to adjust it and try it out if you notice something. I'm open to any changes. Again, this is fully untested 🙄

Answer 11 · 2022-06-22T16:06:13.000Z

Tests checklist v2.5.2

Source code is now honoring the sourceMaps default false and it deployed the application with no issues. (Job finished in 7 minutes 37 seconds)
sourceMaps: true. Job couldn't finish. I was taking too much time. I stop the job at 45min.

Answer 12 · 2022-06-22T16:18:19.000Z

Already 20 minutes in and counting 😄 . No errors but as expected... it will take ages to finish 😄

Answer 13 · 2022-06-22T16:24:35.000Z

😞 How long did it take before my changes to serialize the upload?

Answer 14 · 2022-06-22T16:26:22.000Z

without sourcemaps it took about 7 minutes. with sourcemaps we are now 32 minutes in.

Answer 15 · 2022-06-22T16:28:31.000Z

If you want I can try to cook a PR adding a new option where I can configure the number of files to be uploaded at the same time. I think we can easily increase it to 10 or 20.

Using Keep-Alive Agent inside the super-agent library might also speed up requests considerably.

Answer 16 · 2022-06-22T16:32:55.000Z

Well, that sucks. I don't think we need a configuration - the plugin should use a sensible value that works for most (if not all) scenarios. I'm okay with increasing it to whatever will work with a comparable performance than before. Technically, even 50 should be a problem. Seems that many, many small files still generate too much overhead. In a perfect world we would upload all files in a single request, but I haven't reviewed the Sentry APIs in ages... So, don't know if that would even be possible. Using Keep-Alive would definitely make sense as well!

If you have some availability, I'd be happy to merge in any PR that improves the situation. At least we have a working version now, even if it's unacceptable slow.

Answer 17 · 2022-06-22T16:36:37.000Z

Could you increase it to like 25 for now? I'm trying to avoid having to add another dependency to the project as superagent alone needs an extra library to handle keep-alive within node I suppose.

Answer 18 · 2022-06-22T16:40:57.000Z

moving away to fetch would also be a preferable option 👍

Answer 19 · 2022-06-22T16:42:38.000Z

(I stopped the job at 45min)

Answer 20 · 2022-06-22T16:49:21.000Z

Please give 2.5.3 a shot whenever you have the chance. I increased it to 50 parallel uploads which matches the default the AWS JavaScript SDK uses as well: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-configuring-maxsockets.html

So, this should be a sensible default. Otherwise we can still add a configuration setting or think about more complex improvements for the next update.

Answer 21 · 2022-06-22T16:50:26.000Z

Thanks! 🚀
I'm testing it out and I will let you know.

Answer 22 · 2022-06-22T17:06:01.000Z

With that being said... we definitely need to add some retries and grace periods into this logic. If you consider that you can have multiple builds running at the same time you don't want to fail the entire build because of this.

Answer 23 · 2022-06-22T17:15:51.000Z

Also... worth mentioning that even tho most requests failed because of the concurrency being too high the job finished at 16 minutes 59 seconds.

Answer 24 · 2022-06-22T17:25:52.000Z

Something to explore:
https://superchargejs.com/docs/3.x/promise-pool#error-handling

It can give you access to a list of failed items which you can re-process later at a lower concurrency.

Answer 25 · 2022-06-22T17:27:22.000Z

import { PromisePool } from '@supercharge/promise-pool'

const errors = []

const { results } = await PromisePool
  .for(users)
  .handleError(async (error, user) => {
      // you must collect errors yourself
    if (error instanceof ValidationError) {
      return errors.push(error)
    }

    // Process error handling on specific errors
    if (error instanceof ThrottleError) {
      return await retryUser(user)
    }
  })
  .process(async data => {
    // the harder you work for something,
    // the greater you’ll feel when you achieve it
  })

await handleCollected(errors) // this may throw

return { results }

Answer 26 · 2022-06-27T07:58:25.000Z

Just an update on my side. We found out a few other ways @sentry recommends for source map upload and we ended up using https://github.com/getsentry/sentry-webpack-plugin which also uploads the source maps to their service and it worked nicely for us. 👍