Enabling epsagon tracing at service level causes sls deploy to take a really long time

Question

Enabling epsagon tracing at service level causes sls deploy to take a really long time

Closed this issue 2 years ago · 7 comments

Library version, runtime version, dependencies, etc.

epsagon: v1.106.2
serverless-plugin-epsagon: v1.10.2
serverless: v2.17.0
node: v12.21.0
npm: v6.14.11

Snippet of the usage.

...
plugins:
  - serverless-plugin-epsagon
  - serverless-webpack
  - serverless-domain-manager
  - serverless-dynamodb-local
  - serverless-offline

...
custom:
  epsagon:
    token: ${opt:epsagonToken}
    appName: ${self:service}-${self:provider.stage}
    disable: false
    metadataOnly: false

Details

I found out when disable option is set to false, allowing our functions to be wrapped with Epsagon wrapper, it causes sls deploy to take a really long time to complete about 8+ minutes as opposed when disable is set to true, it only takes 1+ minute to deploy all 18 of our functions

When epsagon is enabled (sls deploy took ~8 minutes)

When epsagon is disabled (sls deploy took ~2 minute)

This is expensive to us because we use Github Action to run our deployments and our app which has epsagon enabled eats up most of our provisioned minutes.

Even more worried if the list of functions grow, sls deploy could take even longer?

Answer 1 · 2021-03-26T21:37:32.000Z

Hi @uccmen, I've tried to replicate it on my end but couldn't yield the same performance impact. Can you kindly share a small reproducible example please?

Answer 2 · 2021-03-28T11:33:55.000Z

Hi @ranrib, thanks for spending some time trying to replicate the issue.

I prepared a simple demo with very similar configuration with our app here - https://github.com/uccmen/epsagon-deploy
with 2 PRs indicating when epsagon is enabled and disabled.

The one similarity I noticed between the demo and our app is that webpack compilation time took longer when epsagon is enabled.

I've provided more information in the README on how to reproduce (with Github Actions triggered).

Hope that helps.

Answer 3 · 2021-04-04T10:44:21.000Z

Thanks @uccmen - looks like locally this doesn't reproduce - I ran both scenarios on my Mac and both took 15-18 seconds. We will keep investigating but if you have any clue it will be highly appreciated.

Answer 4 · 2021-04-05T06:34:54.000Z

@ranrib, played around with nodeExternals thinking it would help but the outcome is still the same, compilation time still takes longer when epsagon is enabled. any ideas if how epsagon is wrapping the functions could contribute to that time during sls deploy?

Answer 5 · 2021-08-11T23:21:45.000Z

Datapoint; I can reproduce this with with the NCC plugin serverless-plugin-ncc but only when setting minified to true when the Epsagon plugin is enabled.

Ie:

service: foo

frameworkVersion: '>=1.2.0'

plugins:
  - serverless-plugin-epsagon
  - serverless-plugin-ncc
<SNIP>
custom:
  ncc:
    minify: true
    externals: [aws-sdk]
  epsagon:
    token: ${ssm:/epsagon/apiToken~true}
    appName: foo-${opt:stage, 'dev'}

Answer 6 · 2021-08-12T03:29:49.000Z

Another datapoint, if use a local version of serverless-plugin-ncc but use @vercel/ncc instead of the deprecated @zeit/ncc that long delay goes away.

Answer 7 · 2021-08-12T03:45:48.000Z

Last comment, promise, but we've just switched to https://github.com/floydspace/serverless-esbuild which looks better maintained, and performs better.