Crash when deploying in Bitbucket CI/CD environment with more than 3 clients

[yossarian21]

Checklist

• I have looked into the README and have not found a suitable solution or answer.
• I have looked into the documentation and have not found a suitable solution or answer.
• I have searched the issues and have not found a suitable solution or answer.
• I have upgraded to the latest version of this tool and the issue still persists.
• I have searched the Auth0 Community forums and have not found a suitable solution or answer.
• I agree to the terms within the Auth0 Code of Conduct.

Description

We have an Auth0 environment with about 8 Clients and are using the auth0-deploy-cli to provision multiple Auth0 tenants. I have been migrating the deployment script from "run it on a laptop" to proper CI/CD in Bitbucket Pipelines.

I have a simple Node script that retrieves our configuration (e.g. client id/client secret) and then invokes the CLI via a module:

    await deploy({
        input_file: path.resolve(`${__dirname}/../config/tenant.yaml`),
        config,
    });

This has been working fine when running locally in a Docker containerized environment; however, in Bitbucket Pipelines, I was receiving the following crash. If I comment out most of our tenants, I can run the script successfully in Pipelines. Once I have configured more than 3 clients, I start getting the crash:

2023-04-14T19:38:49.084Z - info: Getting access token for <clientid>/<tenant>.us.auth0.com
2023-04-14T19:38:53.873Z - info: Updated [rules]: {"name":"auth0-account-link-extension","order":1}
2023-04-14T19:38:54.345Z - info: Updated [resourceServers]: {"name":"...","identifier":"..."}
2023-04-14T19:38:54.586Z - info: Updated [clients]: {"name":"Auth0 Management Client","client_id":"..."}
/usr/local/bin/node[290]: ../src/async_wrap.cc:324:static void node::AsyncWrap::QueueDestroyAsyncId(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `args[0]->IsNumber()' failed.
 1: 0xb02930 node::Abort() [/usr/local/bin/node]
 2: 0xb029ae  [/usr/local/bin/node]
 3: 0xa5040b node::AsyncWrap::QueueDestroyAsyncId(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
 4: 0xd396ae  [/usr/local/bin/node]
 5: 0xd3aacf v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
 6: 0x15d5519  [/usr/local/bin/node]

A developer on the team who uses a Mac occasionally receives the same crash, when running our Docker container. He has an ARM processor so Docker must emulate the amd64 environment. The issue in Bitbucket is always reproducible.

Expectation

I should be able to use auth0-deploy-cli from the Bitbucket Pipelines CI/CD environment.

Reproduction

Given a tenant configuration with more than 3 clients,
when I run the deployment script from Bitbucket Pipeplines CI/CD,
then node crashes.

Deploy CLI version

7.17.1

Node version

16.17.0

[willvedd]

Thanks for looking into this a bit more. It seems like concurrent requests triggers this error in Bitbucket but sometimes locally too if I'm reading correctly.

I'd like to better understand why this is happening in hopes of applying an appropriate fix; throttling the concurrent API requests to one at a time strikes me as heavy-handed and will drastically bloat your build times.

I'm inclined to believe that trouble originates from the Bitbucket Pipeline instead of being a Deploy CLI or Node issue. I'd inquire about your Bitbucket service tier and investigate if there are any resource limits imposed on your CI. We've had customers who have run up against port limits in the past but I could see this being a memory limit too. A quick search reveals that BB does have some limits, especially for the lower tiers.

[yossarian21]

FYI, we upgraded to a paid plan and I changed back to the default concurrency, and we still see the same crash.

Incidentally, we have also upgraded to Node 18.16.0, so we've seen the crash on node 16 and 18.

[willvedd]

@yossarian21 Do you have any further updates on this issue? I certain sympathize and can see how this would be frustrating. However, I don't believe that this is currently actionable. I'm unable to reproduce nor do I have adequate information on the root cause of this issue. Further, I'm still firm that allowing users to disable concurrent requests is a net negative to this project; it would neuter the utility of this tool for the majority of users.

Similar issues in the past have usually been caused by tight resource constraints. I still suspect this is the case here too.

I intend to close this issue out unless there are some reproducible steps or logs that can help troubleshoot.

[yossarian21]

I tried to create a minimum reproducible case but I am also not able to reproduce it with a smaller example. We are using a patched version of auth0-deploy-cli with the changes from my PR, setting the concurrency to 1, as a workaround. You can close this and I'll reply if I'm able to create a reproducible repo.