Deploy broken in v7.23.0: SCIM support for connection management requires read:scim_config scope or update to excluded_props

[bweisberger]

Checklist

• I have looked into the README and have not found a suitable solution or answer.
• I have looked into the documentation and have not found a suitable solution or answer.
• I have searched the issues and have not found a suitable solution or answer.
• I have upgraded to the latest version of this tool and the issue still persists.
• I have searched the Auth0 Community forums and have not found a suitable solution or answer.
• I agree to the terms within the Auth0 Code of Conduct.

Description

We successfully deployed as recently as two days ago, but suddenly our deploys are breaking with the following error:
error: SCIM request failed with statusCode 403 (insufficient_scope). Insufficient scope, expected any of: read:scim_config.

Looking at the auth0-deploy-cli repo, I can see this PR to add SCIM configuration management was merged yesterday.

We do use the a0deploy cli tool to manage our connections. Our deploy script installs the package without pinning a version, so I'm assuming it is installing the latest version:

npm install -g auth0-deploy-cli

We don't have the read:scim_config permission enabled for our Auth0 Deploy CLI client in Auth0, and we don't have the connections.scim_configuration property in our excluded props. I'm reasoning we shouldn't have to make a code change without a major version update.

Thanks for your help!

Expectation

Installing the latest minor version of the auth0-deploy-cli package should not require code updates or cause deploys to fail.

Reproduction

1. Given your Auth0 Deploy CLI client does not have the read:scim_config scope, and connections.scim_configuration is not an excluded prop
2. When you download the latest minor version (7.23.0) of the auth0-deploy-cli npm package and run a0deploy import to deploy IaC
3. The deploy will fail with an error that the read:scim_config scope is required.

Deploy CLI version

7.23.0

Node version

16.20.2

[jbatet-ut]

We're getting the very same issue here, our pipelines broke today.

To add some information, just worth mentioning that, while investigating, we added the scope read:scim_config to the list of permissions our a0deploy Application in our tenants to see if that helped and after that we're now facing this other error:

error: SCIM request failed with statusCode 429 (too_many_requests). Global limit has been reached.

In the meantime we're trying to pin to the previous working version, 7.22.1, but would be nice to have feedback on this one. Thanks!

[bweisberger]

I added scim_configuration to our excluded_props in our config and that did the trick I think (confirmed locally, not yet in CI/CD: That didn't work either (my local version seems pinned at 7.21.0). We're pinning to 7.22.1 as well.

Thanks!

[nandan-bhat]

Hello @bweisberger,

Thanks for your patience. I am looking into this issue.

[kschmi-icims]

We also ran into issues, where our deploy is suddenly taking 10X longer , since there's now a call for every enterprise connection. Build time went from 1m 58s to 11m 8s . We also are seeing our builds fail now with a 524 error, retrying works, but we didn't see errors like this until v7.23.0.

2024-07-26T19:41:44.091Z - warn: SCIM configuration is not enabled on connection "con_YLzAe8C8VNUGKfzn".
2024-07-26T19:41:44.341Z - debug: Getting SCIM configuration from connection con_v6eW9RcHnJLRPcHH
2024-07-26T19:43:24.512Z - error: SCIM request failed with statusCode 524 (). .
2024-07-26T19:43:24.513Z - error: Problem running command import during stage processChanges when processing type connections
2024-07-26T19:43:24.513Z - error: Request failed with status code 524
2024-07-26T19:43:24.513Z - debug: AxiosError: Request failed with status code 524
at settle (/opt/hostedtoolcache/node/18.20.4/x64/lib/node_modules/auth0-deploy-cli/node_modules/axios/dist/node/axios.cjs:1983:12)
at IncomingMessage.handleStreamEnd (/opt/hostedtoolcache/node/18.20.4/x64/lib/node_modules/auth0-deploy-cli/node_modules/axios/dist/node/axios.cjs:3085:11)
at IncomingMessage.emit (node:events:529:35)
at IncomingMessage.emit (node:domain:489:12)
at endReadableNT (node:internal/streams/readable:1400:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
at Axios.request (/opt/hostedtoolcache/node/18.20.4/x64/lib/node_modules/auth0-deploy-cli/node_modules/axios/dist/node/axios.cjs:4224:41)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[nandan-bhat]

Hi @kschmi-icims,

The current version 7.23.1 should generally avoid encountering a 429 error, though it may still experience significant delays. This is because, when SCIM is enabled, there is no direct method to check if SCIM configurations are active on a connection (It's currently a limitation of our management api). As a result, we have to iterate through each connection individually and read responses.

If SCIM is not in use on any of your connections, you can temporarily remove the read:scim_config, create:scim_config, update:scim_config, and delete:scim_config permissions from your application. This should help minimize delays on 7.23.1.

We plan to release another minor version by the end of this week, which should reduce SCIM-related delays, though it may not completely eliminate them.

And regarding 524 error, could you confirm if the 524 error is occurring consistently ?. I don't have much information on this issue at the moment, but if you can provide more details, I can investigate further.

[kschmi-icims]

Hi @nandan-bhat , we aren't using SCIM, we worked around the issue by disabling connection updates. Appreciate the upcoming patch to reduce delays.

The 524 errors were not consistent, we've seen them before under heavy load Auth0 Mgmt APIs can sometimes return a 524, the issue is the entire deploy cli run would fail and we'd have to start over.

[nandan-bhat]

Hi @kschmi-icims,

Thank you!
Please feel free to contact me if you have any further questions or need additional information.