[custom-roles-analyzer] Quota exceeded for quota metric 'SearchAllIamPolicies Requests' and limit 'SearchAllIamPolicies Requests per minute' of service 'cloudasset.googleapis.com'

[jenson-gabriel]

We are running into a Quota limit error on the Cloud Asset API for SearchAllIamPolicies. The max quota for SearchAllIamPolicies is 400, which is the current value in the project.

No other applications are using the Cloud Asset API in the current project.

If the query parameter is set to a specific group in AssetServiceUtil.java, then we did see the output CSV file generate successfully. The Custom Role Analyzer application should work once we get around this quota issue.

Error output:

Apr 11, 2022 10:03:02 PM com.google.cloud.pso.security.asset.AssetServiceUtil getAssetServiceClient
INFO: Get AssetServiceClient.
Apr 11, 2022 10:03:02 PM com.google.cloud.pso.security.asset.AssetServiceUtil createAssetServiceClient
INFO: Creating AssetServiceClient.
Apr 11, 2022 10:08:58 PM com.google.cloud.pso.security.CustomRoleAnalyzer main
SEVERE: Unable to process bindings for the roles.
com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'SearchAllIamPolicies Requests' and limit 'SearchAllIamPolicies Requests per minute' of service 'cloudasset.googleapis.com' for consumer 'project_number:[REDACTED]'.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1133)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:564)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:534)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:455)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
        Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
                at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
                at com.google.api.gax.paging.AbstractPage.getNextPageImpl(AbstractPage.java:118)
                at com.google.api.gax.paging.AbstractPage.getNextPage(AbstractPage.java:91)
                at com.google.api.gax.paging.AbstractPage$AllResourcesIterator.computeNext(AbstractPage.java:177)
                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
                at com.google.cloud.pso.security.asset.AssetServiceUtil.searchAllIamPolicies(AssetServiceUtil.java:125)
                at com.google.cloud.pso.security.asset.AssetServiceUtil.analyzeBindings(AssetServiceUtil.java:82)
                at com.google.cloud.pso.security.CustomRoleAnalyzer.main(CustomRoleAnalyzer.java:111)
                at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:254)
                ... 1 more
Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'SearchAllIamPolicies Requests' and limit 'SearchAllIamPolicies Requests per minute' of service 'cloudasset.googleapis.com' for consumer 'project_number:[REDACTED]'.
        at io.grpc.Status.asRuntimeException(Status.java:535)
        ... 14 more

Can the "custom-roles-analyzer" application be adjusted to better handle the SearchAllIamPolicies max quotas?