Terragrunt run-all plan failing with exec.ExitError exit status 126.

[nishilampally]

Recently we have implemented tfenv and tgenv to manage multiple versions of terraform and terragrunt . When I try to run terragrunt run-all plan I'm getting a 126 error exit code. I've enabled debug logging and it's not very helpful. The issue is intermittent where run--all plan get succeeds sometimes and fails intermittently. User(AzDevOps) has execute permissions on both terraform and terragrunt. I am not sure why I am getting this error. There is not much information available from the logs.

Setup:
Ubuntu machine 20_04-lts

Command:
terragrunt run-all plan --terragrunt-source-update -no-color --terragrunt-include-external-dependencies

Permissions:

drwxr-xr-x 3 root root 4096 Sep 20 16:48 aws-cli
drwxr-xr-x 3 root root 4096 Sep 20 16:48 bin
drwxr-xr-x 2 root root 4096 Aug 31 21:39 etc
drwxr-xr-x 2 root root 4096 Aug 31 21:39 games
drwxr-xr-x 2 root root 4096 Aug 31 21:39 include
drwxr-xr-x 3 root root 4096 Aug 31 21:39 lib
lrwxrwxrwx 1 root root 9 Aug 31 21:39 man -> share/man
drwxr-xr-x 2 root root 4096 Aug 31 21:39 sbin
drwxr-xr-x 5 root root 4096 Sep 20 16:44 share
drwxr-xr-x 2 root root 4096 Aug 31 21:39 src
drwxrwxrwx 10 root root 4096 Sep 20 16:46 tfenv
drwxrwxrwx 9 root root 4096 Sep 20 16:46 tgenv
/agent/_work/1/s
total 60

total 44
-rwxrwxrwx 1 root root 304 Sep 20 16:46 CHANGELOG.md
-rwxrwxrwx 1 root root 1086 Sep 20 16:46 LICENSE
-rwxrwxrwx 1 root root 5440 Sep 20 16:46 README.md
drwxrwxrwx 2 root root 4096 Sep 20 16:46 assets
drwxrwxrwx 2 root root 4096 Sep 20 16:46 bin
drwxrwxrwx 2 root root 4096 Sep 20 16:46 libexec
-rwxrwxrwx 1 root root 1165 Sep 20 16:46 list_all_versions_offline
drwxrwxrwx 2 root root 4096 Sep 20 16:46 test
-rwxrwxrwx 1 root root 7 Sep 20 16:46 version
drwxrwxrwx 4 root root 4096 Oct 19 19:41 versions
/agent/_work/1/s
total 8
drwxr-xr-x 2 AzDevOps AzDevOps 4096 Oct 19 19:42 1.0.1
drwxrwxrwx 2 root root 4096 Sep 20 16:46 1.5.5
total 8
drwxr-xr-x 2 AzDevOps AzDevOps 4096 Oct 19 19:41 0.37.1
drwxrwxrwx 2 root root 4096 Sep 20 16:46 0.49.1
/agent/_work/1/s
total 78812
-rwxr-xr-x 1 AzDevOps AzDevOps 80702103 Jun 24 2021 terraform
total 35700
-rwxr-xr-x 1 AzDevOps AzDevOps 36554016 Oct 19 19:41 terragrunt
/agent/_work/1/s
-rwxr-xr-x 1 AzDevOps AzDevOps 80702103 Jun 24 2021 /usr/local/tfenv/versions/1.0.1/terraform
-rwxr-xr-x 1 AzDevOps AzDevOps 36554016 Oct 19 19:41 /usr/local/tgenv/versions/0.37.1/terragrunt

Error:

time=2023-11-06T03:03:51Z level=error msg=169 errors occurred:
* Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado_access_policies/awslandingzone_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/key_vault_ado]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/omnicc/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault_access_policies/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault, /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/disk_encryption_set]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/appdeploy_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* exit status 126
* exit status 126
* Cannot process module Module /agent/_work/1/s/shared_cac/services/private_dns_zones/privatelink_postgres_database_azure_com (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault_keys/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/onprem_pool/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/apps/dea/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/dea/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/dea/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/agent_pool/vmss_iac (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/prd/network/subnet_adoagents, /agent/_work/1/s/prd/apps/omnicc/agent_pool/disk_encryption_set]) because one of its dependencies, Module /agent/_work/1/s/prd/network/subnet_adoagents (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/vnet]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/network/subnet_adoagents (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/vnet]) because one of its dependencies, Module /agent/_work/1/s/prd/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/resource_group_vnet]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/network/resource_group_vnet]) because one of its dependencies, Module /agent/_work/1/s/prd/network/resource_group_vnet (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault_keys/disk_encryption_set]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault_keys/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault_keys/disk_encryption_set (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault]) because one of its dependencies, Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/azure_devops/agent_pool/vmss_mfbuild/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/azure_devops/agent_pool/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/prd/services/virtual_desktop/biz_users/app_group]) because one of its dependencies, Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/services/aks (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/services/container_registry_forestersfinancial, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace, /agent/_work/1/s/shared_cac/services/private_dns_zones/privatelink_canadacentral_azmk8s_io, /agent/_work/1/s/prd/network/subnet_kubernetes, /agent/_work/1/s/prd/network/route_table_kubernetes, /agent/_work/1/s/prd/network/resource_group_vnet]) because one of its dependencies, Module /agent/_work/1/s/prd/network/resource_group_vnet (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/apps/omnicc/key_vault_ado (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/apps/omnicc/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/prd/apps/omnicc/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_share_profiles (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_account_profiles]) because one of its dependencies, Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_account_profiles (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/storage_account_profiles (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault]) because one of its dependencies, Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]), finished with an error: Cannot process module Module /agent/_work/1/s/prd/services/virtual_desktop/biz_users/key_vault (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/prd/services/virtual_desktop/biz_users/resource_group, /agent/_work/1/s/shared_cac/operations/loganalytics_workspace]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/operations/loganalytics_workspace (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/operations/resource_group_loganalytics]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/operations/resource_group_loganalytics (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/shared_cac/services/private_dns_zones/privatelink_canadacentral_azmk8s_io (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/services/private_dns_zones/resource_group (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
* Cannot process module Module /agent/_work/1/s/shared_cac/network/subnet_azurefw (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/network/vnet]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/network/resource_group_vnet]), finished with an error: Cannot process module Module /agent/_work/1/s/shared_cac/network/vnet (excluded: false, assume applied: false, dependencies: [/agent/_work/1/s/shared_cac/network/resource_group_vnet]) because one of its dependencies, Module /agent/_work/1/s/shared_cac/network/resource_group_vnet (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126

[iokiwi]

Note that a similar issue has been raised for tgenv/tgenv here tgenv/tgenv#27