"Error retrieving keys for Storage Account" when using Management group scoped service principal using workload identity federation

[captainhook]

Describe the bug
When I run any of the tasks such as init, it fails saying it is unable to access the storage account.

To Reproduce
Steps to reproduce the behavior:

1. Setup service connection using 'workload identity federation' auth method
2. Grant Service Principal relevant access at Mgmt group level in Azure
3. main.tf:

# Azure Provider source and version being used
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.86.0"
    }
  }
  backend "azurerm" {
    # use_azuread_auth = true
  }
}

# Configure the Microsoft Azure Provider
provider "azurerm" {
  skip_provider_registration = false
  features {
    key_vault {
      purge_soft_delete_on_destroy = false
    }
    resource_group {
      prevent_deletion_if_contains_resources = true
    }
    subscription {
      prevent_cancellation_on_destroy = true
    }
  }

  subscription_id = "34b7d951-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

provider "azurerm" {
  alias                      = "Connectivity-prod"
  skip_provider_registration = false
  features {
    key_vault {
      purge_soft_delete_on_destroy = false
    }
    resource_group {
      prevent_deletion_if_contains_resources = true
    }
    subscription {
      prevent_cancellation_on_destroy = true
    }
  }

  subscription_id = azurerm_subscription.Connectivity-prod-001.subscription_id
}

4. Configure release pipeline with Terraform init task and configure it to use the Management group scoped service connection.

steps:
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV4@4
  displayName: 'Terraform : init'
  inputs:
    workingDirectory: '$(System.DefaultWorkingDirectory)/_AzureLandingZones/azl-tf'
    commandOptions: '-upgrade'
    backendServiceArm: 'Terraform Central Mgmt'
    backendAzureRmResourceGroupName: 'rg-TerraformMgmt-Store-prod-001'
    backendAzureRmStorageAccountName: redactedterraformstorage
    backendAzureRmContainerName: tfstate
    backendAzureRmKey: terraform.tfstate

The json I wrote and imported for this release pipeline had "ensureBackend" and "backendAzureRmSubscriptionId" defined for this task but it does not appear to be showing anymore in the DevOps portal yaml viewer...

Expected behavior
Terraform init should find the correct existing subscription.

Screenshots
N/A

Pipeline Logs

2024-01-10T15:46:20.4009700Z ##[section]Starting: Terraform : init
2024-01-10T15:46:20.4014826Z ==============================================================================
2024-01-10T15:46:20.4014983Z Task         : Terraform
2024-01-10T15:46:20.4015042Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2024-01-10T15:46:20.4015186Z Version      : 4.227.24
2024-01-10T15:46:20.4015245Z Author       : Microsoft Corporation
2024-01-10T15:46:20.4015331Z Help         : [Learn more about this task](https://aka.ms/AAf0uqr)
2024-01-10T15:46:20.4015420Z ==============================================================================
2024-01-10T15:46:21.2373862Z [command]/opt/hostedtoolcache/terraform/1.6.6/x64/terraform init -upgrade -backend-config=storage_account_name=redactedterraformstorage -backend-config=container_name=tfstate -backend-config=key=terraform.tfstate -backend-config=resource_group_name=rg-TerraformMgmt-Store-prod-001 -backend-config=tenant_id=3f9427d9-xxxx-xxxx-xxxx-xxxxxxxxxxxx -backend-config=client_id=*** -backend-config=oidc_token=*** -backend-config=use_oidc=true
2024-01-10T15:46:21.2782288Z 
2024-01-10T15:46:21.2785116Z �[0m�[1mInitializing the backend...�[0m
2024-01-10T15:46:21.2822122Z �[31m�[31m╷�[0m�[0m
2024-01-10T15:46:21.2823037Z �[31m│�[0m �[0m�[1m�[31mError: �[0m�[0m�[1mFailed to get existing workspaces: Error retrieving keys for Storage Account "redactedterraformstorage": storage.AccountsClient#ListKeys: Invalid input: autorest/validation: validation failed: parameter=client.SubscriptionID constraint=MinLength value="" details: value length must be greater than or equal to 1�[0m
2024-01-10T15:46:21.2823827Z �[31m│�[0m �[0m
2024-01-10T15:46:21.2824087Z �[31m│�[0m �[0m�[0m
2024-01-10T15:46:21.2824340Z �[31m╵�[0m�[0m
2024-01-10T15:46:21.2824574Z �[0m�[0m
2024-01-10T15:46:21.2915737Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/1.6.6/x64/terraform' failed with exit code 1
2024-01-10T15:46:21.2963068Z ##[section]Finishing: Terraform : init

Agent Configuration

• OS: Ubuntu lastest
• Hosted
• Terraform version used: 3.86.0
• AzureCLI version used: Default

Additional context
When we use a subscription-scoped service connection, it appears to work fine. Once we switch to a mgmt group scoped service connection, it cannot find a subscription ID according to the error log. We have included subscription IDs in the main.tf under each provider.

[captainhook]

I think I just realised I am using the extension by Microsoft DevLabs... this is what happens when you're glued to an issue for too long without breaks.