Azure/azure-iot-ops-cli-extension

[bug] The command "az iot ops init" fails if the service principal running the command has access to Graph API https://graph.microsoft.com/v1.0/applications/{id}

flecoqui opened this issue · 3 comments

flecoqui commented

Describe the bug
I wanted to test the deployment of Azure IoT Operations using a service principal with all the permissions required to deploy Azure IoT Operations with the command "az iot ops init".
I created a service principal/application with a permission to use Graph API "Application.ReadWrite.All"
When the deployment script called "az iot ops init", it failed returning the error below:

ERROR: cli.azure.cli.core.azclierror: Not Found({"error":{"code":"Request_ResourceNotFound","message":"Resource '2de8b284-5ac5-4367-9723-a6728afab99c' does not exist or one of its queried reference-property objects are not present.","innerError":{"date":"2024-03-12T11:50:58","request-id":"99445284-9e99-43d1-99d6-301acfa26910","client-request-id":"99445284-9e99-43d1-99d6-301acfa26910"}}})

To Reproduce
Steps to reproduce the behavior:

I created the service principal using the following commands:

spName="spaio"
AZURE_SUBSCRIPTION_ID="TO_BE_COMPLETED"

# Create the application 
appId=$(az ad app create  --display-name "${spName}"  --only-show-errors | jq -r ".appId")
if [ -n "${appId}" ] && [ "${appId}" != 'null' ] ; then

# Create the service principal
spjson=$(az ad sp create-for-rbac --name '${spName}' --role owner --scopes /subscriptions/${AZURE_SUBSCRIPTION_ID}  --json-auth --only-show-errors)
spId=$(echo "$spjson" | jq -r .clientId)
spPassword=$(echo "$spjson" | jq -r .clientSecret)

# Create the custom role
cmd="az role definition create --role-definition '{ \
    \"Name\": \"IoT Operation Contributor\", \
    \"IsCustom\": true, \
    \"Description\": \"IoT Operation Contributor\", \
    \"Actions\": [ \
        \"Microsoft.IoTOperationsOrchestrator/register/action\", \
        \"Microsoft.IoTOperationsOrchestrator/register/action\", \
       \"Microsoft.IoTOperationsMQ/register/action\", \
        \"Microsoft.IoTOperationsDataProcessor/register/action\",  \
        \"Microsoft.DeviceRegistry/register/action\", \
        \"Microsoft.AzureStackHCI/register/action\", \
        \"Microsoft.AzureStackHCI/Unregister/Action\", \
        \"Microsoft.AzureStackHCI/clusters/*\", \
        \"Microsoft.HybridCompute/register/action\", \
        \"Microsoft.GuestConfiguration/register/action\", \
        \"Microsoft.GuestConfiguration/guestConfigurationAssignments/read\", \
        \"Microsoft.Resources/subscriptions/resourceGroups/write\", \
        \"Microsoft.Resources/subscriptions/resourceGroups/delete\", \
        \"Microsoft.HybridConnectivity/register/action\", \
        \"Microsoft.Authorization/roleAssignments/write\", \
        \"Microsoft.Authorization/roleAssignments/delete\", \
        \"Microsoft.Authorization/*/read\", \
        \"Microsoft.Resources/deployments/*\", \
        \"Microsoft.Resources/subscriptions/resourceGroups/read\", \
        \"Microsoft.Resources/subscriptions/read\", \
        \"Microsoft.Management/managementGroups/read\", \
        \"Microsoft.Support/*\", \
        \"Microsoft.AzureStackHCI/*\", \
        \"Microsoft.Insights/AlertRules/Write\", \
        \"Microsoft.Insights/AlertRules/Delete\", \
        \"Microsoft.Insights/AlertRules/Read\", \
        \"Microsoft.Insights/AlertRules/Activated/Action\", \
        \"Microsoft.Insights/AlertRules/Resolved/Action\", \
        \"Microsoft.Insights/AlertRules/Throttled/Action\", \
        \"Microsoft.Insights/AlertRules/Incidents/Read\", \
        \"Microsoft.Resources/subscriptions/resourcegroups/deployments/read\", \
        \"Microsoft.Resources/subscriptions/resourcegroups/deployments/write\", \
        \"Microsoft.Resources/subscriptions/resourcegroups/deployments/operations/read\", \
        \"Microsoft.Resources/subscriptions/resourcegroups/deployments/operationstatuses/read\", \
        \"Microsoft.ResourceHealth/availabilityStatuses/read\", \
        \"Microsoft.Resources/subscriptions/read\", \
        \"Microsoft.Resources/subscriptions/operationresults/read\", \
        \"Microsoft.HybridCompute/machines/read\", \
        \"Microsoft.HybridCompute/machines/write\", \
        \"Microsoft.HybridCompute/machines/delete\", \
        \"Microsoft.HybridCompute/machines/UpgradeExtensions/action\", \
        \"Microsoft.HybridCompute/machines/assessPatches/action\", \
        \"Microsoft.HybridCompute/machines/installPatches/action\", \
        \"Microsoft.HybridCompute/machines/extensions/read\", \
        \"Microsoft.HybridCompute/machines/extensions/write\", \
        \"Microsoft.HybridCompute/machines/extensions/delete\", \
        \"Microsoft.HybridCompute/operations/read\", \
        \"Microsoft.HybridCompute/locations/operationresults/read\", \
        \"Microsoft.HybridCompute/locations/operationstatus/read\", \
        \"Microsoft.HybridCompute/machines/patchAssessmentResults/read\", \
        \"Microsoft.HybridCompute/machines/patchAssessmentResults/softwarePatches/read\", \
        \"Microsoft.HybridCompute/machines/patchInstallationResults/read\", \
        \"Microsoft.HybridCompute/machines/patchInstallationResults/softwarePatches/read\", \
        \"Microsoft.HybridCompute/locations/updateCenterOperationResults/read\", \
        \"Microsoft.HybridCompute/machines/hybridIdentityMetadata/read\", \
        \"Microsoft.HybridCompute/osType/agentVersions/read\", \
        \"Microsoft.HybridCompute/osType/agentVersions/latest/read\", \
        \"Microsoft.HybridCompute/machines/runcommands/read\", \
        \"Microsoft.HybridCompute/machines/runcommands/write\", \
        \"Microsoft.HybridCompute/machines/runcommands/delete\", \
        \"Microsoft.HybridCompute/machines/licenseProfiles/read\", \
        \"Microsoft.HybridCompute/machines/licenseProfiles/write\", \
        \"Microsoft.HybridCompute/machines/licenseProfiles/delete\", \
        \"Microsoft.HybridCompute/licenses/read\", \
        \"Microsoft.HybridCompute/licenses/write\", \
        \"Microsoft.HybridCompute/licenses/delete\", \
        \"Microsoft.ResourceConnector/register/action\", \
        \"Microsoft.ResourceConnector/appliances/read\", \
        \"Microsoft.ResourceConnector/appliances/write\", \
        \"Microsoft.ResourceConnector/appliances/delete\", \
        \"Microsoft.ResourceConnector/locations/operationresults/read\", \
        \"Microsoft.ResourceConnector/locations/operationsstatus/read\", \
        \"Microsoft.ResourceConnector/appliances/listClusterUserCredential/action\", \
        \"Microsoft.ResourceConnector/appliances/listKeys/action\", \
        \"Microsoft.ResourceConnector/operations/read\", \
        \"Microsoft.ExtendedLocation/register/action\", \
        \"Microsoft.ExtendedLocation/customLocations/read\", \
        \"Microsoft.ExtendedLocation/customLocations/deploy/action\", \
        \"Microsoft.ExtendedLocation/customLocations/write\", \
        \"Microsoft.ExtendedLocation/customLocations/delete\", \
        \"Microsoft.ExtendedLocation/customLocations/resourceSyncRules/write\", \
        \"Microsoft.IoTOperationsDataProcessor/instances/write\", \
        \"Microsoft.IoTOperationsMQ/mq/write\", \
        \"Microsoft.IoTOperationsMQ/mq/broker/write\", \
        \"Microsoft.IoTOperationsMQ/mq/diagnosticService/write\", \
        \"Microsoft.IoTOperationsMQ/mq/broker/listener/write\", \
        \"Microsoft.IoTOperationsMQ/mq/broker/authentication/write\", \
        \"Microsoft.IoTOperationsOrchestrator/targets/write\", \
        \"Microsoft.OperationalInsights/workspaces/read\", \
        \"Microsoft.Insights/dataCollectionRules/read\", \
        \"Microsoft.Insights/dataCollectionRules/write\", \
        \"Microsoft.OperationalInsights/workspaces/sharedKeys/action\", \
        \"Microsoft.Insights/dataCollectionRuleAssociations/write\", \
        \"Microsoft.Kubernetes/connectedClusters/listClusterUserCredential/action\", \
        \"Microsoft.EdgeMarketplace/offers/read\", \
        \"Microsoft.EdgeMarketplace/publishers/read\", \
        \"Microsoft.Kubernetes/register/action\", \
        \"Microsoft.KubernetesConfiguration/register/action\", \
        \"Microsoft.KubernetesConfiguration/extensions/write\", \
        \"Microsoft.KubernetesConfiguration/extensions/read\", \
        \"Microsoft.KubernetesConfiguration/extensions/delete\", \
        \"Microsoft.KubernetesConfiguration/extensions/operations/read\", \
        \"Microsoft.KubernetesConfiguration/namespaces/read\", \
        \"Microsoft.KubernetesConfiguration/operations/read\", \
        \"Microsoft.Resources/subscriptions/resourceGroups/read\", \
        \"Microsoft.AzureStackHCI/StorageContainers/Write\", \
        \"Microsoft.AzureStackHCI/StorageContainers/Read\", \
        \"Microsoft.HybridContainerService/register/action\", \
        \"Microsoft.Authorization/*/read\", \
        \"Microsoft.Insights/alertRules/*\", \
        \"Microsoft.Resources/deployments/write\", \
        \"Microsoft.Resources/subscriptions/operationresults/read\", \
        \"Microsoft.Resources/subscriptions/read\", \
        \"Microsoft.Resources/subscriptions/resourceGroups/read\", \
        \"Microsoft.Kubernetes/connectedClusters/Write\", \
        \"Microsoft.Kubernetes/connectedClusters/read\", \
        \"Microsoft.Support/*\"                  \
    ], \
    \"NotActions\": [ \
    ], \
    \"AssignableScopes\": [\"/subscriptions/${AZURE_SUBSCRIPTION_ID}\"] \
  }'"
 eval "${cmd}" 
fi
# Grant Permission to API Application.ReadWrite.All for the service principal
spObjectId=$(az ad sp show --id "$spId" --query "id" --output tsv --only-show-errors)
API_Microsoft_Graph="00000003-0000-0000-c000-000000000000"
API_Microsoft_GraphId=$(az ad sp show --id $API_Microsoft_Graph --query "id" --output tsv --only-show-errors )
PERMISSION_MG_Application_ReadWrite_All=$(az ad sp show --id $API_Microsoft_Graph --query "appRoles[?value=='Application.ReadWrite.All']"  --only-show-errors | jq -r ".[].id")
cmd="az rest --method POST --uri https://graph.microsoft.com/v1.0/servicePrincipals/${spObjectId}/appRoleAssignments --body '{\"principalId\": \"${spObjectId}\",\"resourceId\": \"$API_Microsoft_GraphId\",\"appRoleId\": \"$PERMISSION_MG_Application_ReadWrite_All\"}' "
eval "$cmd" 
# Assign role owner
cmd="az role assignment create --assignee \"$spId\"  --role \"Owner\"  --scope \"/subscriptions/${AZURE_SUBSCRIPTION_ID}\" --only-show-errors"
eval "${cmd}"
# Assign role "IoT Operation Contributor"
cmd="az role assignment create --assignee \"$spId\"  --role \"IoT Operation Contributor\"  --scope \"/subscriptions/${AZURE_SUBSCRIPTION_ID}\" --only-show-errors"
eval "${cmd}"

Using this service principal with all the required permissions I tried to deploy "Azure IoT Operations" on an Azure Arc Enabled K3S cluster running in a virtual machine with the following commands:

AZURE_TENANT_ID="TO_BE_COMPLETED"
AZURE_RESOURCE_GROUP="TO_BE_COMPLETED"
AZURE_CLUSTER="TO_BE_COMPLETED"
AZURE_KEYVAULT="TO_BE_COMPLETED"

az login --tenant "${AZURE_TENANT_ID}" --service-principal -u "$spId" --password="$spPassword"
az account set --subscription "${AZURE_SUBSCRIPTION_ID}"
az extension add --upgrade --name azure-iot-ops
spObjectId=$(az ad sp show --id "$spId" --query 'id' -o tsv)
az iot ops init --cluster "${AZURE_CLUSTER}" -g "${AZURE_RESOURCE_GROUP}" --kv-id "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${AZURE_KEYVAULT}" --sp-app-id  "$spId" --sp-object-id "${spObjectId}" --sp-secret="$spPassword"  --ca-dir "/home/azureuser/ca-dir" --kubernetes-distro k3s --disable-rsync-rules true --debug

The az iot ops init command failed:

ERROR: cli.azure.cli.core.azclierror: Not Found({"error":{"code":"Request_ResourceNotFound","message":"Resource '2de8b284-5ac5-4367-9723-a6728afab99c' does not exist or one of its queried reference-property objects are not present.","innerError":{"date":"2024-03-12T11:50:58","request-id":"99445284-9e99-43d1-99d6-301acfa26910","client-request-id":"99445284-9e99-43d1-99d6-301acfa26910"}}})

debug logs below:

```bash
An error occurred exiting from the current bash

[stderr]
__init__.py", line 701, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 334, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/home/azureuser/.azure/cliextensions/azure-iot-ops/azext_edge/edge/commands_edge.py", line 155, in init
    app_principal = logged_in_principal.fetch_self_if_app()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/azureuser/.azure/cliextensions/azure-iot-ops/azext_edge/edge/util/sp.py", line 57, in fetch_self_if_app
    raise http_error
  File "/home/azureuser/.azure/cliextensions/azure-iot-ops/azext_edge/edge/util/sp.py", line 48, in fetch_self_if_app
    result = send_raw_request(
             ^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/util.py", line 1007, in send_raw_request
    raise HTTPError(reason, r)
azure.cli.core.azclierror.HTTPError: Not Found({"error":{"code":"Request_ResourceNotFound","message":"Resource '2de8b284-5ac5-4367-9723-a6728afab99c' does not exist or one of its queried reference-property objects are not present.","innerError":{"date":"2024-03-12T11:50:58","request-id":"99445284-9e99-43d1-99d6-301acfa26910","client-request-id":"99445284-9e99-43d1-99d6-301acfa26910"}}})

ERROR: cli.azure.cli.core.azclierror: Not Found({"error":{"code":"Request_ResourceNotFound","message":"Resource '2de8b284-5ac5-4367-9723-a6728afab99c' does not exist or one of its queried reference-property objects are not present.","innerError":{"date":"2024-03-12T11:50:58","request-id":"99445284-9e99-43d1-99d6-301acfa26910","client-request-id":"99445284-9e99-43d1-99d6-301acfa26910"}}})
ERROR: az_command_data_logger: Not Found({"error":{"code":"Request_ResourceNotFound","message":"Resource '2de8b284-5ac5-4367-9723-a6728afab99c' does not exist or one of its queried reference-property objects are not present.","innerError":{"date":"2024-03-12T11:50:58","request-id":"99445284-9e99-43d1-99d6-301acfa26910","client-request-id":"99445284-9e99-43d1-99d6-301acfa26910"}}})
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7b94c7d39e40>]
INFO: az_command_data_logger: exit code: 1
INFO: cli.__main__: Command ran in 0.984 seconds (init: 0.241, invoke: 0.744)
INFO: cli.azure.cli.core.decorators: Suppress exception:
Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/azure/cli/__main__.py", line 62, in <module>
    raise ex
  File "/opt/az/lib/python3.11/site-packages/azure/cli/__main__.py", line 55, in <module>
    sys.exit(exit_code)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/decorators.py", line 79, in _wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/telemetry.py", line 125, in generate_payload
    payload = json.dumps(self.events, separators=(',', ':'))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/opt/az/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type HTTPError is not JSON serializable

INFO: telemetry.main: Split cli events and extra events failure: the JSON object must be str, bytes or bytearray, not NoneType
'. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.

Expected behavior
Normally, the command "az iot ops init" should have deployed smoothly Azure IoT Operation on the K3S cluster

Investigation results:
The issue seems in the file .\azext_edge\edge\util\sp.py in the function fetch_self_if_app

    def fetch_self_if_app(self) -> Optional[AppPrincipal]:
        if self.is_app():
            app_id = self.claims.get("appid")
            obj_id = self.claims.get("oid")
            if app_id:
                try:
                    result = send_raw_request(
                        cli_ctx=self.cli_ctx,
                        method="GET",
                        url=f"https://graph.microsoft.com/v1.0/applications/{app_id}",
                    ).json()
                    return AppPrincipal(app_id=app_id, object_id=obj_id, app=result)
                except HTTPError as http_error:
                    if http_error.response.status_code in [401, 403]:
                        return None
                    raise http_error

When I used a service principal without permission to use the Graph API "Application.ReadWrite.All", this function returns None as the call to https://graph.microsoft.com/v1.0/applications/{app_id} return 403 (no access to this endpoint. Then the command line az iot ops init fallbacks and used the input parameters --sp-app-id --sp-object-id --sp-secret for the authentication with Azure.
In our case, as Azure CLI has access to the application it returns 404 has this API should be called with application object id as parameter not the app_id (https://graph.microsoft.com/v1.0/applications/{app_object_id}).
Nevertheless, the value of obj_id in the code is the object_id of the service principal calling https://graph.microsoft.com/v1.0/applications/{obj_id} will also returns 404.

So far, I'm using this turn around which consists in patching the file .\azext_edge\edge\util\sp.py when the service principal has access to the Graph API "Application.ReadWrite.All".
Below the patch:

sed -i "s/applications\/{app_id}/applications(appId='{app_id}')/"  /home/${AZURE_VM_ADMINUSERNAME}/.azure/cliextensions/azure-iot-ops/azext_edge/edge/util/sp.py

Calling https://graph.microsoft.com/v1.0/applications(appId='{app_id}') return 200 and the Azure CLI uses the service principal credentials for the authentication with Azure.

But, if the service principal without permission to the Graph API "Application.ReadWrite.All" https://graph.microsoft.com/v1.0/applications(appId='{app_id}') it will also return 200 when the call to https://graph.microsoft.com/v1.0/applications/{app_id} will return 403.
So, when the service principal has no access to the Graph API "Application.ReadWrite.All", the patch is removed and we use the input parameters --sp-app-id --sp-object-id --sp-secret.

Environment (please complete the following information):

az --version
azure-cli 2.58.0

core 2.58.0
telemetry 1.1.0

Extensions:
azure-iot-ops 0.4.0b1
connectedk8s 1.6.6
customlocation 0.1.3
k8s-configuration 1.7.0
k8s-extension 1.6.1

Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2

Python location '/opt/az/bin/python3'
Extensions directory '/home/azureuser/.azure/cliextensions'

Python (Linux) 3.11.7 (main, Feb 29 2024, 02:08:19) [GCC 11.4.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context
Add any other context about the problem here.

digimaun commented

@flecoqui, thanks for submitting this detailed issue. Can you please try your scenario using the latest 0.4.0b2 without the ad-hoc patching?

flecoqui commented

@digimaun, thanks a lot for your responsiveness, I did run two tests:

  • one deployment with a service principal with Graph API access (northeurope)
  • one deployment with a service principal without Graph API access (eastus2)

Both deployments were successful.

  • 👍4
digimaun commented

Thank you for confirming. We'll close this issue, let us know if you run into further problems.