/azugsweden-dapr

Primary LanguageC#MIT LicenseMIT

Azure User Group Sweden Dapr

Steps

The workshop is build around five steps.

  1. Two C# services with one dapr with http call and docker compose.
  2. Setup of Kubernetes cluster, Azure Container Apps and deployment.
  3. Splitting solution into two projects, containeraizing them and adding DAPR runtime.
  4. Adding DAPR pub/sub component and connecting service via topic.
  5. Storage component for DAPR via Azure Table Storage.
  6. Adding secrets and observability

Prerequisites

Good mood :).

  1. Visual Studio or Visual Studio Code with .NET Framework 3.1.
  2. Docker Desktop to run the containerized application locally. https://www.docker.com/products/docker-desktop
  3. DAPR CLI installed on a local machine. https://docs.dapr.io/getting-started/install-dapr-cli/
  4. Kompose tool for Kubernetes manifest generation (optional). https://kompose.io/getting-started/
  5. AZ CLI tools installation(for cloud deployment) https://aka.ms/installazurecliwindows
  6. Azure subscription, if you want to deploy applications to Kubernetes(AKS). https://azure.microsoft.com/en-us/free/
  7. Kubectl installation https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/#install-kubectl-binary-with-curl-on-windows
  8. Good mood :)

Step 0. Azure infrastructure

Script below should be run via Azure Portal bash console. You will receive database connection strings with setx command as output of this script. Please add a correct name of your subscription to the first row of the script.

subscriptionID=$(az account list --query "[?contains(name,'Microsoft')].[id]" -o tsv)
echo "Test subscription ID is = " $subscriptionID
az account set --subscription $subscriptionID
az account show

location=northeurope
postfix=$RANDOM

#----------------------------------------------------------------------------------
# Database infrastructure
#----------------------------------------------------------------------------------

export dbResourceGroup=ms-action-dapr-data$postfix
export dbServername=ms-action-dapr$postfix
export dbPoolname=dbpool
export dbAdminlogin=FancyUser3
export dbAdminpassword=Sup3rStr0ng52$postfix
export dbPaperName=paperorders
export dbDeliveryName=deliveries

az group create --name $dbResourceGroup --location $location

az sql server create --resource-group $dbResourceGroup --name $dbServername --location $location \
--admin-user $dbAdminlogin --admin-password $dbAdminpassword
	
az sql elastic-pool create --resource-group $dbResourceGroup --server $dbServername --name $dbPoolname \
--edition Standard --dtu 50 --zone-redundant false --db-dtu-max 50

az sql db create --resource-group $dbResourceGroup --server $dbServername --elastic-pool $dbPoolname \
--name $dbPaperName --catalog-collation SQL_Latin1_General_CP1_CI_AS
	
az sql db create --resource-group $dbResourceGroup --server $dbServername --elastic-pool $dbPoolname \
--name $dbDeliveryName --catalog-collation SQL_Latin1_General_CP1_CI_AS	

sqlClientType=ado.net

SqlPaperString=$(az sql db show-connection-string --name $dbPaperName --server $dbServername --client $sqlClientType --output tsv)
SqlPaperString=${SqlPaperString/Password=<password>;}
SqlPaperString=${SqlPaperString/<username>/$dbAdminlogin}

SqlDeliveryString=$(az sql db show-connection-string --name $dbDeliveryName --server $dbServername --client $sqlClientType --output tsv)
SqlDeliveryString=${SqlDeliveryString/Password=<password>;}
SqlDeliveryString=${SqlDeliveryString/<username>/$dbAdminlogin}

SqlPaperPassword=$dbAdminpassword

#----------------------------------------------------------------------------------
# AKS infrastructure
#----------------------------------------------------------------------------------

location=northeurope
groupName=ms-action-dapr-cluster$postfix
clusterName=msaction-cluster$postfix
registryName=msactionregistry$postfix
accountSku=Standard_LRS
accountName=msactionstorage$postfix
queueName=msactionqueue
queueResultsName=msactionqueueresults

az group create --name $groupName --location $location

az acr create --resource-group $groupName --name $registryName --sku Standard
az acr identity assign --identities [system] --name $registryName

az aks create --resource-group $groupName --name $clusterName --node-count 3 --generate-ssh-keys --network-plugin azure
az aks update --resource-group $groupName --name $clusterName --attach-acr $registryName

#----------------------------------------------------------------------------------
# Service bus queue and application insights
#----------------------------------------------------------------------------------

groupName=ms-action-dapr-extras$postfix
location=northeurope
az group create --name $groupName --location $location
namespaceName=msActionDapr$postfix
queueName=createdelivery

az servicebus namespace create --resource-group $groupName --name $namespaceName --location $location
az servicebus queue create --resource-group $groupName --name $queueName --namespace-name $namespaceName
serviceBusString=$(az servicebus namespace authorization-rule keys list --resource-group $groupName --namespace-name $namespaceName --name RootManageSharedAccessKey --query primaryConnectionString --output tsv)

insightsName=msactiondaprlogs$postfix
az monitor app-insights component create --resource-group $groupName --app $insightsName --location $location --kind web --application-type web --retention-time 120

instrumentationKey=$(az monitor app-insights component show --resource-group $groupName --app $insightsName --query  "instrumentationKey" --output tsv)

#----------------------------------------------------------------------------------
# Azure function app with storage account
#----------------------------------------------------------------------------------

accountSku=Standard_LRS
accountName=msactionstorage$postfix

az storage account create --name $accountName --location $location --kind StorageV2 \
--resource-group $groupName --sku $accountSku --access-tier Hot  --https-only true

accountKey=$(az storage account keys list --resource-group $groupName --account-name $accountName --query "[0].value" | tr -d '"')

accountConnString="DefaultEndpointsProtocol=https;AccountName=$accountName;AccountKey=$accountKey;EndpointSuffix=core.windows.net"

applicationName=msactiondaprfunc$postfix
echo "applicationName  = " $applicationName

az functionapp create --resource-group $groupName \
--name $applicationName --storage-account $accountName \
--consumption-plan-location $location --functions-version 3

az functionapp update --resource-group $groupName --name $applicationName --set dailyMemoryTimeQuota=400000
az functionapp config appsettings set --resource-group $groupName --name $applicationName --settings "MSDEPLOY_RENAME_LOCKED_FILES=1"
az functionapp config appsettings set --resource-group $groupName --name $applicationName --settings ASPNETCORE_ENVIRONMENT=Production
az functionapp config appsettings set --resource-group $groupName --name $applicationName --settings "StorageConnectionString=$accountConnString"

keyvaultName=msActionDapr$postfix
principalName=vaultadmin
principalCertName=vaultadmincert

az keyvault create --resource-group $groupName --name $keyvaultName --location $location
az keyvault secret set --name SqlPaperPassword --vault-name $keyvaultName --value $SqlPaperPassword

az ad sp create-for-rbac --name $principalName --create-cert --cert $principalCertName --keyvault $keyvaultName --skip-assignment --years 3

# get appId from output of this step and use commented code below to grant access.

# az ad sp show --id 88511b82-8ced-4ba3-bd9b-0599f479e870
# get objectId from command output above and set it to command below 

# az keyvault set-policy --name $keyvaultName --object-id b3535a27-26f0-4c59-a50a-bd13886e4185 --secret-permissions get

#----------------------------------------------------------------------------------
# Azure Container Apps
#----------------------------------------------------------------------------------

az extension add \
  --source https://workerappscliextension.blob.core.windows.net/azure-cli-extension/containerapp-0.2.2-py2.py3-none-any.whl
  
az provider register --namespace Microsoft.Web

acaGroupName=ms-action-containerapp$postfix
location=northeurope
logAnalyticsWorkspace=container-apps-logs$postfix
containerAppsEnv=shared-environment$postfix

az group create --name $acaGroupName --location $location

az monitor log-analytics workspace create \
--resource-group $acaGroupName --workspace-name $logAnalyticsWorkspace

logAnalyticsWorkspaceClientId=`az monitor log-analytics workspace show --query customerId -g $acaGroupName -n $logAnalyticsWorkspace -o tsv | tr -d '[:space:]'`

logAnalyticsWorkspaceClientSecret=`az monitor log-analytics workspace get-shared-keys --query primarySharedKey -g $acaGroupName -n $logAnalyticsWorkspace -o tsv | tr -d '[:space:]'`

az containerapp env create \
--name $containerAppsEnv \
--resource-group $acaGroupName \
--logs-workspace-id $logAnalyticsWorkspaceClientId \
--logs-workspace-key $logAnalyticsWorkspaceClientSecret \
--instrumentation-key $instrumentationKey \
--location $location

#----------------------------------------------------------------------------------
# SQL connection strings
#----------------------------------------------------------------------------------

printf "\n\nRun string below in local cmd prompt to assign secret to environment variable SqlPaperString:\nsetx SqlPaperString \"$SqlPaperString\"\n\n"
printf "\n\nRun string below in local cmd prompt to assign secret to environment variable SqlDeliveryString:\nsetx SqlDeliveryString \"$SqlDeliveryString\"\n\n"
printf "\n\nRun string below in local cmd prompt to assign secret to environment variable SqlPaperPassword:\nsetx SqlPaperPassword \"$SqlPaperPassword\"\n\n"
printf "\n\nRun string below in local cmd prompt to assign secret to environment variable SqlDeliveryPassword:\nsetx SqlDeliveryPassword \"$SqlPaperPassword\"\n\n"
printf "\n\nRun string below in local cmd prompt to assign secret to environment variable AzureWebJobsStorage:\nsetx AzureWebJobsStorage \"$accountConnString\"\n\n"
printf "\n\nRun string below in local cmd prompt to assign secret to environment variable ServiceBusString:\nsetx ServiceBusString \"$serviceBusString\"\n\n"

echo "Update open-telemetry-collector-appinsights.yaml in Step 4 End => <INSTRUMENTATION-KEY> value with:  " $instrumentationKey

Step 1. Two projects, docker compose and DAPR initialization.

We adding containerization via Visual Studio tooling and manually adding DAPR sidecar configuration for each server.

Start folder contains solution with two projects. Along with Dockefile generated by visual studio and updates for code. Including the new reference for localhost container url - host.docker.internal and environment variable file. Don't forget to add env file with a secrets content along with changes to docker-compose.yaml in the root folder.

Solution will work with two containers, so there is a need to put the correct container port for Delivery service.

!! Be aware, if you have docker build exceptions in Visual studio with errors related to the File system, there is a need to configure docker desktop. Open Docker desktop => configuration => Resources => File sharing => Add your project folder or entire drive, C:\ for example. Dont forget to remove drive setting later on.

End folder contains solution with DAPR, service invocation via HTTP and docker compose files with sidecar.

Lets start with right clicking on each solution and adding orchestration with Container orchestration via Docker compose. Visual studio will generate docker compose files for you.

Step 2. Application deployment to Azure Kubernetes service.

We will create AKS manifests for our services and add DAPR sections. Deploy dapr to AKS cluster and add containers to the private repository.

Start folder contains solution with local env variables added to docker compose. At this point we will enable database communication with our AKS cluster and setup connection from local machine to private container registry and kubernetes cluster.

End folder contains solution with Kubernetes manifests ready for deployment, secrets included right into manifests to simplify flow.

You will need an Azure subscription ID.

Lets start with CMD.

az login
az account set --subscription 95cd9078f8c
az account show
az acr login --name msactionregistry
az aks get-credentials --resource-group msaction-cluster --name msaction-cluster --overwrite-existing
kubectl config use-context msaction-cluster
kubectl get all

And initialize DAPR.

dapr init -k

and validate it with

dapr status -k

Then we will need to build our solution in release mode and observe results with command. You can start docker desktop application for GUI container handling.

docker images

Lets tag our newly built container with azure container registry name and version.

docker tag tpaperorders:latest msactionregistry.azurecr.io/tpaperorders:v1
docker tag tpaperdelivery:latest msactionregistry.azurecr.io/tpaperdelivery:v1

Check results with

docker images

And push images to container registry

docker push msactionregistry.azurecr.io/tpaperorders:v1
docker push msactionregistry.azurecr.io/tpaperdelivery:v1

There is need to change version of container in YAML manifest files inside Step 3 End directory, and change this files each time you preparing a new version of container.

    spec:
      containers:
        - name: tpaperorders
          image: msactionregistry.azurecr.io/tpaperorders:v1
          imagePullPolicy: Always
          ports:
            - containerPort: 80
              protocol: TCP

Now we need to pray the "demo gods" for our deployment and run commands below

kubectl apply -f tpaperorders-deploy.yaml
kubectl apply -f tpaperdelivery-deploy.yaml

And then check results with

kubectl get all

You can use set of commands below for quick container/publish re-deployments. Just change version in kubernetes manifest and commands below.

docker tag tpaperorders:latest msactionregistry.azurecr.io/tpaperorders:v2
docker images
docker push msactionregistry.azurecr.io/tpaperorders:v2
kubectl apply -f tpaperorders-deploy.yaml
kubectl get all

docker tag tpaperdelivery:latest msactionregistry.azurecr.io/tpaperdelivery:v2
docker images
docker push msactionregistry.azurecr.io/tpaperdelivery:v2
kubectl apply -f tpaperdelivery-deploy.yaml
kubectl get all

We cam observe our deployment with get all command and checking of external public endpoints(public load balancer endpoints).

20.67.14.15/api/order/create/1
20.67.15.202/api/deliveries/get

In case of the problems we need to investigate logs via command prompt.

kubectl logs tpaperdelivery-8c4bdc475-j89kx daprd
kubectl logs tpaperdelivery-8c4bdc475-j89kx tpaperdelivery

Step 3. Introduction to the DAPR pubsub.

We will deploy DAPR pubsub component to Azure. Make changes to our code and take a look into the pod logs to see whats happening.

Start folder contains all needed files for this step.

We need to deploy pubsub component and RabbitMQ broker with

kubectl apply -f rabbitmq.yaml
kubectl apply -f pubsub-rabbitmq.yaml

Then we updating C# code and DAPR service manifest files to container v2 and building solution in Visual Studio.

docker tag tpaperorders:latest msactionregistry.azurecr.io/tpaperorders:v2
docker tag tpaperdelivery:latest msactionregistry.azurecr.io/tpaperdelivery:v2

docker push msactionregistry.azurecr.io/tpaperorders:v2
docker push msactionregistry.azurecr.io/tpaperdelivery:v2

And then deployment via service manifests.

kubectl apply -f rabbitmq.yaml
kubectl apply -f pubsub-rabbitmq.yaml

And testing results with slightly updated endpoints

20.67.14.15/api/order/create/1
20.67.15.202/api/deliveries/get

We will need following commands to get logs from AKS cluster. You should get correct pod names from get all command and change log command accordingly.

kubectl get all

kubectl logs tpaperdelivery-599b8cd4b7-8nxzz daprd
kubectl logs tpaperdelivery-599b8cd4b7-8nxzz tpaperdelivery

In the folder END we have additional file for Application insight integration.

Check out the file open-telemetry-collector-appinsights.yaml and replace the placeholder with your Application Insights Instrumentation Key. Apply the configuration with

kubectl apply -f open-telemetry-collector-appinsights.yaml

Open collector-config.yaml file and check its content Apply the configuration with

kubectl apply -f collector-config.yaml

Update services manifestst with following code and update container version to the new version.

        dapr.io/log-level: debug
        dapr.io/config: "appconfig"

Rebuild solution in visual studio and deploy new container versions.

docker tag tpaperorders:latest msactionregistry.azurecr.io/tpaperorders:v4
docker tag tpaperdelivery:latest msactionregistry.azurecr.io/tpaperdelivery:v4

docker push msactionregistry.azurecr.io/tpaperorders:v4
docker push msactionregistry.azurecr.io/tpaperdelivery:v4

kubectl apply -f tpaperorders-deploy.yaml
kubectl apply -f tpaperdelivery-deploy.yaml

Step 4. DAPR storage component.

And a little bit of debugging with logs and overall applicationm logging.

Step 5. Secrets via Azure KeyVault and Dapr State component.

  • We created an Azure Key Vault with our infrastructure beforehand. But steps below included just in case.
az keyvault create --resource-group $groupName --name $keyvaultName --location $location
  • Create a service principal

Create a service principal with a new certificate and store the 3-year certificate inside [your keyvault]'s certificate vault.

Note you can skip this step if you want to use an existing service principal for keyvault instead of creating new one

az ad sp create-for-rbac --name $principalName --create-cert --cert $principalCertName --keyvault $keyvaultName --skip-assignment --years 3

{
  "appId": "88511b82-8ced-4ba3-bd9b-0599f479e870",
  "displayName": "vaultadmin",
  "name": "88511b82-8ced-4ba3-bd9b-0599f479e870",
  "password": null,
  "tenant": "53e93ede-ec5b-4d7a-8376-48e080d23e88"
}

Save the both the appId and tenant from the output which will be used in the next step

  • Get the Object Id for [your_service_principal_name]
az ad sp show --id 88511b82-8ced-4ba3-bd9b-0599f479e870

{
    ...
  "objectId": "b3535a27-26f0-4c59-a50a-bd13886e4185",
  "objectType": "ServicePrincipal",
    ...
}
  • Grant the service principal the GET permission to your Azure Key Vault
az keyvault set-policy --name $keyvaultName --object-id b3535a27-26f0-4c59-a50a-bd13886e4185 --secret-permissions get

Get the autogenerated password for Kubernetes secret

az ad app credential reset \
--id "d5fc07b8-b6b8-4d24-a966-548caad59a74" --years 2 --password $(openssl rand -base64 30)

And set password for value of vaultsec3key

kubectl create secret generic vaultsec3name --from-literal=vaultsec3key="fewfpUDdVDV"
  1. Create azurekeyvault-deploy.yaml component file

The component yaml refers to the Kubernetes secretstore using auth property and secretKeyRef refers to the certificate stored in Kubernetes secret store.

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: azurekeyvault
  namespace: default
spec:
  type: secretstores.azure.keyvault
  version: v1
  metadata:
  - name: vaultName
    value: "msActionDapr"
  - name: azureTenantId
    value: "53e93ede-ec5b-4d7a-8376-48e080d23e88"
  - name: azureClientId
    value: "d5fc07b8-b6b8-4d24-a966-548caad59a74"
  - name: azureClientSecret
    secretKeyRef:
      name: "vaultsec3name"
      key: "vaultsec3key"
auth:
  secretStore: kubernetes
  1. Apply azurekeyvault.yaml component
kubectl apply -f azurekeyvault-deploy.yaml
  1. We already stored SQL password in KeyVault, but for clarification.
keyvaultName=msActionDapr
az keyvault secret set --vault-name $keyvaultName --name SuperSecret --value "TopSecretValue"

We not changing service component

Make sure that secretstores.azure.keyvault is loaded successfully in daprd sidecar log

Step 5. Azure Container apps introduction.

registryPassword="okDW9rjpguB=BH3y3fl1KcPJqQD0U4jF"
echo $registryPassword

az containerapp create \
--name tamopsapp \
--resource-group $RESOURCE_GROUP \
--environment $CONTAINERAPPS_ENVIRONMENT \
--image msactionregistry.azurecr.io/sampleapp:latest \
--min-replicas 2 \
--max-replicas 2 \
--registry-login-server msactionregistry.azurecr.io \
--registry-username msactionregistry \
--registry-password $registryPassword \
--target-port 80 \
--ingress 'external'

Useful commands and notes.

You might need to delete all deployments

kubectl get deployments

kubectl delete deployments tpaperdelivery
kubectl delete deployments tpaperorders

kubectl delete svc tpaperorders
kubectl delete svc tpaperdelivery

If you want to purge containers from Azure container registry

az acr repository delete --name msactionregistry --repository tpaperdelivery
az acr repository delete --name msactionregistry --repository tpaperorders

To cleanup local docker images via cmd. It is recommended to do after each step.

for /F %i in ('docker images -a -q') do docker rmi -f %i