/aro-private

The goal is to secure ARO cluster by routing Egress traffic through an Azure Firewall

Primary LanguageShell

Private ARO cluster setup

The goal is to secure ARO cluster by routing Egress traffic through an Azure Firewall

Before:

Before

After:

After

Create network for an ARO cluster

Update/install the aro extension

Not necessary
az extension add -n aro --index https://az.aroapp.io/stable
az extension update -n aro

Set the variables

LOCATION=westeurope
RESOURCEGROUP=aro-v4-private
CLUSTER=aroprivate

Create a resource group

az group create -g "$RESOURCEGROUP" -l $LOCATION

Create the virtual network

az network vnet create \
  -g "$RESOURCEGROUP" \
  -n vnet \
  --address-prefixes 10.0.0.0/8

Add two empty subnets to your virtual network

  az network vnet subnet create \
    -g "$RESOURCEGROUP" \
    --vnet-name vnet \
    -n "$CLUSTER-master" \
    --address-prefixes 10.10.1.0/24 \
    --service-endpoints Microsoft.ContainerRegistry

  az network vnet subnet create \
    -g "$RESOURCEGROUP" \
    --vnet-name vnet \
    -n "$CLUSTER-worker" \
    --address-prefixes 10.20.1.0/24 \
    --service-endpoints Microsoft.ContainerRegistry

Disable network policies for Private Link Service on your virtual network and subnets. This is a requirement for the ARO service to access and manage the cluster.

az network vnet subnet update \
  -g "$RESOURCEGROUP" \
  --vnet-name vnet \
  -n "$CLUSTER-master" \
  --disable-private-link-service-network-policies true

Create a Firewall Subnet

az network vnet subnet create \
    -g "$RESOURCEGROUP" \
    --vnet-name vnet \
    -n "AzureFirewallSubnet" \
    --address-prefixes 10.100.1.0/26

Create a jump-host VM

Create a jump-subnet

  az network vnet subnet create \
    -g "$RESOURCEGROUP" \
    --vnet-name vnet \
    -n "jumphost" \
    --address-prefixes 10.30.1.0/24 \
    --service-endpoints Microsoft.ContainerRegistry

Create a jump-host VM

VMUSERNAME=aroadmin

az vm create --name ubuntu-jump \
             --resource-group $RESOURCEGROUP \
             --ssh-key-values ~/.ssh/id_rsa.pub \
             --admin-username $VMUSERNAME \
             --image UbuntuLTS \
             --subnet "jumphost" \
             --public-ip-address jumphost-ip \
             --vnet-name vnet

Create an Azure Red Hat OpenShift cluster

Get a pull-secret value from Red Hat Customer Portal and store it in a variables:

PULL_SECRET='<your_pull_secret>'
az aro create \
  -g "$RESOURCEGROUP" \
  -n "$CLUSTER" \
  --vnet vnet \
  --master-subnet "$CLUSTER-master" \
  --worker-subnet "$CLUSTER-worker" \
  --apiserver-visibility Private \
  --ingress-visibility Private \
  --pull-secret $PULL_SECRET

Create an Azure Firewall

Create a public IP Address

az network public-ip create -g $RESOURCEGROUP -n fw-ip --sku "Standard" --location $LOCATION

Update install Azure Firewall extension

az extension add -n azure-firewall
az extension update -n azure-firewall

Create Azure Firewall and configure IP Config

az network firewall create -g $RESOURCEGROUP -n aro-private -l $LOCATION
az network firewall ip-config create -g $RESOURCEGROUP -f aro-private -n fw-config --public-ip-address fw-ip --vnet-name vnet

Capture Azure Firewall IPs for a later use

FWPUBLIC_IP=$(az network public-ip show -g $RESOURCEGROUP -n fw-ip --query "ipAddress" -o tsv)
FWPRIVATE_IP=$(az network firewall show -g $RESOURCEGROUP -n aro-private --query "ipConfigurations[0].privateIpAddress" -o tsv)

echo $FWPUBLIC_IP
echo $FWPRIVATE_IP

Create a UDR and Routing Table for Azure Firewall

az network route-table create -g $RESOURCEGROUP --name aro-udr

az network route-table route create -g $RESOURCEGROUP --name aro-udr --route-table-name aro-udr --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP

Add Application Rules for Azure Firewall

Rule for OpenShift to work based on this list:

az network firewall application-rule create -g $RESOURCEGROUP -f aro-private \
 --collection-name 'OpenShift' \
 --action allow \
 --priority 100 \
 -n 'required' \
 --source-addresses '*' \
 --protocols 'http=80' 'https=443' \
 --target-fqdns 'registry.redhat.io' '*.quay.io' 'sso.redhat.com' 'management.azure.com' 'mirror.openshift.com' 'api.openshift.com' 'quay.io' '*.blob.core.windows.net' 'gcs.prod.monitoring.core.windows.net' 'registry.access.redhat.com' 'login.microsoftonline.com' '*.servicebus.windows.net' '*.table.core.windows.net' 'grafana.com'

Optional rules for Docker images:

az network firewall application-rule create -g $RESOURCEGROUP -f aro-private \
 --collection-name 'Docker' \
 --action allow \
 --priority 200 \
 -n 'docker' \
 --source-addresses '*' \
 --protocols 'http=80' 'https=443' \
 --target-fqdns '*cloudflare.docker.com' '*registry-1.docker.io' 'apt.dockerproject.org' 'auth.docker.io'

Associate ARO Subnets to FW

az network vnet subnet update -g $RESOURCEGROUP --vnet-name vnet --name "$CLUSTER-master" --route-table aro-udr
az network vnet subnet update -g $RESOURCEGROUP --vnet-name vnet --name "$CLUSTER-worker" --route-table aro-udr

Test the configuration

These steps works only if you added rules for Docker images.

Configure the jumpbox

Log into a jumpbox VM and install azure-cli, oc-cli, and jq utils. For the installation of openshift-cli check the Red Hat customer portal.

#Install Azure-cli
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
#Install jq
sudo apt install jq -y
#Install aro extension
az extension add -n aro --index https://az.aroapp.io/stable

Log into the ARO cluster

List cluster credentials:

# Initialize variables
LOCATION=eastus
RESOURCEGROUP=aro-v4-private
CLUSTER=aroprivate

# Login to Azure
az login
#Get the cluster credentials
ARO_PASSWORD=$(az aro list-credentials -n $CLUSTER -g $RESOURCEGROUP -o json | jq -r '.kubeadminPassword')
ARO_USERNAME=$(az aro list-credentials -n $CLUSTER -g $RESOURCEGROUP -o json | jq -r '.kubeadminUsername')

Get an API server endpoint:

ARO_URL=$(az aro show -n $CLUSTER -g $RESOURCEGROUP -o json | jq -r '.apiserverProfile.url')

Log in using oc login:

oc login $ARO_URL -u $ARO_USERNAME -p $ARO_PASSWORD

Run CentOS to test outside connectivity

Create a pod

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: centos
spec:
  containers:
  - name: centos
    image: centos
    ports:
    - containerPort: 80
    command:
    - sleep
    - "3600"
EOF

Once the Pod is Running exec into it and test outside connectivity

oc exec -it centos -- /bin/bash
curl microsoft.com

Run a test app and expose it internally via Route

Create a welcome-app deployment:

oc apply -f https://raw.githubusercontent.com/akamenev/aro-private/master/welcome-app-deployment.yaml

Create a welcome-app service with a ClusterIP:

oc apply -f https://raw.githubusercontent.com/akamenev/aro-private/master/welcome-app-clusterip-svc.yaml

Expose a welcome-app service via route:

oc expose service welcome-app

Check the service

APP_ROUTE=$(oc get route welcome-app -o json | jq -r .spec.host)
curl $APP_ROUTE

Expose the app externally with an Internal Load Balancer and Azure Firewall

Create a welcome-app service with an Internal LB:

oc apply -f https://raw.githubusercontent.com/akamenev/aro-private/master/welcome-app-internallb-svc.yaml

Get the internal IP of a service:

INTERNAL_APP_IP=$(oc get svc welcome-app-internal -o json | jq -r '.status.loadBalancer.ingress[0].ip')

Configure a DNAT rule in Azure Firewall:

# Get a Firewall's public ip
FWPUBLIC_IP=$(az network public-ip show -g $RESOURCEGROUP -n fw-ip --query "ipAddress" -o tsv)
echo $FWPUBLIC_IP

# Add a firewall extension
az extension add -n azure-firewall

# Configure a rule
az network firewall nat-rule create -g $RESOURCEGROUP -f aro-private \
 --collection-name 'welcome-app-nat' \
 --priority 100 \
 --action 'Dnat' \
 -n 'welcome-app' \
 --source-addresses '*' \
 --destination-address $FWPUBLIC_IP \
 --destination-ports '80' \
 --translated-address $INTERNAL_APP_IP \
 --translated-port '80' \
 --protocols 'TCP'

Open Firewall's public IP in a browser or curl it:

curl $FWPUBLIC_IP