This LAB is for POC purpose only!
This lab is designed to set up an AKS cluster and a VM with Splunk installed. We then configure the AKS diagnostics settings (control plane and audit logs) to be sent to Splunk via an Event Hub. Finally, we configure the cluster by installing the solution collectorforkubernetes to send application metrics and logs to Splunk using an HTTP Event Collector.
- Infrastructure Provisionning
- Configuration - Control Plane Logs & Audit logs to Splunk
- Configuration - Applications Logs & Metrics to Splunk
- Create Splunk VM:
location="westeurope"
rg_name="splunk-rg"
vnet_name="vnet"
vnet_prefix="10.0.0.0/24"
subnet_name="snet-splunk"
subnet_prefix="10.0.0.0/24"
vm_name="splunk-vm"
vm_size="Standard_B8ms"
ssh_admin_user="davidsantiago"
ssh_admin_password="Microsoft=1Microsoft=1"
splunk_fqdn_prefix="demo-splunk-david"
splunk_fqdn="${splunk_fqdn_prefix}.${location}.cloudapp.azure.com"
az group create -n "$rg_name" -l "$location"
az network vnet create -g "$rg_name" -n "$vnet_name" --address-prefixes "$vnet_prefix" --subnet-name "$subnet_name" --subnet-prefixes "$subnet_prefix" --location "$location"
az vm create -g "$rg_name" -n "$vm_name" --image Ubuntu2204 --admin-username "$ssh_admin_user" --admin-password "$ssh_admin_password" --vnet-name "$vnet_name" --subnet "$subnet_name" --public-ip-sku Standard --public-ip-address-dns-name "$splunk_fqdn_prefix" --nsg-rule SSH --size "$vm_size"
az network nsg rule create -g "$rg_name" --nsg-name "${vm_name}NSG" -n "Splunk" --priority 1010 --source-address-prefixes '*' --destination-address-prefixes '*' --destination-port-ranges 8000 8089 9997 8088 9998 514 9999 1514 --access Allow --protocol Tcp --description "Splunk ports"
echo "Connect to Splunk VM: ssh ${ssh_admin_user}@${splunk_fqdn}"- Create and account on Splunk Enteprise website and copy Linux .deb installer download link:
- Connect to Splunk VM in SSH and execute below commands:
sudo su -
apt update -y && apt upgrade -y
apt install wget apt-transport-https gnupg2 -y
wget https://download.splunk.com/products/splunk/releases/9.0.4.1/linux/splunk-9.0.4.1-419ad9369127-linux-2.6-amd64.deb
dpkg -i splunk-9.0.4.1-419ad9369127-linux-2.6-amd64.deb
/opt/splunk/bin/splunk enable boot-start --accept-license
# Enter login & password
systemctl start splunk- Sign In to http://${splunk_fqdn}:8000
- Install "Splunk Add-on for Microsoft Cloud Services":
- Install "Monitoring Kubernetes - Metrics and Log Forwarding":
- Create AKS cluster:
AKS_RESOURCE_GROUP="aks-splunk-rg"
AKS_REGION="westeurope"
AKS_NAME="aks-demo-splunk"
az group create -n "$AKS_RESOURCE_GROUP" -l "$AKS_REGION"
az aks create -g "$AKS_RESOURCE_GROUP" -n "$AKS_NAME" --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring --generate-ssh-keys- Create Event Hub Namespace & Event Hub:
EH_RESOURCE_GROUP="eventhub-rg"
EH_REGION="westeurope"
EH_NS_NAME="eh-ns-monitoring-$RANDOM"
EH_NAME="eh-monitoring-$RANDOM"
az group create -n "$EH_RESOURCE_GROUP" -l "$EH_REGION"
az eventhubs namespace create --name "$EH_NS_NAME" --resource-group "$EH_RESOURCE_GROUP" -l "$AKS_REGION"
az eventhubs eventhub create --name "$EH_NAME" --resource-group "$EH_RESOURCE_GROUP" --namespace-name "$EH_NS_NAME"- Configure AKS cluster to send all logs & metrics to Event Hub:
- Create an Azure Service Principal:
az ad sp create-for-rbac --name "AZ_SP_SPLUNK" --skip-assignment- Assign
Azure Event Hubs Data Receiverrole to Service Principal on Event Hub Namespace:
az role assignment create --role "Azure Event Hubs Data Receiver" --assignee "4653eade-c757-43b9-xxxx-97855b3aab40" --scope "/subscriptions/2c49b441-xxxx-xxxx-xxxx-cd28d472544d/resourceGroups/eventhub-rg/providers/Microsoft.EventHub/namespaces/eh-ns-monitoring-1602"-
Configure SP in Splunk Add-On for Microsoft Cloud Services
-
Add Event Hub as an input of the Add-On:
- Check logs are available:
- Add a new HTTP Event Collector:
- Give it a name:
- Leave Input Settings blank:
- Submit:
- Enable HTTP Event Collector:
- Go to Global Settings:
- Enable & Save:
Installation guide is available here: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/v5/installation/
To achieve this question, you must request a trial license to outcold solutions.
- Download yaml file:
wget https://www.outcoldsolutions.com/docs/monitoring-kubernetes/v5/configuration/1.24/collectorforkubernetes.yaml- Edit it, only below fields with your license / token / url:
vi collectorforkubernetes.yaml[general]
acceptLicense = true
license = Q0gwSDJISUVQUjBMNjoxMDA6MTY4NDU3Nzg2Mjo0.HZXmL7Gd8j57C4PJZpQuKu4zC2MCN+43+Y4jHQ.JkmlmuUbrP/o2OHclNzpVhMvsWoWZCGubGdRRg
fields.kubernetes_cluster = aks-demo-splunk
...
# Splunk output
[output.splunk]
# Splunk HTTP Event Collector url
url = http://demo-splunk-david.westeurope.cloudapp.azure.com:8088/services/collector/event/1.0
# Splunk HTTP Event Collector Token
token = 7451d8c8-799e-4f28-ab67-09ea3b0c28c8
# Allow invalid SSL server certificate
insecure = true- Apply the configuration to AKS cluster:
az account set --subscription 2c49b441-xxxx-xxxx-xxxx-cd28d472544d
az aks get-credentials --resource-group aks-splunk-rg --name aks-demo-splunk
kubectl apply -f ./collectorforkubernetes.yaml- Check Daemon sets:
kubectl get ds -n collectorforkubernetes
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
collectorforkubernetes 1 1 1 1 1 <none> 91s
collectorforkubernetes-master 0 0 0 0 0 <none> 91s- Go to Splunk App - Monitoring Kubernetes:
