Monitor application using custom logs

The developer at Mystique Unicorn are looking monitor their app from the custom logs generated by the app running inside a Azure VM Can you show them how the can get started ?

🎯 Solutions

We will use Azure Monitor Agent¹ to collect the application logs from a specified location(for exampe: /var/log/miztiik*.json) and push them to a Log Analytics Workspace³. To receive the logs in the correct format we need to define a custom table^5,6 and a data collection rule. The rule will collect logs from our source(VM App), transform the logs using Kusto Query Language(KQL) to a structure defined in the custom table and store them in the sink(log analytics workspace).

This repository holds the bicep code to deploy the necessary infrastructure for this demo,

Virtual Machine(RedHat) inside a Vnet with public access.
- Bootstrapped with custom libs using customData script.
Log Analytics Workspace ( Analytics tier)
- Custom Table
- Data Collection Endpoint - With public access
- Data Collection Rule
  - Configured with the log location, KQL query and custom table

🧰 Prerequisites

This demo, instructions, scripts and bicep template is designed to be run in westeurope. With few or no modifications you can try it out in other regions as well(Not covered here).
- 🛠 Azure CLI Installed & Configured - Get help here
- 🛠 Bicep Installed & Configured - Get help here
- 🛠 VS Code & Bicep Extenstions - Get help here

⚙️ Setting up the environment

Get the application code

https://github.com/miztiik/send-vm-logs-to-azure-monitor
cd send-vm-logs-to-azure-monitor

🚀 Prepare the environment

Let check you have Azure Cli working with

  # You should have azure cli preinstalled
  az account show

You should see an output like this,

 {
   "environmentName": "AzureCloud",
   "homeTenantId": "16b30820b6d3",
   "id": "1ac6fdbff37cd9e3",
   "isDefault": true,
   "managedByTenants": [],
   "name": "YOUR-SUBS-NAME",
   "state": "Enabled",
   "tenantId": "16b30820b6d3",
   "user": {
     "name": "miztiik@",
     "type": "user"
   }
 }

🚀 Deploying the application

Let us walk through each of the stacks,

Stack: Main Bicep The params required for the modules are in params.json. Do modify them to suit your need.(Especially the adminPassword.secureString for the VM. You are strongly encouraged to Just-In-Time access or use SSH key instead of password based authentication). The helper deployment script deploy.sh will deploy the main.bicep file. This will inturn call the modules for Resource Group(RG), VNet & Virtual Machine, Log analytics workspace, Custom table, Data collection endpoint, Data collection rule(DCR) and finally attach the DCR to the VM.

sh deploy.sh

This stack will create almost all of the resoureces. The RG is created as a deployment at the subscription level. The other resources are deployments at resource group level.

Virtual Machine - A Redhat vm is bootstrapped using customData to install python3, git. We will use a bash script generate_data.sh. It will generate a json conformant log in this location /var/log/miztiik-yyyy-MM-dd.json for example /var/log/miztiik-2023-04-03.json. All new entries are appended to the file with new-line being the delimitter as expected by Azure Custom Logs.

{"request_id": "08b6b258-c6d6-44ff-802e-69d52422095a", "event_type": "inventory_event", "store_id": 45, "cust_id": 4416, "category": "Camera", "sku": 17293, "price": 44.61, "qty": 15
, "discount": 73.2, "gift_wrap": 0, "variant": "MystiqueAutomatoon", "priority_shipping": 1, "TimeGenerated": "2023-04-03T10:54:23", "contact_me": "github.com/miztiik" }
{"request_id": "60051c39-6817-4f52-bd8f-59cc9a38be0f", "event_type": "inventory_event", "store_id": 64, "cust_id": 4081, "category": "Camera", "sku": 17815, "price": 50.92, "qty": 16
, "discount": 29.2, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:24", "contact_me": "github.com/miztiik" }
{"request_id": "f340f343-fc0f-40a1-b9f5-2abdd14b929d", "event_type": "inventory_event", "store_id": 35, "cust_id": 338, "category": "Camera", "sku": 16158, "price": 53.76, "qty": 13,
"discount": 34.3, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:25", "contact_me": "github.com/miztiik" }

Formatting a single log line entry properly, will look like this,

{
  "request_id": "f5570d0f-5389-4bce-98c3-7ed5380eefb2",
  "event_type": "inventory_event",
  "store_id": 6,
  "cust_id": 206,
  "category": "Camera",
  "sku": 100657,
  "price": 2.52,
  "qty": 18,
  "discount": 8.2,
  "gift_wrap": true,
  "variant": "red",
  "priority_shipping": true,
  "TimeGenerated": "2021-04-25T12:52:40.938937",
  "contact_me": "github.com/miztiik"
}

After successfully deploying the stack, Check the Resource Groups/Deployments section for the resources.

🔬 Testing the solution
1. Connect to the VM
Connect to the VM using Just-In-Time access. This should allow your IP/IP list to securelly connect to your VM using private IP address. Use the password you updated in the params.json
```
ssh miztiik@publicIP
```
1. Generate Log Data
For some reason the customData, bootstrap script fails, will fix it later. Until then manually copy the generate_data.sh from this repo to the server. _You may need elevated privileges to create log file in /var/log.
```
sudo su
cd /var/log/
# Copy generate_data.sh to the server
sh generate_data.sh &
```
The script by default is configured to write one log entry every 5 seconds and generate a total 100000 log lines to/var/log/miztiik*.json
1. Query in Azure Log Analytics Workspace
Login to Azure Log Anaytics portal. You should be able to find it under /Miztiik_Enterprises_Custom_Logs_To_Monitor_xxx/providers/Microsoft.OperationalInsights/workspaces/lumberyard-payGTier-xxx",. Navigate to logs, select the custom table(lumberyardTablexxx_CL) & Run
Troubleshooting Azure Monitor Agent
- In case if you are facing issues, log into the VM and check the logs /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/ for potential causes.
- From the portal, you can query to find if the agent is reporting to Azure Monitor
```
Heartbeat | where Category == "Azure Monitor Agent" and Computer == "<computer-name>" | take 10
```
  Check if the AMA is running,
```
systemctl status azuremonitoragent
```
📒 Conclusion

Here we have demonstrated how to configure a VM to send application logs to Azure monitor. You can extend the solution to setup alerts on these logs or secure the data collection endpoing to use a private link etc.,
🧹 CleanUp

If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well

Resources created during Deploying The Application
Any other custom resources, you have created for this demo

# Delete from resource group
az group delete --name Miztiik_Enterprises_xxx --yes
# Follow any on-screen prompt

This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.

📌 Who is using this

This repository aims to show how to Bicep to new developers, Solution Architects & Ops Engineers in Azure.

💡 Help/Suggestions or 🐛 Bugs

Thank you for your interest in contributing to our project. Whether it is a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here

miztiik/send-vm-logs-to-azure-monitor

Monitor application using custom logs

🎯 Solutions

🧰 Prerequisites

⚙️ Setting up the environment

🚀 Prepare the environment

🚀 Deploying the application

🔬 Testing the solution

📒 Conclusion

🧹 CleanUp

📌 Who is using this

💡 Help/Suggestions or 🐛 Bugs

👋 Buy me a coffee

📚 References

🏷️ Metadata