| services | platforms | author |
|---|---|---|
virtual-network |
dotnet |
JayHCho |
Code for downloading operational network logs and converting them to .CSV files. Files can to be then uploaded to Power BI for analysis.
- Visual Studio 2015
- .NET Framework 4.6
- Microsoft Azure SDK - Latest
- Cloud Explorer for Visual Studio 2015 - Visual Studio Extension (recommended)
The solution contains 5 executable console app projects CountersLogConverter, EventsLogConverter, OperationsLogConverter, LoadBalancerAlertLogConverter and LoadBalancerHealthProbeLogConverter.
#####1. CountersLogConverter
In order to use this code, logging must be turned on via SDK or Ibiza portal (soon to be released) and familiarity with Azure Resource Manager is required.
The counters logs are stored in the Azure Storage Container as JSON blobs
{
"time": "2015-09-11T23:14:22.6940000Z",
"systemId": "e22a0996-e5a7-XXXX-XXXX-4357a6e8f0c5",
"category": "NetworkSecurityGroupRuleCounter",
"resourceId": "/SUBSCRIPTIONS/D763EE4A-9131-XXXX-XXXX-876035455EC4/RESOURCEGROUPS/INSIGHTOBONRPFOO/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSGINSIGHTOBONRPFOO",
"operationName": "NetworkSecurityGroupCounters",
"properties": {
"vnetResourceGuid":"{DD0074B1-4CB3-XXXX-XXXX-8719DFBA3568}",
"subnetPrefix":"10.0.0.0/24",
"macAddress":"001517D9C43C",
"ruleName":"DenyAllOutBound",
"direction":"Out",
"type":"block",
"matchedConnections":0
}
}
Converted .CSV file of counter log has following columns
- time
- systemId
- resourceId
- operationName
- properties.vnetResourceGuid
- properties.subnetPrefix
- properties.macAddress
- properties.ruleName
- properties.direction
- properties.type
- properties.matchedConnections
#####2. EventsLogConverter
In order to use this code, logging must be turned on. Familiarity with Azure Resource Manager is required.
The events logs are stored in the Azure Storage Container as JSON blobs
{
"time": "2015-09-11T23:05:22.6860000Z",
"systemId": "e22a0996-e5a7-XXXX-XXXX-4357a6e8f0c5",
"category": "NetworkSecurityGroupEvent",
"resourceId": "/SUBSCRIPTIONS/D763EE4A-9131-455F-8C5E-876035455EC4/RESOURCEGROUPS/INSIGHTOBONRPFOO/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSGINSIGHTOBONRPFOO",
"operationName": "NetworkSecurityGroupEvents",
"properties": {
"vnetResourceGuid":"{DD0074B1-4CB3-XXXX-XXXX-8719DFBA3568}",
"subnetPrefix":"10.0.0.0/24",
"macAddress":"001517D9C43C",
"ruleName":"AllowVnetOutBound",
"direction":"Out",
"priority":65000,
"type":"allow",
"conditions":{
"destinationPortRange":"0-65535",
"sourcePortRange":"0-65535",
"destinationIP":"10.0.0.0/8,172.16.0.0/12,169.254.0.0/16,192.168.0.0/16,168.63.129.16/32",
"sourceIP":"10.0.0.0/8,172.16.0.0/12,169.254.0.0/16,192.168.0.0/16,168.63.129.16/32"
}
}
}
Converted .CSV file of events log has following columns
- time
- systemId
- resourceId
- operationName
- properties.vnetResourceGuid
- properties.subnetPrefix
- properties.macAddress
- properties.ruleName
- properties.direction
- properties.priority
- properties.type
- properties.conditions.destinationPortRange
- properties.conditions.sourcePortRange
- properties.conditions.sourceIP
- properties.conditions.destinationIP
- properties.conditions.protocols
#####3. OperationsLogConverter
This app needs to be authorized to access Azure AD management API. Fore more information on Azure AD authorization go to: https://azure.microsoft.com/en-us/documentation/articles/resource-group-create-service-principal-portal/
Operations logs are retrieved through the Azure Insight API. For available properties look at member of Microsoft.Azure.Insight.Models.EventData class definition.
Converted .CSV file of operations log has following columns.
- SubscriptionId
- EventTimeStamp
- EventDate
- EventDataId
- CorrelationId
- EventName
- Level
- ResourceGroupName
- ResourceProviderName
- ResourceId
- ResourceName
- ResourceLocation
- Status
- Caller
- OperationId
- OperationName
- OperationRP
- OperationResType
- OperationType
- Description
- Title
- Service
- Region
- Transcript
- IncidentId
- IncidentType
#####4. LoadBalancerAlertLogConverter
In order to use this code, logging must be turned on, for how-to: https://github.com/Azure/azure-content/blob/master/articles/load-balancer/load-balancer-monitor-log.md
The alert event logs are stored in the Azure Storage Container as JSON blobs
{
"time": "2016-01-26T10:37:46.6024215Z",
"systemId": "32077926-b9c4-42fb-94c1-762e528b5b27",
"category": "LoadBalancerAlertEvent",
"resourceId": "/SUBSCRIPTIONS/XXXXXXXXXXXXXXXXX-XXXX-XXXX-XXXXXXXXX/RESOURCEGROUPS/RG7/PROVIDERS/MICROSOFT.NETWORK/LOADBALANCERS/WWEBLB",
"operationName": "LoadBalancerProbeHealthStatus",
"properties": {
"eventName": "Resource Limits Hit",
"eventDescription": "Ports exhausted",
"eventProperties": {
"public ip address": "40.117.227.32"
}
}
}
Converted .CSV file of alert event log has following columns
- time
- systemId
- category
- resourceId
- operationName
- properties.eventName
- properties.eventDescription
- properties.eventProperties.publicIpAddress
#####5. LoadBalancerHealthProbeLogConverter
In order to use this code, logging must be turned on, for how-to: https://github.com/Azure/azure-content/blob/master/articles/load-balancer/load-balancer-monitor-log.md
The healt status event logs are stored in the Azure Storage Container as JSON blobs
{
"time": "2016-01-26T10:37:46.6024215Z",
"systemId": "32077926-b9c4-42fb-94c1-762e528b5b27",
"category": "LoadBalancerProbeHealthStatus",
"resourceId": "/SUBSCRIPTIONS/XXXXXXXXXXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX/RESOURCEGROUPS/RG7/PROVIDERS/MICROSOFT.NETWORK/LOADBALANCERS/WWEBLB",
"operationName": "LoadBalancerProbeHealthStatus",
"properties": {
"publicIpAddress": "40.83.190.158",
"port": "81",
"totalDipCount": 2,
"dipDownCount": 0,
"healthPercentage": 100.000000
}
}
Converted .CSV file of healt status log has following columns
- time
- systemId
- category
- resourceId
- operationName
- properties.publicIpAddress
- properties.port
- properties.totalDipCount
- properties.dipDownCount
- properties.healthPercentage