We have 1000+ azure log analytic workspace alert rules created with AzureRM Templates and azurepiplines for our different projects. below is the ARM template used for all these alert rule creations and in alertrules we have custom payload section to get certain alert parameters json.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"actionGroupName": {
"type": "string"
},
"query": {
"type": "string"
},
"logAnalyticsWorkspaceId": {
"type": "string"
},
"AlertRuleName": {
"type": "string"
},
"tags": {
"type": "object"
},
"schedule": {
"type": "object"
},
"severity": {
"type": "int"
},
"operator": {
"type": "string"
},
"threshold": {
"type": "int"
},
"autoMitigate": {
"type": "string",
"defaultValue": false
},
"enabled": {
"type": "string"
},
"customWebhookPayload": {
"type": "object"
},
"location": {
"defaultValue": "[resourceGroup().location]",
"type": "string"
}
},
"resources":[
{
"type":"Microsoft.Insights/scheduledQueryRules",
"name": "[parameters('AlertRuleName')]",
"apiVersion": "2018-04-16",
"location": "[parameters('location')]",
"tags": "[parameters('tags')]",
"properties":{
"displayName": "[parameters('AlertRuleName')]",
"description": "[parameters('AlertRuleName')]",
"enabled": "[parameters('enabled')]",
"source": {
"query": "[parameters('query')]",
"dataSourceId": "[parameters('logAnalyticsWorkspaceId')]",
"queryType":"ResultCount"
},
"schedule":"[parameters('schedule')]",
"action":{
"odata.type": "Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.Microsoft.AppInsights.Nexus.DataContracts.Resources.ScheduledQueryRules.AlertingAction",
"severity": "[parameters('severity')]",
"aznsAction":{
"customWebhookPayload": "{ "AlertRuleName":"#alertrulename", "AlertType":"#alerttype", "Severity":"#severity", "Application":"#{appname}#", "Text":"#alertrulename fired with #searchresultcount records. #{alertDescription}#", "SearchQuery":"#searchquery" }",
"actionGroup": "[array(parameters('actionGroupName'))]"
},
"trigger":{
"thresholdOperator": "[parameters('operator')]",
"threshold": "[parameters('threshold')]"
}
}
}
}
]
}
As of now the application teams are getting alert notifications in MS teams and doing a manual verification for the alert root cause by checking the application traces from the Azure AppInsight failure for that timespan over the operations/exceptions traces.
Here we were trying for further enhanced solution to the application teams, so that when an alert is created and notified, they can directly view or analyze the corresponding traces or events from the appinsight than manually verifying them from azure appinsight ?
Can we make use of the payload json file created from each alert rules to dynamically get the application trace from appinsight for that alert occurrence?