QuickTricks: Alerting during business hours with Log Analytics

In Log Analytics you can create an alert which runs on a specific schedule frequency which will alert when the search query matches the criteria that you specify. That alert can send an email, activate a webhook, activate a runbook or activate an ITSM Action (an example of this configuration is shown below). What this alerting approach does not consider is that some alerts only need to fire during business hours. This blog post will show a quick trick which allows you to define alerts which will only fire during business hours.

We can start from an existing query such as the one below which provides results where the % Processor Time counter is > 60.

Perf

| where (ObjectName == "Processor"
or ObjectName == "System") and CounterName == "% Processor Time"

| where CounterValue > 60

| sort
by TimeGenerated desc

To fire alerts only during business hours we take the query above and add a few lines to it so that the query itself will define whether or not the alert should fire based on the original condition plus the time of the day. The new pieces of the query are bolded and explained below.

// Define startDateTime to the start of the day

let startDatetime = startofday(now());

// Define StartNotification to 7 hours after the start of the day (7:00 am)

let StartNotification = startDatetime + 7hours;

// Define StopNotification to 17 hours after the start of the day (5:00 pm)

let StopNotification = startDatetime + 17hours;

Perf

// Restrict the TimeGenerated field to only fire if it's in the time range (between 7:00 am and 5:00 pm)

| where TimeGenerated > StartNotification and TimeGenerated < StopNotification

| where (ObjectName == "Processor"
or ObjectName == "System") and CounterName == "% Processor Time"

| where CounterValue > 60

| sort
by TimeGenerated desc

A sample result for this is query is shown below.

For the alert form we can directly cut and paste over the query (minus the comments) as this form accepts multiple-line queries including comments so we can use the query above directly in the “Search Query” field as shown below.


Tip: To run the above query directly in log search directly it needs to be converted from a multiple-line query to a single-line query (for a quick trick on how to covert the multiple-line version into a single line version see this blog post). The single-lined version is shown below.

let startDatetime = startofday(now());let StartNotification = startDatetime + 7hours;let StopNotification = startDatetime + 17hours;Perf| where TimeGenerated > StartNotification and TimeGenerated < StopNotification| where (ObjectName == "Processor"
or ObjectName == "System") and CounterName == "% Processor Time"| where CounterValue > 60| sort
by TimeGenerated desc

Add a Saturday/Sunday exclusion:

We can even exclude specific days of the week from alerting by defining variables for Saturday and Sunday and excluding these days from the query. The new lines to perform this are in bold below.

let startDatetime = startofday(now());

let StartNotification = startDatetime +7hours;

let StopNotification = startDatetime +17hours;

let Saturday = "6.00:00:00";

let Sunday = "7.00:00:00";

Perf

| where TimeGenerated > StartNotification and TimeGenerated < StopNotification

| extend DayVar = dayofweek(TimeGenerated)

| where DayVar != Saturday and DayVar != Sunday

| where (ObjectName == "Processor"
or ObjectName == "System") and CounterName == "% Processor Time"

| where CounterValue > 60

Summary: If you are looking for a quick way to provide alerts only during business hours try using the query itself to restrict when an alert condition will match. This approach could be used to alert only during business hours, or to have one alert which notifies during business hours and a different alert which fires in off-hours (by reversing the conditions in the query).

 

UPDATE: Please note, UTC does need to be factored into these calculations. The example below shows how a UTC for CST (6 hour difference) can be applied.

let startDatetime = startofday(now());
// StartNotification is 8:00 am (8) plus 6 hours for the UTC Offset (14)
let StartNotification = startDatetime + 14hours;
// StopNotification is 5:00 pm (17) plus 6 hours for the UTC Offset (23)
let StopNotification = startDatetime + 23hours;
Perf
| where TimeGenerated > StartNotification and TimeGenerated < StopNotification
| where (ObjectName == "Processor" or ObjectName == "System") and CounterName == "% Processor Time"
| where CounterValue > 60
| sort by TimeGenerated desc

 

Note: This approach will need to be tweaked for time frames like Daylight Savings Time as CST isn’t always 6 hours different than UTC.

Leave a Reply