Alert Management Pack

Introduction

By default, the “Owner” field for each OpsMgr alert is left blank. Using the Console, administrators can assign ownership to an individual or team. The field is free text: any value can be entered.

Alert Ownership

Why assign an “owner” to each alert? There many reasons why you might want to assign an alert Owner, chief among these is accountability. If you’ve been around OpsMgr long enough, you’re familiar with the “Alerts” view which lists out every open alert in the system. Who owns these alerts? What needs to be done with them?

Most management packs have their own Alert Views which are specific to object classes defined within that pack. However, this can quickly become cumbersome to dig through dozens or hundreds of nested folders searching for the right Alert View.

In addition, companies organize their IT teams differently and assign different areas of responsibility to each team. For example, at Company A, the Windows Team may also be responsible for Active Directory and DNS, but in Company B these technologies might be managed by different teams.

OpsMgr is deliberately engineered to allow for these different organizational complexities, but implementation is left up to individual customers.

The SCOM Alert Management Pack

With the SCOM Alert Management Pack, we assign alert ownership primarily based on the Management Pack that generated the alert. We use a customizable configuration file that allows OpsMgr admins to adjust alert assignment based on how their teams are organized. We allow you to define exceptions, so that you can make assignments more granular based on the properties of the alert. For example, we can differentiate ownership based on a substring of the full name for the monitoring object associated with the alert.

Now, with the Alert Owner field populated, we can create a custom view for the “Windows Team” that includes any alert where the Owner field equals “Windows Team”. This can include all the alerts from the basic Windows Management Packs, plus alerts from the Cluster Management Pack, the DNS Management Pack and others that you can choose. If you have created custom management packs with alerts that are specific to your Windows team, you can include those alerts as well. This allows you to put all the alerts for your Windows Team in one view and you don’t have to create a single group.

Getting Started

STOP!!!!!

1. WARNING! If you have subscriptions that are tied to the “New” Alert Resolution State, you will want to pause before you enable the Assign SCOM Alerts rule. When we populate the Owner field in an alert, we update the Resolution State to “Assigned” (in our default configuration, this is Resolution State 5). If your notifications are tied to alerts being in “New” Resolution State, it *may* break your subscriptions. At a minimum, the Owner field of an alert will not yet be populated until the Alert is in the Assigned Resolution State.
2. WARNING! If you have already implemented custom resolution states, be aware that by default, we use the following custom Resolution States:
   1. Resolution State 5 (Assigned)
   2. Resolution State 15 (Verified)
   3. Resolution State 18 (Alert Storm)

Installation

Download the Alert Management MP: https://github.com/hmscott4/AlertManagement .
Detailed installation instructions: Install Management Pack · hmscott4/AlertManagement Wiki (github.com)

A basic overview of the steps:

1. Import the Management Pack
2. Add custom resolution states
3. Deploy the Configuration Files
   1. Edit Ownership Assignments to suit your environment
   2. Enable/Disable Ownership Assignments as needed
4. Enable the “Assign SCOM Alerts” rule

Configuration Files

We generate two Configuration Files as part of the Management Pack. The default configuration files are meant to serve as a “starting point”. You may edit them as needed to suit your operational requirements.

IMPORTANT: The configuration files should be stored on a file share that is accessible to all the management servers in your Management Group.

IMPORTANT: When we generate the default assign.alert.config file, we iterate through all the management packs in your Management Group and make an educated guess as to the default owner. However:

1. It’s likely that you will need to review alerts in the “Unassigned” assignment rule. These are likely to be custom management packs that we didn’t know what to do with.
2. If you add new management packs to your Management Group, you will have to create the associations yourself.

Let’s have a closer look at the “assign.alert.config” file. The XML is broken down into two primary regions: “exceptions” and “assignments“. Think of assignments as being the “default” behavior. Exceptions are just that: exceptions to the defaults.

<exceptions>
     <exception ID="1" Name="Server Offline" Owner="EFG Windows Team" enabled="true">
        <Alert Name="Health Service Heartbeat Failure" > 
           <AlertProperty>MonitoringObjectDisplayName</AlertProperty>
           <AlertPropertyMatches>efg.lcl</AlertPropertyMatches>
        </Alert>
     </exception> 
     <exception ID="2" Name="Server Offline" Owner="Windows Team" enabled="true">
        <Alert Name="Health Service Heartbeat Failure" /> 
     </exception> 
     <exception ID="3" Name="Windows Server EFG" Owner="EFG Windows Team" enabled="true">
        <ManagementPack Name="Microsoft.Windows.Server.2016.Monitoring" /> 
           <AlertProperty>MonitoringObjectFullName</AlertProperty>
           <AlertPropertyMatches>efg.lcl</AlertPropertyMatches>
     </exception> 
 </exceptions>
 <assignments>
     <assignment ID="1" Name="Windows" Owner="Windows Team" enabled="true">
       <ManagementPack Name="Microsoft.Windows.Server.2016.Monitoring" />
       <ManagementPack Name="Microsoft.Windows.Server.2012.Monitoring" />
       <ManagementPack Name="Microsoft.Windows.Server.2012.R2.Monitoring" />
     </assignment>
     <assignment ID="2" Name="Monitoring" Owner="Monitoring Team" enabled="true">
       <ManagementPack Name="Microsoft.SystemCenter.2007" />
     </assignment>
 </assignments>

First things first:

1. Exceptions that have an AlertProperty filter always get processed first
2. Exceptions that do not have an AlertProperty filter get processed next.
3. Assignments are always processed after exceptions
4. Each rule (exception or assignment) may be enabled or disabled by changing the “enabled” attribute

Ownership Assignment Process

When the “Assign SCOM Alerts” rule runs, we do the following:

We evaluate each open alert where the Owner field is blank or “Unassigned”. We then match it to an exception or an assignment in the configuration file. We then update the alert Owner and set the resolution state to “Assigned”. Once a match is found, we’re done with that alert.

If we are unable to find a match, then the Owner will be updated to “Unassigned“. We always check for open “Unassigned” alerts during the “Assign SCOM Alerts” phase. So, if you update the configuration file, we’ll update the ownership assignment for matching “Unassigned” alerts.

Interpretation

Using the example above, alerts are evaluated using the following rules (in order):

1. Exception ID 1 assigns any alert with the Name “Health Service Heartbeat Failure” AND where the MonitoringObjectDisplayName matches “efg.lcl” to the “EFG Windows Team”.
2. Exception ID 2 assigns any other alert with the Name “Health Service Heartbeat Failure” to the “Windows Team” (meaning “Health Service Heartbeat Failure” alerts where the MonitoringObjectDisplayName does not match “efg.lcl”.
3. Exception ID 3 assigns any alert from the Windows Server 2016 Management Pack AND where the MonitoringObjectFullName matches “efg.lcl” to the “EFG Windows Team”.
4. Assignment ID 1 assigns any alert generated by the Windows Server Management Packs (and which do NOT match “efg.lcl”) to the “Windows Team”.
5. Assignment ID 2 assigns any alert generated by the Microsoft.SystemCenter.2007 to the “Monitoring Team”

Enable Ownership Assignment

By default, the “Assign SCOM Alerts” rule is disabled. To enable the rule, use an override:

There are two overrides that are required:

1. Configuration File: this is the fully qualified UNC path to the assign.alert.config file. Remember to put it on a file share accessible to all Management Servers.
2. Enabled: set to true

Optionally, you can also set the following Overrides:

1. Assigned Resolution State: String value; if you don’t want to impact your current subscriptions, you could set a value of “New”
2. Unassigned Resolution State: String value; if you want to use an alternate Resolution State for alerts that can’t be matched in the configuration file, enter it here. Note: you must create this Resolution State yourself.
3. Debug Logging: If you are having issues (or you just want to see what’s going on underneath the covers), enable Debug Logging. Log entries will appear in the Operations Manager Event log. Search for Event Id 9931.

Results

In my lab environment, I have three separate domains “administered” by two independent teams:

1. Exception 1 ensures that “Health Service Heartbeat” alerts get assigned to the “EFG Windows Team”
2. Exception 2 ensures that remaining “Health Service Heartbeat” alerts get assigned to the main “Windows Team“. These are alerts generated in the Microsoft.SystemCenter.2007 Management Pack, which we otherwise would target to the “Monitoring Team“).
3. Exception 3 ensures that alerts generated by Windows Servers in the efg.lcl domain get assigned to the “EFG Windows Team“.
4. Assignment 1 assigns other Windows alerts to the main “Windows Team“
5. Assignment 2 assigns alerts associated with the Operations Manager Infrastructure to the “Monitoring Team”.

What does this look like in OpsMgr? I have shut off four servers in my lab: 3 from my primary (“ABCD”) domain, and 1 from my secondary (“EFG”) domain.

When the alerts first show up, they look just as the always do:

After a couple of minutes, the Management Pack gets the new alerts, updates the Owner field and sets the Resolution State to “Assigned“:

Note: By default, the Owner field is not displayed in the main “Alerts” view. You can add it to the display by right-clicking the title bar and selecting “Personalize View…“. Then, click on the check-box next to the “Owner” field. While you’re at it, consider checking the fields “Last Modified” and “Repeat Count“.

By double-clicking on the alert, we can see the Owner field and the Resolution State:

Internally, if you look at the History tab in the Alert, you’ll see what was done. We add a comment that indicates:

1. The value to which the Owner field was updated;
2. the specific entry from the configuration file which was used in the assignment process

The History tab looks like:

Now, if I’m a member of the EFG Windows Team, I can create a custom Alert view and display all the alerts that have been assigned to the “EFG Windows Team”. It doesn’t matter which Management Pack generated the alert, I can filter just those alerts that have been Assigned to the EFG Windows Team.

After saving this new view, I can open it and see all the Open alerts that have been assigned to the EFG Windows Team. Note that I snuck in a couple of additional alerts. These were generated by the Windows Server 2016 Management Pack and were assigned to the EFG Windows Team because the Monitoring Object Full Name matches the text “efg.lcl”. This action was performed by Exception ID 3.

Benefits

Now, instead of having to search all over the OpsMgr console to find the alerts that apply to my team, I can find all of my alerts in one place. I don’t have to create a lot of complex Dynamic Groups to ensure that my support teams only see the alerts that they “own”.

We can also leverage the alert Owner field when generating incidents for a Service Desk application. We can map the Owner field to a specific queue within the Service Desk application. This helps ensure that alerts get routed to the correct team immediately.

Finally, with PowerShell, we can generate reports with the number of open alerts by Owner. This can help drive greater accountability into our monitoring solution.

Here’s a quick one-liner in PowerShell to show the count of open alerts by Owner:

Get-SCOMAlert -Criteria "ResolutionState < 255" | Group-Object Owner

And the results:

Acknowledgements

There are a lot of people who have helped make this management pack a reality. It would be impossible to thank all of them, but I would like to specifically acknowledge:

• Dan Reist
• Shane Hutchens
• Tyson Paul