Alert Subscription Acknowledgement Performance Monitor

Microsoft.SystemCenter.HealthServiceModules.AlertSubscriptionDSModule.AcknowledgementPerformanceMonitor (UnitMonitor)

Alert Subscription Data Source Module Acknowledgement performance threshold monitor.

Knowledge Base article:

Summary

Generation of notification is delayed due to overloaded resources causing acknowledgements for Notifications to take longer than expected. Acknowledgements are received when the Notification subsystem generates notification based on a subscription. The acknowledgements have slowed down identifying an overall slowing of the subsystem.

Causes

Notifications Server has slowed down and is unable to keep up with the generation of notifications for the alerts raised. The acknowledgements on the sending of notifications through the channels have slowed down. This may happen due to

1. Alert storms - too many alerts have been generated than can be handled by the Notification server in reasonable time period.

2. Operations DB is overloaded and has caused overall system slow down.

The Notification acknowledgements can slow down usually because the Notifications Resource Pool, and its member Management server that has the Notification subsystem is overloaded. It is possible that the Operations database is under heavy load or the resources are being exhausted.

If these alerts are generated during or shortly after installation of the Data Warehouse or reporting components likely cause is due to the opening the flow of alerts and state transitions to the DW. However, the corresponding data structures are not there yet since Data Warehouse did not deploy its components as of yet. Deployment happens within ~30 minutes of installation.

While deployment is not yet done, data will be retained by alert and state sync workflows to be later written to the DW. Thus, these workflows do not put data back to data sources and that in turn generates the alerts in question. Once deployment is complete, data is written and alerts clear up automatically.

If alerts do not auto-resolve within 120 minutes of Data Warehouse installation then another problem exists.

Resolutions

Notification subsystem is taking a long time to generate acknowledgements for notification messages. This is due to slowing down of the lack of resources available to the Notification subsystem.

Note the high and low watermarks for the Notification subsystem. These indicate the extents of the Alerts data in the Operations Database that is being processed.
Check the subscriptions in the Management group. If there are any subscriptions that have complex queries associated with it they can slow down the generation of all notifications. See if you can simplify the criteria for the alert.
You may have too many subscriptions. Check if you can disable or delete unnecessary subscriptions.
The computer hosting the Operations Database may not have sufficient resources. Diagnose the resources hardware and software that are being over utilized. Adding more resources could ease the load.
Inspect the messages in the Operations Manager event log. Other indications of problems in the system may direct you to further diagnostic steps.

1. Check the Alerts view for newly created Alerts. If there are too many alerts being generated, identify the root cause of the alerts and fix it.

2. If the Alerts cannot be stopped, disable the Alert generation on the monitor until the rate of Alerts can be controlled.

3. If all the notifications about the same Alert, identify the subscription that is causing the Notification and disable it. The Subscription ID of the subscription behind the notification can usually be found in the body of the notification message.

4. The Operations DB or the machine hosting it may be overloaded. Please use Operations Database diagnostic steps to verify if these problems can be fixed.

Element properties:

Target

Microsoft.SystemCenter.AlertNotificationSubscriptionServer

Parent Monitor

Microsoft.SystemCenter.HealthServiceModules.AlertSubscriptionDSModule.Rollup

Category

PerformanceHealth

Enabled

True

Alert Generate

True

Alert Severity

Warning

Alert Priority

High

Alert Auto Resolve

True

Monitor Type

Microsoft.Windows.2SingleEventLog2StateMonitorType

Remotable

True

Accessibility

Public

Alert Message

Alert subscription data source module acknowledgements taking longer than expected.

{0}

RunAs

Default

Source Code:

<UnitMonitor ID="Microsoft.SystemCenter.HealthServiceModules.AlertSubscriptionDSModule.AcknowledgementPerformanceMonitor" Accessibility="Public" Enabled="true" Target="Microsoft.SystemCenter.AlertNotificationSubscriptionServer" ParentMonitorID="Microsoft.SystemCenter.HealthServiceModules.AlertSubscriptionDSModule.Rollup" Remotable="true" Priority="Normal" TypeID="Windows!Microsoft.Windows.2SingleEventLog2StateMonitorType" ConfirmDelivery="true">

  <Category>PerformanceHealth</Category>

  <AlertSettings AlertMessage="Microsoft.SystemCenter.HealthServiceModules.AlertSubscriptionDSModule.AcknowledgementPerformanceMonitorMessage">

    <AlertOnState>Warning</AlertOnState>

    <AutoResolve>true</AutoResolve>

    <AlertPriority>High</AlertPriority>

    <AlertSeverity>Warning</AlertSeverity>

    <AlertParameters>

      <AlertParameter1>$Data/Context/EventDescription$</AlertParameter1>

    </AlertParameters>

  </AlertSettings>

  <OperationalStates>

    <OperationalState ID="AlertSubscriptionDataSourceAcknowledgementsRunningSlow" MonitorTypeStateID="FirstEventRaised" HealthState="Warning"/>

    <OperationalState ID="AlertSubscriptionDataSourceAcknowledgementsRunningNormal" MonitorTypeStateID="SecondEventRaised" HealthState="Success"/>

  </OperationalStates>

  <Configuration>

    <FirstComputerName>.</FirstComputerName>

    <FirstLogName>Operations Manager</FirstLogName>

    <FirstExpression>

      <And>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="Integer">EventDisplayNumber</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="Integer">11411</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="String">PublisherName</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="String">Health Service Modules</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

      </And>

    </FirstExpression>

    <SecondComputerName>.</SecondComputerName>

    <SecondLogName>Operations Manager</SecondLogName>

    <SecondExpression>

      <And>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="Integer">EventDisplayNumber</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="Integer">11412</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="String">PublisherName</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="String">Health Service Modules</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

      </And>

    </SecondExpression>

  </Configuration>

</UnitMonitor>