Processing Backlogged Events Taking a Long Time

Microsoft.SystemCenter.HealthServiceModules.WindowsEventLog.ProcessingBackLoggedEventsTooLong (UnitMonitor)

This monitor checks the if the Windows Event Log module is processing back logged events for some period of time.

Knowledge Base article:

Summary

This monitor and alert indicates that the Health Service has been processing older events for a particular event log longer than expected.

Below is a summary of the default configuration of this monitor:

Warning state: Transition to warning state if we are still processing back logged events after 10 minutes. This means that we processed Event 25017 (started processing back logged events), but we have not reached the current record (indicated by event 25018).
Critical state: Transition from warning state to critical state if after 20 minutes, we are still processing back logged events. This means in addition to the 10 minutes that the Health Service has already been processing back logged events, an additional 10 minutes has passed for a total of 20 minutes since we processed Event 25017 has passed.

Causes

Both the warning and critical state can indicate the following may be happening on the agent:

The computer where this event was raised is logging hundreds to thousands of events per minute that all need to be processed for monitoring.
The computer may be low on available resources (for example; memory).
The Health Service was stopped for an extended period of time and must process all events from the last one it successfully processed.

Resolutions

You can perform the following checks to further determine the root cause of the problem:

1. Open the event viewer on the computer where for this alert or monitor state.

2. Check to see if there is an application or event source that seems to be logging many events per minute to the event log

3. If there is no clear indication of the application that may be logging these events, check the resource utilization on this computer. If there is an application that is consuming large amounts of memory or CPU, check with the application owner or administrator if this is expected behavior.

4. If you are not concerned with the loss of monitoring from the existing events, you can clear the event log.

Note: Clearing the event log when the Health Service is still processing back logged events will result in loss of monitoring.

Element properties:

Target

Microsoft.SystemCenter.HealthService

Parent Monitor

Microsoft.SystemCenter.HealthServiceModules.WindowsEventLog.Rollup

Category

AvailabilityHealth

Enabled

True

Alert Generate

True

Alert Severity

Warning

Alert Priority

Normal

Alert Auto Resolve

True

Monitor Type

Microsoft.SystemCenter.HealthServiceModules.CorrelatedEventAndMissingEvent

Remotable

True

Accessibility

Internal

Alert Message

Processing Backlogged Events Taking a Long Time

{0}

RunAs

Default

Source Code:

<UnitMonitor ID="Microsoft.SystemCenter.HealthServiceModules.WindowsEventLog.ProcessingBackLoggedEventsTooLong" Accessibility="Internal" Enabled="true" Target="SCLibrary!Microsoft.SystemCenter.HealthService" ParentMonitorID="Microsoft.SystemCenter.HealthServiceModules.WindowsEventLog.Rollup" Remotable="true" Priority="Normal" TypeID="Microsoft.SystemCenter.HealthServiceModules.CorrelatedEventAndMissingEvent" ConfirmDelivery="true">

  <Category>AvailabilityHealth</Category>

  <AlertSettings AlertMessage="Microsoft.SystemCenter.HealthServiceModules.WindowsEventLog.ProcessingBackLoggedEventsTooLong_AlertMessageResourceID">

    <AlertOnState>Warning</AlertOnState>

    <AutoResolve>true</AutoResolve>

    <AlertPriority>Normal</AlertPriority>

    <AlertSeverity>Warning</AlertSeverity>

    <AlertParameters>

      <AlertParameter1>$Data/Context/Item0Context/DataItem/EventDescription$</AlertParameter1>

    </AlertParameters>

  </AlertSettings>

  <OperationalStates>

    <OperationalState ID="FinishedProcessingBackLoggedEvents" MonitorTypeStateID="SuccessEventRaised" HealthState="Success"/>

    <OperationalState ID="StillProccessingBackLoggedEventsWarning" MonitorTypeStateID="CorrelatedMissingEventWarningRaised" HealthState="Warning"/>

    <OperationalState ID="StillProccessingBackLoggedEventsError" MonitorTypeStateID="CorrelatedMissingEventErrorRaised" HealthState="Error"/>

  </OperationalStates>

  <Configuration>

    <ComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</ComputerName>

    <LogName>Operations Manager</LogName>

    <Expression>

      <Or>

        <Expression>

          <And>

            <Expression>

              <Or>

                <Expression>

                  <SimpleExpression>

                    <ValueExpression>

                      <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                    </ValueExpression>

                    <Operator>Equal</Operator>

                    <ValueExpression>

                      <Value Type="UnsignedInteger">25018</Value>

                    </ValueExpression>

                  </SimpleExpression>

                </Expression>

                <Expression>

                  <SimpleExpression>

                    <ValueExpression>

                      <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                    </ValueExpression>

                    <Operator>Equal</Operator>

                    <ValueExpression>

                      <Value Type="UnsignedInteger">26018</Value>

                    </ValueExpression>

                  </SimpleExpression>

                </Expression>

              </Or>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="String">PublisherName</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="String">Health Service Modules</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <Value Type="String">$Target/ManagementGroup/Name$</Value>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <XPathQuery Type="String">Params/Param[1]</XPathQuery>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </And>

        </Expression>

        <Expression>

          <And>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">2002</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="String">PublisherName</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="String">HealthService</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </And>

        </Expression>

        <Expression>

          <And>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="String">PublisherName</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="String">Health Service ESE Store</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">102</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </And>

        </Expression>

      </Or>

    </Expression>

    <FirstCorrelatedMissingEventWarningComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</FirstCorrelatedMissingEventWarningComputerName>

    <FirstCorrelatedMissingEventWarningLogName>Operations Manager</FirstCorrelatedMissingEventWarningLogName>

    <FirstCorrelatedMissingEventWarningExpression>

      <And>

        <Expression>

          <Or>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">25017</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">26017</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </Or>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="String">PublisherName</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="String">Health Service Modules</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <Value Type="String">$Target/ManagementGroup/Name$</Value>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <XPathQuery Type="String">Params/Param[1]</XPathQuery>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

      </And>

    </FirstCorrelatedMissingEventWarningExpression>

    <SecondCorrelatedMissingEventWarningComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</SecondCorrelatedMissingEventWarningComputerName>

    <SecondCorrelatedMissingEventWarningLogName>Operations Manager</SecondCorrelatedMissingEventWarningLogName>

    <SecondCorrelatedMissingEventWarningExpression>

      <And>

        <Expression>

          <Or>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">25018</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">26018</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </Or>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="String">PublisherName</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="String">Health Service Modules</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <Value Type="String">$Target/ManagementGroup/Name$</Value>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <XPathQuery Type="String">Params/Param[1]</XPathQuery>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

      </And>

    </SecondCorrelatedMissingEventWarningExpression>

    <CorrelatorWarning>

      <CorrelationExpression>

        <Expression/>

      </CorrelationExpression>

      <Count>1</Count>

      <Interval>600</Interval>

      <CorrelationOrder>InSequence</CorrelationOrder>

      <CorrelationItemPolicy>ResetWindow</CorrelationItemPolicy>

    </CorrelatorWarning>

    <FirstCorrelatedMissingEventErrorComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</FirstCorrelatedMissingEventErrorComputerName>

    <FirstCorrelatedMissingEventErrorLogName>Operations Manager</FirstCorrelatedMissingEventErrorLogName>

    <FirstCorrelatedMissingEventErrorExpression>

      <And>

        <Expression>

          <Or>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">25017</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">26017</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </Or>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="String">PublisherName</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="String">Health Service Modules</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <Value Type="String">$Target/ManagementGroup/Name$</Value>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <XPathQuery Type="String">Params/Param[1]</XPathQuery>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

      </And>

    </FirstCorrelatedMissingEventErrorExpression>

    <SecondCorrelatedMissingEventErrorComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</SecondCorrelatedMissingEventErrorComputerName>

    <SecondCorrelatedMissingEventErrorLogName>Operations Manager</SecondCorrelatedMissingEventErrorLogName>

    <SecondCorrelatedMissingEventErrorExpression>

      <And>

        <Expression>

          <Or>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">25018</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="UnsignedInteger">26018</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </Or>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <XPathQuery Type="String">PublisherName</XPathQuery>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <Value Type="String">Health Service Modules</Value>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

        <Expression>

          <SimpleExpression>

            <ValueExpression>

              <Value Type="String">$Target/ManagementGroup/Name$</Value>

            </ValueExpression>

            <Operator>Equal</Operator>

            <ValueExpression>

              <XPathQuery Type="String">Params/Param[1]</XPathQuery>

            </ValueExpression>

          </SimpleExpression>

        </Expression>

      </And>

    </SecondCorrelatedMissingEventErrorExpression>

    <CorrelatorError>

      <CorrelationExpression>

        <Expression/>

      </CorrelationExpression>

      <Count>1</Count>

      <Interval>1200</Interval>

      <CorrelationOrder>InSequence</CorrelationOrder>

      <CorrelationItemPolicy>ResetWindow</CorrelationItemPolicy>

    </CorrelatorError>

  </Configuration>

</UnitMonitor>