An interesting request by a customer recently, where they wanted to have Operations Manager alerts of a certain criteria escalated to other teams, should the alert remain unresolved for x number of minutes. You’d probably begin to think that were going to need an Orchestrator runbook to achieve this, or for bells and whistles, there are other third-party options like Derdack Enterprise Alert to name one. Both Orchestrator and Derdack would easily be able to handle alert email escalations of a multitude of scenarios. However, the customer didn’t want to spend time and money on other solutions, so we suggested that we look at creating multiple Operations Manager email notification channels and subscription to achieve this in the product natively. The customer wanted to escalate alert emails relating to ‘Low disk space’:
When Operations Manager raises ‘Logical Disk Free Space Low’ type alerts: Email 1st line team immediately
Should the alert remain active for 30 minutes: Email 2nd line team
Should the alert remain active for 1 hour: Email the team manager
Or to put it graphically….
The requirements to achieve this scenario in Operations Manager were:
x3 – Notification Channels
x3 – Notification Subscribers
x3 – Notification Subscriptions
In the example, we used the HTML offering from SquaredUp. This allows you to create HTML formatted notification channels in Operations Manager, which produces far more detailed and richer/cleaner notification emails. Running the PowerShell script provided by SquaredUp three times to create the custom “HTML….” notification channels:
To add some more context to the formatted emails, we then edited the HTML structure for the ‘Escalation 1’ and ‘Escalation 2’ channels to include additional text in the email subject line and body of the email. Example –
This step simply involved associating the relevant teams and individuals that would be receiving the email alerts based on their team membership.
The notification subscriptions include the criteria to trigger the email, the subscribers and the notification channel to use. The customer wanted email escalation on ‘Logical Disk Free Space Low’ type alerts, so we needed to create three subscriptions with the required subscribers and HTML formatted channel. So a little more work required to configure, multiple subscriptions (could be done in bulk with Operations Manager Shell) and channels to configure, but the end result was exactly as required and worked well, teams now get notifications if alerts are 30mins in state and if they remain 1 hr in state…
30 Minute escalation email
1 hour escalation email