Cloud Monitoring Adds PagerDuty Integration
Cloud Monitoring now supports PagerDuty integration! With this new notification type, alarm notifications can automatically create new incidents and resolve them once Cloud Monitoring detects things are okay.<!-- more -->
Because the Cloud Control Panel doesn't support creating notifications and notification plans, raxmon is one of the easier ways to set up PagerDuty notifications. If you haven't used raxmon before, there is a great guide for setting things up here.
In order to do this, we just have to go through the following steps:
- Figure out your PagerDuty service key
- Create the new PagerDuty notification
- Create or update a Cloud Monitoring notification plan
- Update an alarm to use your new notification plan
- Do a happy dance
If you'd like to dive right into things, there is a tl;dr at the bottom of the page.
The first thing you'll need to do is make sure you have configured a PagerDuty service to use with Cloud Monitoring.
- In your PagerDuty account, under the Services tab, click "Add New Service."
- Enter a name for the service, select an escalation policy and choose "Generic API system" for the Service Type.
- Click the "Add Service" button.
- Once the service is created, you'll be taken to the service page. On this page, you'll see the "Service key," which you will use when you create your notification.
Next you'll need to create the Cloud Monitoring notification that you'll attach to a notification plan with the service key you've obtained from PagerDuty. To do this with raxmon, do the following:
raxmon-notifications-create --type=pagerduty \ --details=service_key=abcd1234abcd1234abcd1234abcd1234
After you do this, raxmon will return a new ID for the notification that
will look something like
nt23k123. You'll want to keep this ID handy
for the next step of creating a
To view a detailed list of your existing notification plans, you can use the following raxmon command:
By default Cloud Monitoring has a single, dynamic notification plan
npTechnicalContactsEmail that sends an email notification to each of the
technical contacts on your Rackspace account. If this is the only
notification plan you see, you'll want to create a new one to take
advantage of the new PagerDuty integration.
Notification plans allow you to specify what notifications you want to use per alarm state(e.g. OK, WARNING, CRITICAL). This gives you the flexibility of doing something like emailing on WARNING and actually creating an incident in PagerDuty for CRITICAL events. Use raxmon to do this by using the following commands:
raxmon-notification-plan-create --ok-notifications=nt23k123 \ --warning-notifications=nt23k123 --critical-notifications=nt23k123
This command will create a new notification plan and return an ID
that looks something like
npTY46f7. Your new notification plan will
trigger an incident per alarm on WARNING and CRITICAL events and automatically
resolve the incident when an OK event is triggered.
Note: Incidents are triggered per alarm, meaning that a new incident will be created per alarm, as opposed to creating an incident per check or entity. Any additional alarm state changes for an incident will be appended to the currently active incident until it is resolved.
If you wanted to trigger incidents on WARNING and CRITICAL
events and resolve the incidents with PagerDuty while also sending an
email notification (e.g.
nt76df3U) on OK events, you could run the
raxmon-notification-plan-create --ok-notifications=nt23k123,nt76df3U \ --warning-notifications=nt23k123 --critical-notifications=nt23k123
Note: Once you've created or updated your notification plan, you can use the Cloud Control Panel to configure alarms to use it. Read on for how to do this with raxmon.
Next you will want to identify which alarm you want to update. To do
this list your alarms for an entity with the id of
enKEb23jB by using:
This will show you the alarms associated with an entity and their
labels. If you would like more information about the alarms, you can use
Your final step to take advantage of the new PagerDuty integration is
to update your alarms
to use your newly-created notification plan. Use
the ID that your call to
raxmon-notification-plan-create returned and
the alarm ID you've picked out, and running the following:
raxmon-alarms-update --entity-id=enKEb23JB --id=alwp0UoI45 \ --notification-plan=npTY46f7
After doing this you are all set! Cloud Monitoring will now automatically trigger and resolve incidents through PagerDuty.
# Create a new PagerDuty notification using the service key you've # created for Cloud Monitoring. This will return an ID that you'll use # when creating/updating a notification plan. raxmon-notifications-create --type=pagerduty \ --details=service_key=abcd1234abcd1234abcd1234abcd1234 # List existing notification plans to determine if you need to create # a new notification plan or can update an existing one. raxmon-notification-plans-list --details # Create a new notification plan using your newly-created PagerDuty # notification. Use the ID that this returns to update an alarm's # associated notification plan. raxmon-notification-plan-create --ok-notifications=nt23k123 \ --warning-notifications=nt23k123 --critical-notifications=nt23k123 # List your existing alarms for a specific entity to grab the alarm id # for the alarm you'd like to use PagerDuty notifications on. raxmon-alarms-list --entity-id=enKEb23JB # Update your alarm to use your new notification plan. raxmon-alarms-update --entity-id=enKEb23JB --id=alwp0UoI45 \ --notification-plan=npTY46f7