James Duffy

Terraform→PagerDuty Change Notifications

// By •

An important thing when you are setting up PagerDuty is to make sure that that the responder has as much context as possible to recent changes of the system. At PlushCare we use Terraform heavily and making sure that a responder to an incident can attribute to a recent change is extremely important in making sure that our uptime and SLOs are met.

We currently use Terraform Cloud to manage our Terraform state and planning/applying any changes pushed to GitHub. Below I will walk you through how we setup to send webhooks from Terraform Cloud to PagerDuty.

  1. Go to your Service Directory

    Image of PagerDuty Menu

  2. Click “+ Add Change Integration”

    Image of recent changes in service page

  3. Click “+ New Change Integration”

  4. Select “PagerDuty Integrations” and from the drop down select “Custom Change Event Transformer”

  5. Enter a name for your Integration. Something like “Terraform Cloud” is a good one.

  6. Select the service you will be adding this to

    Image of PagerDuty Integration form

  7. Click “Next”

  8. Click “View Integration”

  9. Click “Edit Integration”

  10. Take the code below and replace the boilerplate PagerDuty gave you to start with:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

    var data = PD.inputRequest.body;

var workspace_name = data.workspace_name ? data.workspace_name : "None";

var payload = {
  summary: workspace_name + " — " + data.notifications[0].message,
  source: data.notification_configuration_id,
  custom_details: {
    "notification": data.notifications,
    "Organization": data.organization_name,
    "Created At": data.run_created_at,
    "Created By": data.run_created_by,
    "Run ID": data.run_id,
    "Workspace ID": data.workspace_id,
    "Workspace Name": workspace_name,
  },
};

var links = data.run_url ? [{
  href: data.run_url,
  text: data.run_message
}] : null;

var changeEvent = {
  payload: payload,
  links: links
};

PD.emitChangeEvents([changeEvent]);

  11. Click “Save changes”

  12. Copy the Integration URL. It should look something like: https://events.pagerduty.com/integration/RANDOM_TEXT/change/enqueue

  13. Go to Terraform Cloud and the Workspace you would like to add the integration to

  14. Go to Settings > Notifications

Image of notification menu

  1. Click “Create a Notification”

  2. Below is my recommended configuration: Image of settings listed below

    • Name: “PagerDuty Change Events”
    • Webhook URL: Paste the URL you copied earlier
    • Token: Leave blank
    • Triggers: Only certain events
      • Completed
      • Errored

  3. Click “Create a Notification”

  4. You should see a Last Response section with a status of 202

    Image of successful last response

  5. Go back to PagerDuty service page and you should see the verification event in Recent Changes.

    Image of successful change event

Congratulations you should now receive change notifications from Terraform to your PagerDuty Service. One thing we also do is have this setup on many different PagerDuty services, but because we currently have a single “Production” workspace in Terraform we make sure to add multiple webhooks on the Terraform workspace to each service.