Send email alerts with a pipeline

PRO TIP: You can use the Email Alerts integration in the Sensu Catalog to send email alerts based on Sensu event data instead of following this guide. Follow the Catalog prompts to configure the Sensu resources you need and start processing your observability data with a few clicks.

Pipelines are Sensu resources composed of observation event processing workflows that include filters, mutators, and handlers. You can use pipelines to send email alerts, create or resolve incidents (in PagerDuty, for example), or store metrics in a time-series database like InfluxDB.

When you are using Sensu in production, events will come from a check or metric you configure. For this guide, you will create an ad hoc event that you can trigger manually to test your email handler.

Your backend will execute a pipeline with a handler that sends notifications to the email address you specify. The pipeline will also include an event filter to make sure you only receive a notification when your event represents a status change.

Requirements

To follow this guide, install the Sensu backend, make sure at least one Sensu agent is running, and configure sensuctl to connect to the backend as the admin user.

Add the email handler dynamic runtime asset

Dynamic runtime assets are shareable, reusable packages that help you deploy Sensu plugins. In this guide, you’ll use the sensu/sensu-email-handler dynamic runtime asset to power an email handler.

Use the following sensuctl example to register the sensu/sensu-email-handler dynamic runtime asset:

sensuctl asset add sensu/sensu-email-handler:1.2.2 -r email-handler

The response will confirm that the asset was added:

fetching bonsai asset: sensu/sensu-email-handler:1.2.2
added asset: sensu/sensu-email-handler:1.2.2

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["email-handler"].

The -r (rename) flag allows you to specify a shorter name for the dynamic runtime asset (in this case, email-handler).

To confirm that the handler dynamic runtime asset was added correctly, run:

sensuctl asset list

The list should include the email-handler dynamic runtime asset. For a detailed list of everything related to the asset that Sensu added automatically, run:

sensuctl asset info email-handler

The sensu/sensu-email-handler dynamic runtime asset includes the sensu-email-handler command, which you will use when you create the email handler definition later in this guide.

NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution.

Add an event filter

Event filters allow you to fine-tune how your events are handled and reduce alert fatigue. In this guide, your event filter will send notifications only when your event’s state changes (for example, for any change between 0 OK, 1 warning, and 2 critical).

Here’s an overview of how the state_change_only filter will work:

  • If your event status changes from 0 to 1, you will receive one email notification for the change to warning status.
  • If your event status stays at 1 for the next hour, you will not receive repeated email notifications during that hour.
  • If your event status changes to 2 after 1 hour at 1, you will receive one email notification for the change from warning to critical status.
  • If your event status fluctuates between 0, 1, and 2 for the next hour, you will receive one email notification each time the status changes.

To create the event filter, run:

cat << EOF | sensuctl create
---
type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: state_change_only
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1
  runtime_assets: []
EOF
cat << EOF | sensuctl create
{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "annotations": null,
    "labels": null,
    "name": "state_change_only"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.occurrences == 1"
    ],
    "runtime_assets": [

    ]
  }
}
EOF

Create the email handler definition

After you add an event filter, create the email handler definition to specify the email address where the handler will send notifications. In the handler definition’s command value, you’ll need to change a few things:

  • <sender@example.com>: Replace with the email address you want to use to send email alerts.
  • <recipient@example.com>: Replace with the email address you want to receive email alerts.
  • <smtp_server@example.com>: Replace with the hostname of your SMTP server.
  • <username>: Replace with your SMTP username, typically your email address.
  • <password>: Replace with your SMTP password, typically the same as your email password.

NOTE: To use Gmail or G Suite as your SMTP server, follow Google’s instructions to send email via SMTP. If you have enabled 2-step verification on your Google account, use an app password instead of your login password. If you have not enabled 2-step verification, you may need to adjust your app access settings to follow the example in this guide.

After you update the command with your email, server, username, and password values in the example below, run the updated code to create the email handler definition:

cat << EOF | sensuctl create
---
api_version: core/v2
type: Handler
metadata:
  name: email
spec:
  type: pipe
  command: sensu-email-handler -f <sender@example.com> -t <recipient@example.com> -s <smtp_server@example.com> -u username -p password
  timeout: 10
  runtime_assets:
  - email-handler
EOF
cat << EOF | sensuctl create
{
  "api_version": "core/v2",
  "type": "Handler",
  "metadata": {
    "name": "email"
  },
  "spec": {
    "type": "pipe",
    "command": "sensu-email-handler -f <sender@example.com> -t <recipient@example.com> -s <smtp_server@example.com> -u username -p password",
    "timeout": 10,
    "runtime_assets": [
      "email-handler"
    ]
  }
}
EOF

Create a pipeline

With your event filter and handler configured, you can build a pipeline workflow. A single pipeline workflow can include one or more filters, one mutator, and one handler.

In this case, the pipeline includes your state_change_only event filter and email handler, as well as two built-in event filters, is_incident and not_silenced. These two built-in filters are included in every Sensu backend installation, so you don’t have to create them. The is_incident and not_silenced event filters ensure that you receive alerts for unsilenced events with only warning (1) or critical (2) status:

To create the pipeline, run:

cat << EOF | sensuctl create
---
type: Pipeline
api_version: core/v2
metadata:
  name: alerts_pipeline
spec:
  workflows:
  - name: email_alerts
    filters:
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: not_silenced
      type: EventFilter
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
EOF
cat << EOF | sensuctl create
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "alerts_pipeline"
  },
  "spec": {
    "workflows": [
      {
        "name": "email_alerts",
        "filters": [
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "not_silenced",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
EOF

Before your pipeline can send alerts to your email, you need an event that generates the alerts. In the final step, you will create an ad hoc event that you can trigger manually.

Create and trigger an ad hoc event

To create an ad hoc event, first use sensuctl env to set up environment variables. The environment variables will provide the required Sensu API access token credential for the Sensu API:

eval $(sensuctl env)

Verify that the SENSU_ACCESS_TOKEN environment variable is set by echoing its value:

echo $SENSU_ACCESS_TOKEN

The response will list the SENSU_ACCESS_TOKEN value.

With the environment variables set, you can use the Sensu API to create your ad hoc observability event.

NOTE: The example events use the default namespace. If you are using a different namespace, replace default in the event definitions and the API URLs with the name of the desired namespace.

This event outputs the message “Everything is OK.” when it occurs:

curl -sS -H 'Content-Type: application/json' \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "Everything is OK.",
    "status": 0
  }
}' \
http://localhost:8080/api/core/v2/namespaces/default/events

As configured, the event status is 0 (OK). Now it’s time to trigger an event and view the results!

To generate a status change event, use the update event endpoint to create a 1 (warning) event. Run:

curl -sS -X PUT \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "This is a warning.",
    "status": 1
  },
  "pipelines": [
    {
      "type": "Pipeline",
      "api_version": "core/v2",
      "name": "alerts_pipeline"
    }
  ]
}' \
http://localhost:8080/api/core/v2/namespaces/default/events/server01/server-health

NOTE: If you receive an invalid credentials error, refresh your token. Run eval $(sensuctl env).

Check your email — you should receive a message from Sensu!

Create another event with status set to 0. Run:

curl -sS -X PUT \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "Everything is OK.",
    "status": 0
  },
  "pipelines": [
    {
      "type": "Pipeline",
      "api_version": "core/v2",
      "name": "alerts_pipeline"
    }
  ]
}' \
http://localhost:8080/api/core/v2/namespaces/default/events/server01/server-health

You should receive another email because the event status changed to 0 (OK).

Next steps

Now that you know how to apply a pipeline to a check and take action on events, reuse this email pipeline with the check_cpu check from our Monitor server resources guide. Check out Route alerts with event filters for a complex pipeline example that includes several workflows with different event filters and handlers.

The sensu/sensu-email-handler dynamic runtime asset makes it possible to add a template that provides context for your email notifications. The email template functionality uses tokens to populate the values provided by the event, and you can use HTML to format the email.