Route alerts with event filters

Every alert has an ideal first responder: a team or person who knows how to triage and address the issue. Sensu contact routing lets you alert the right people using their preferred contact methods, reducing mean time to response and recovery.

In this guide, you’ll set up alerts for two teams (ops and dev) with separate Slack channels. Assume each team wants to be alerted only for the things they care about, using their team’s Slack channel. To achieve this, you’ll create two types of Sensu resources:

Event handlers to store contact preferences for the ops team, the dev team, and a fallback option
Event filters to match contact labels to the right handler

Here’s a quick overview of the configuration to set up contact routing. The check definition includes the contacts: dev label, which will result in an alert sent to the dev team but not to the ops team or the fallback option.

Sensu Go contact routing: Route alerts to the dev team using a check label

Prerequisites

To complete this guide, you’ll need:

A Sensu backend
At least one Sensu agent
Sensuctl (configured to talk to the Sensu backend)
cURL
A Slack webhook URL and three different Slack channels to receive test alerts (one for each team)

To set up a quick testing environment, download and start the Sensu sandbox.

Configure contact routing

1. Register the has-contact filter dynamic runtime asset

Contact routing is powered by the has-contact filter dynamic runtime asset. To add the has-contact dynamic runtime asset to Sensu, use sensuctl asset add:

sensuctl asset add sensu/sensu-go-has-contact-filter:0.2.0 -r contact-filter

The response will indicate that the asset was added:

fetching bonsai asset: sensu/sensu-go-has-contact-filter:0.2.0
added asset: sensu/sensu-go-has-contact-filter:0.2.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["contact-filter"].

This example uses the -r (rename) flag to specify a shorter name for the asset: contact-filter.

You can also download the latest dynamic runtime asset definition from Bonsai.

Run sensuctl asset list --format yaml to confirm that the dynamic runtime asset is ready to use.

NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution. Read the asset reference for more information about dynamic runtime asset builds.

2. Create contact filters

The Bonsai documentation for the asset explains that the has-contact dynamic runtime asset supports two functions:

has_contact, which takes the Sensu event and the contact name as arguments
no_contact, which is available as a fallback in the absence of contact labels and takes only the event as an argument

You’ll use these functions to create event filters that represent the three actions that the Sensu Slack handler can take on an event: contact the ops team, contact the dev team, and contact the fallback option.

event filter name	expression	description
`contact_ops`	`has_contact(event, "ops")`	Allow events with the entity or check label `contacts: ops`
`contact_dev`	`has_contact(event, "dev")`	Allow events with the entity or check label `contacts: dev`
`contact_fallback`	`no_contacts(event)`	Allow events without an entity or check `contacts` label

Use sensuctl to create the three event filters:

echo '---
type: EventFilter
api_version: core/v2
metadata:
  name: contact_ops
spec:
  action: allow
  runtime_assets:
    - sensu-go-has-contact-filter_any_noarch
  expressions:
    - has_contact(event, "ops")
---
type: EventFilter
api_version: core/v2
metadata:
  name: contact_dev
spec:
  action: allow
  runtime_assets:
    - contact-filter
  expressions:
    - has_contact(event, "dev")
---
type: EventFilter
api_version: core/v2
metadata:
  name: contact_fallback
spec:
  action: allow
  runtime_assets:
    - contact-filter
  expressions:
    - no_contacts(event)' | sensuctl create

echo '{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "contact_ops"
  },
  "spec": {
    "action": "allow",
    "runtime_assets": [
      "sensu-go-has-contact-filter_any_noarch"
    ],
    "expressions": [
      "has_contact(event, \"ops\")"
    ]
  }
}
{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "contact_dev"
  },
  "spec": {
    "action": "allow",
    "runtime_assets": [
      "contact-filter"
    ],
    "expressions": [
      "has_contact(event, \"dev\")"
    ]
  }
}
{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "contact_fallback"
  },
  "spec": {
    "action": "allow",
    "runtime_assets": [
      "contact-filter"
    ],
    "expressions": [
      "no_contacts(event)"
    ]
  }
}' | sensuctl create

You can also save these event filter resource definitions to a file named filters.yml or filters.json in your Sensu installation. When you’re ready to manage your observability configurations the same way you do any other code, your filters.yml or filters.json file can become a part of your monitoring as code repository.

Use sensuctl to confirm that the event filters were added:

sensuctl filter list

The response should list the new contact_ops, contact_dev, and contact_fallback event filters:

        Name         Action           Expressions          
 ────────────────── ──────── ───────────────────────────── 
  contact_dev        allow    (has_contact(event, "dev"))  
  contact_fallback   allow    (no_contacts(event))         
  contact_ops        allow    (has_contact(event, "ops"))

3. Create a handler for each contact

With your contact filters in place, you can create a handler for each contact: ops, dev, and fallback. If you haven’t already, add the Slack handler dynamic runtime asset to Sensu with sensuctl:

sensuctl asset add sensu/sensu-slack-handler:1.0.3 -r sensu-slack-handler

The response will confirm that the asset was added:

fetching bonsai asset: sensu/sensu-slack-handler:1.0.3
added asset: sensu/sensu-slack-handler:1.0.3

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-slack-handler"].

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sensu-slack-handler.

In each handler definition, you will specify:

A unique name: slack_ops, slack_dev, or slack_fallback
A customized command with the contact’s preferred Slack channel
The contact filter
The built-in is_incident and not_silenced filters to reduce noise and enable silences
An environment variable that contains your Slack webhook URL
The sensu-slack-handler dynamic runtime asset

Before you run the following code to create the handlers with sensuctl, make these changes:

Replace #alert-ops, #alert-dev, and #alert-all with the names of the channels you want to use to receive alerts in your Slack instance.
Replace SLACK_WEBHOOK_URL with your Slack webhook URL.

After you update the code to use your preferred Slack channels and webhook URL, run:

echo '---
type: Handler
api_version: core/v2
metadata:
  name: slack_ops
spec:
  command: sensu-slack-handler --channel "#alert-ops"
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX
  filters:
  - is_incident
  - not_silenced
  - contact_ops
  runtime_assets:
  - sensu-slack-handler
  type: pipe
---
type: Handler
api_version: core/v2
metadata:
  name: slack_dev
spec:
  command: sensu-slack-handler --channel "#alert-dev"
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX
  filters:
  - is_incident
  - not_silenced
  - contact_dev
  runtime_assets:
  - sensu-slack-handler
  type: pipe
---
type: Handler
api_version: core/v2
metadata:
  name: slack_fallback
spec:
  command: sensu-slack-handler --channel "#alert-all"
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX
  filters:
  - is_incident
  - not_silenced
  - contact_fallback
  runtime_assets:
  - sensu-slack-handler
  type: pipe' | sensuctl create

echo '{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack_ops"
  },
  "spec": {
    "command": "sensu-slack-handler --channel \"#alert-ops\"",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX"
    ],
    "filters": [
      "is_incident",
      "not_silenced",
      "contact_ops"
    ],
    "runtime_assets": [
      "sensu-slack-handler"
    ],
    "type": "pipe"
  }
}
{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack_dev"
  },
  "spec": {
    "command": "sensu-slack-handler --channel \"#alert-dev\"",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX"
    ],
    "filters": [
      "is_incident",
      "not_silenced",
      "contact_dev"
    ],
    "runtime_assets": [
      "sensu-slack-handler"
    ],
    "type": "pipe"
  }
}
{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack_fallback"
  },
  "spec": {
    "command": "sensu-slack-handler --channel \"#alert-all\"",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX"
    ],
    "filters": [
      "is_incident",
      "not_silenced",
      "contact_fallback"
    ],
    "runtime_assets": [
      "sensu-slack-handler"
    ],
    "type": "pipe"
  }
}' | sensuctl create

Just like the event filters, you can save these handlers to a YAML or JSON file to create a handlers configuration file if you’re implementing monitoring as code.

Use sensuctl to confirm that the handlers were added:

sensuctl handler list

The response should list the new slack_ops, slack_dev, and slack_fallback handlers:

       Name        Type   Timeout                    Filters                    Mutator                        Execute                                                Environment Variables                                    Assets         
 ──────────────── ────── ───────── ─────────────────────────────────────────── ───────── ─────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────── ───────────────────── 
  slack_dev        pipe         0   is_incident,not_silenced,contact_dev                  RUN:  sensu-slack-handler --channel "#alert-dev"    SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX      sensu-slack-handler  
  slack_fallback   pipe         0   is_incident,not_silenced,contact_fallback             RUN:  sensu-slack-handler --channel "#alert-all"    SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX      sensu-slack-handler  
  slack_ops        pipe         0   is_incident,not_silenced,contact_ops                  RUN:  sensu-slack-handler --channel "#alert-ops"    SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX      sensu-slack-handler

4. Create a handler set

To centralize contact management and simplify configuration, create a handler set that combines your contact-specific handlers under a single handler name, slack:

echo '---
type: Handler
api_version: core/v2
metadata:
  name: slack
  namespace: default
spec:
  handlers:
  - slack_ops
  - slack_dev
  - slack_fallback
  type: set' | sensuctl create

echo '{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack",
    "namespace": "default"
  },
  "spec": {
    "handlers": [
      "slack_ops",
      "slack_dev",
      "slack_fallback"
    ],
    "type": "set"
  }
}' | sensuctl create

Run sensuctl handler list again. The updated output should include the slack handler set:

       Name        Type   Timeout                    Filters                    Mutator                       Execute                                                Environment Variables                                  Assets         
 ──────────────── ────── ───────── ─────────────────────────────────────────── ───────── ────────────────────────────────────────────────── ──────────────────────────────────────────────────────────────────────── ───────────────────── 
  slack            set          0                                                         CALL: slack_ops,slack_dev,slack_fallback                                                                                                         
  slack_dev        pipe         0   is_incident,not_silenced,contact_dev                  RUN:  sensu-slack-handler --channel "#alert-dev"   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX   sensu-slack-handler  
  slack_fallback   pipe         0   is_incident,not_silenced,contact_fallback             RUN:  sensu-slack-handler --channel "#alert-all"   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX   sensu-slack-handler  
  slack_ops        pipe         0   is_incident,not_silenced,contact_ops                  RUN:  sensu-slack-handler --channel "#alert-ops"   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX   sensu-slack-handler

Congratulations! Your Sensu contact routing is set up. Next, test your contact filters to make sure they work.

Test contact routing

To make sure your contact filters work the way you expect, use the agent API to create ad hoc events and send them to your Slack pipeline.

First, create an event without a contacts label. You may need to modify the URL with your Sensu agent address.

curl -X POST \
-H 'Content-Type: application/json' \
-d '{
  "check": {
    "metadata": {
      "name": "example-check"
    },
    "status": 1,
    "output": "You should receive this example event in the Slack channel specified by your slack_fallback handler.",
    "handlers": ["slack"]
  }
}' \
http://127.0.0.1:3031/events

You should see a 202 response from the API. Since this event doesn’t include a contacts label, you should also see an alert in the Slack channel specified by the slack_fallback handler. Behind the scenes, Sensu uses thecontact_fallback filter to match the event to the slack_fallback handler.

Now, create an event with a contacts label:

curl -X POST \
-H 'Content-Type: application/json' \
-d '{
  "check": {
    "metadata": {
      "name": "example-check",
      "labels": {
        "contacts": "dev"
      }
    },
    "status": 1,
    "output": "You should receive this example event in the Slack channel specified by your slack_dev handler.",
    "handlers": ["slack"]
  }
}' \
http://127.0.0.1:3031/events

Because this event contains the contacts: dev label, you should see an alert in the Slack channel specified by the slack_dev handler.

Resolve the events by sending the same API requests with status set to 0.

Manage contact labels in checks and entities

To assign an alert to a contact, add a contacts label to the check or entity. The contacts labels should be ops and dev. You’ll also need to update the check to use your slack handler.

For example, you can update the check_cpu check created in Monitor server resources to include the ops and dev contacts and the slack handler.

Use sensuctl to open the check in a text editor:

sensuctl edit check check_cpu

Edit the check metadata to add the following labels:

---
labels:
  contacts: ops, dev

{
  "labels": {
    "contacts": "ops, dev"
  }
}

Save and close the updated check definition. A response will confirm the check was updated. For example:

Updated /api/core/v2/namespaces/default/checks/check_cpu

Next, run this sensuctl command to add the slack handler:

sensuctl check set-handlers check_cpu slack

Again, you will see an Updated confirmation message.

To view the updated resource definition for check_cpu and confirm that it includes the contacts labels and slack handler, run:

sensuctl check info check_cpu --format yaml

sensuctl check info check_cpu --format wrapped-json

The sensuctl response will include the updated check_cpu resource definition in the specified format:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  labels:
    contacts: ops, dev
  name: check_cpu
  namespace: default
spec:
  check_hooks: null
  command: check-cpu.rb -w 75 -c 90
  env_vars: null
  handlers:
  - slack
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - cpu-checks-plugins
  - sensu-ruby-runtime
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "created_by": "admin",
    "labels": {
      "contacts": "ops, dev"
    },
    "name": "check_cpu",
    "namespace": "default"
  },
  "spec": {
    "check_hooks": null,
    "command": "check-cpu.rb -w 75 -c 90",
    "env_vars": null,
    "handlers": [
      "slack"
    ],
    "high_flap_threshold": 0,
    "interval": 60,
    "low_flap_threshold": 0,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "cpu-checks-plugins",
      "sensu-ruby-runtime"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "system"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

Now when the check_cpu check generates an incident, Sensu will filter the event according to the contact_ops and contact_dev event filters and send alerts to #alert-ops and #alert-dev accordingly.

Sensu Go contact routing: Route alerts to two contacts using a check label

Entities

You can also specify contacts using an entity label. For more information about managing entity labels, see the entity reference.

If contact labels are present in both the check and entity, the check contacts override the entity contacts. In this example, the dev label in the check configuration overrides the ops label in the agent definition, resulting in an alert sent to #alert-dev but not to #alert-ops or #alert-all.

Sensu Go contact routing: Check contacts override entity contacts

Next steps

Now that you’ve set up contact routing for two example teams, you can create additional filters, handlers, and labels to represent your team’s contacts. Learn how to use Sensu to Reduce alert fatigue.

Route alerts with event filters

Prerequisites

Configure contact routing

1. Register the has-contact filter dynamic runtime asset

2. Create contact filters

3. Create a handler for each contact

4. Create a handler set

Test contact routing

Manage contact labels in checks and entities

Entities

Next steps

Register your Sensu installation — Get rewards!

About Sensu

Navigation

Resources

Community

Get In Touch

Join Our Community