Route alerts with filters
Every alert has an ideal first responder: a team or person who knows how to triage and address the issue. Sensu contact routing lets you alert the right people using their preferred contact methods, reducing mean time to response and recovery.
In this guide, you’ll set up alerts for two teams (ops and dev) with separate Slack channels. Assume each team wants to be alerted only for the things they care about, using their team’s Slack channel. To achieve this, you’ll create two types of Sensu resources:
- Event handlers to store contact preferences for the ops team, the dev team, and a fallback option
- Event filters to match contact labels to the right handler
Here’s a quick overview of the configuration to set up contact routing.
The check definition includes the contacts: dev
label, which will result in an alert sent to the dev team but not to the ops team or the fallback option.
Sensu Go contact routing: Route alerts to the dev team using a check label
Prerequisites
To complete this guide, you’ll need:
- A Sensu backend
- At least one Sensu agent
- Sensuctl (configured to talk to the Sensu backend)
- cURL
- A Slack webhook URL and three different Slack channels to receive test alerts (one for each team)
To set up a quick testing environment, download and start the Sensu sandbox.
Configure contact routing
1. Register the has-contact filter asset
Contact routing is powered by the has-contact filter asset.
To add the has-contact asset to Sensu, use sensuctl asset add
:
sensuctl asset add sensu/sensu-go-has-contact-filter:0.2.0 -r contact-filter
This example uses the -r
(rename) flag to specify a shorter name for the asset: contact-filter
.
You can also download the latest asset definition from Bonsai.
Run sensuctl asset list --format yaml
to confirm that the asset is ready to use.
NOTE: Sensu does not download and install asset builds onto the system until they are needed for command execution. Read the asset reference for more information about asset builds.
2. Create contact filters
The Bonsai documentation for the asset explains that the has-contact asset supports two functions:
has_contact
, which takes the Sensu event and the contact name as argumentsno_contact
, which is available as a fallback in the absence of contact labels and takes only the event as an argument
You’ll use these functions to create event filters that represent the three actions that the Sensu Slack handler can take on an event: contact the ops team, contact the dev team, and contact the fallback option.
event filter name | expression | description |
---|---|---|
contact_ops |
has_contact(event, "ops") |
Allow events with the entity or check label contacts: ops |
contact_dev |
has_contact(event, "dev") |
Allow events with the entity or check label contacts: dev |
contact_fallback |
no_contacts(event) |
Allow events without an entity or check contacts label |
To add these filters to Sensu, use sensuctl create
:
echo '---
type: EventFilter
api_version: core/v2
metadata:
name: contact_ops
spec:
action: allow
runtime_assets:
- sensu-go-has-contact-filter_any_noarch
expressions:
- has_contact(event, "ops")
---
type: EventFilter
api_version: core/v2
metadata:
name: contact_dev
spec:
action: allow
runtime_assets:
- contact-filter
expressions:
- has_contact(event, "dev")
---
type: EventFilter
api_version: core/v2
metadata:
name: contact_fallback
spec:
action: allow
runtime_assets:
- contact-filter
expressions:
- no_contacts(event)' | sensuctl create
Run sensuctl filter list --format yaml
to confirm that the filters are ready to use.
3. Create a handler for each contact
With your contact filters in place, you can create a handler for each contact: ops, dev, and fallback. If you haven’t already, add the Slack handler asset to Sensu with sensuctl:
sensuctl asset add sensu/sensu-slack-handler:1.0.3 -r sensu-slack-handler
This example uses the -r
(rename) flag to specify a shorter name for the asset: sensu-slack-handler
.
In each handler definition, specify:
- A unique name:
slack_ops
,slack_dev
, orslack_fallback
- A customized command with the contact’s preferred Slack channel
- The contact filter
- The built-in
is_incident
andnot_silenced
filters to reduce noise and enable silences - An environment variable that contains your Slack webhook URL
- The
sensu-slack-handler
runtime asset
To create the slack_ops
, slack_dev
, and slack_fallback
handlers, edit and run this example:
# Edit before running:
# 1. Add your SLACK_WEBHOOK_URL
# 2. Make sure the Slack channels specified in the
# command` attributes match channels available
# to receive test alerts in your Slack instance.
echo '---
type: Handler
api_version: core/v2
metadata:
name: slack_ops
spec:
command: sensu-slack-handler --channel "#alert-ops"
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX
filters:
- is_incident
- not_silenced
- contact_ops
runtime_assets:
- sensu-slack-handler
type: pipe
---
type: Handler
api_version: core/v2
metadata:
name: slack_dev
spec:
command: sensu-slack-handler --channel "#alert-dev"
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX
filters:
- is_incident
- not_silenced
- contact_dev
runtime_assets:
- sensu-slack-handler
type: pipe
---
type: Handler
api_version: core/v2
metadata:
name: slack_fallback
spec:
command: sensu-slack-handler --channel "#alert-all"
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX
filters:
- is_incident
- not_silenced
- contact_fallback
runtime_assets:
- sensu-slack-handler
type: pipe' | sensuctl create
Run sensuctl handler list --format yaml
to confirm that the handlers are ready to use.
4. Create a handler set
To centralize contact management and simplify configuration, create a handler set that combines your contact-specific handlers under a single handler name.
Use sensuctl
to create a slack
handler set:
echo '---
type: Handler
api_version: core/v2
metadata:
name: slack
namespace: default
spec:
handlers:
- slack_ops
- slack_dev
- slack_fallback
type: set' | sensuctl create
You should see updated output of sensuctl handler list
that includes the slack
handler set.
Congratulations! Your Sensu contact routing is set up. Next, test your contact filters to make sure they work.
Test contact routing
To make sure your contact filters work the way you expect, use the agent API to create ad hoc events and send them to your Slack pipeline.
First, create an event without a contacts
label.
You may need to modify the URL with your Sensu agent address.
curl -X POST \
-H 'Content-Type: application/json' \
-d '{
"check": {
"metadata": {
"name": "example-check"
},
"status": 1,
"output": "You should receive this example event in the Slack channel specified by your slack_fallback handler.",
"handlers": ["slack"]
}
}' \
http://127.0.0.1:3031/events
You should see a 202 response from the API.
Since this event doesn’t include a contacts
label, you should also see an alert in the Slack channel specified by the slack_fallback
handler.
Behind the scenes, Sensu uses thecontact_fallback
filter to match the event to the slack_fallback
handler.
Now, create an event with a contacts
label:
curl -X POST \
-H 'Content-Type: application/json' \
-d '{
"check": {
"metadata": {
"name": "example-check",
"labels": {
"contacts": "dev"
}
},
"status": 1,
"output": "You should receive this example event in the Slack channel specified by your slack_dev handler.",
"handlers": ["slack"]
}
}' \
http://127.0.0.1:3031/events
Because this event contains the contacts: dev
label, you should see an alert in the Slack channel specified by the slack_dev
handler.
Resolve the events by sending the same API requests with status
set to 0
.
Manage contact labels in checks and entities
To assign an alert to a contact, add a contacts
label to the check or entity.
Checks
This check definition includes two contacts (ops
and dev
) and the handler slack
.
To add the runtime assets and set up the check_cpu
check, see Monitor server resources.
---
type: CheckConfig
api_version: core/v2
metadata:
name: check_cpu
labels:
contacts: ops, dev
spec:
command: check-cpu.rb -w 75 -c 90
handlers:
- slack
interval: 10
publish: true
subscriptions:
- system
runtime-assets:
- sensu-plugins-cpu-checks
- sensu-ruby-runtime
When the check_cpu
check generates an incident, Sensu filters the event according to the contact_ops
and contact_dev
filters, resulting in an alert sent to #alert-ops and #alert-dev.
Sensu Go contact routing: Route alerts to two contacts using a check label
Entities
You can also specify contacts using an entity label. For more information about managing entity labels, see the entity reference.
If contact labels are present in both the check and entity, the check contacts override the entity contacts.
In this example, the dev
label in the check configuration overrides the ops
label in the agent definition, resulting in an alert sent to #alert-dev but not to #alert-ops or #alert-all.
Sensu Go contact routing: Check contacts override entity contacts
Next steps
Now that you’ve set up contact routing for two example teams, you can create additional filters, handlers, and labels to represent your team’s contacts. Learn how to use Sensu to Reduce alert fatigue.