Using Contact Routing in Sensu Enterprise
Prerequisites
Before diving into this guide, we recommend having the following components ready:
- A working Sensu Enterprise deployment
- Two or more Sensu Enterprise integrations configured
If you’ve not already signed up for Sensu Enterprise, you can do so via this link.
In this guide, we’ll be using Slack and Email handlers.
Overview
Every incident, outage or event has an ideal first responder: a team or individual with the knowledge to triage and address the issue. Sensu Enterprise contact routing makes it possible to assign checks to specific teams and individuals, reducing mean time to response and recovery (MTTR). Contact routing works with all of the Sensu Enterprise third-party notification and metric integrations.
NOTE: Sensu Enterprise supports contact routing for all integrations except EC2, Puppet, Chef, Flapjack, and Event Stream.
In this guide we’ll cover configuring and using Sensu Enterprise Contact Routing.
Contact Routing Basics
In Sensu Enterprise Contact Routing, contacts are composed of a name and
configuration overrides for one or more of Sensu Enterprise’s built-in
integrations or handler configurations. A contact in Sensu Enterprise is
similar to a contact on your phone, in which it has a name and one or
more identifiers for various communication channels (e.g. phone
number, email address, etc). As an example, a Sensu contact may have the following attributes: email address, slack channel, pagerduty service_key, etc.
Contact Configuration
Configuring a contact requires you to define the name of the contact and the
services plus configuration override(s) for that contact under the contacts scope.
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.com"
},
"slack": {
"channel": "#support"
}
}
}
}In the example above we have a contact named support. For support we
have an override configuration for email and slack. In our example we’re having
this contact email support@sensuapp.com instead of the email integration’s
default to address. For Slack, we’re defining the channel the event should
post to, in this case #support.
Check Configuration
Once a contact has been defined, we can now specify which contact(s) to use in a check configuration. The use case for having a contact defined on a check could be that the check is owned by a particular team or group. For instance:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email"
],
"contacts": [
"support"
]
}
}
}Client Configuration
A client definition may include a list of contacts. Providing a list of contacts in the client definition ensures that those contacts receive notifications for check results which do not explicitly define a list of contacts.
NOTE: Contacts specified in a check definition override contacts specified in the client definition for any check results generated by that client/check pair.
{
"client": {
"name": "ec2-west-id-12345",
"address": "8.8.8.8",
"subscriptions": [
"production",
"aws"
],
"contact": "aws-team"
}
}Example Implementations
The following section describes different scenarios and illustrates how contact routing affects notifications being sent in each. We’ll be using Email, Slack and PagerDuty as our example integrations.
Global Configuration
Below is the base configuration for the integrations we’ll be working with in our examples:
{
"slack": {
"webhook_url": "https://hooks.slack.com/services/foo/bar/foobar",
"username": "sensu",
"channel": "#alerts",
"timeout": 10
},
"email": {
"smtp": {
"address": "smtp.example.com",
"port": 587,
"openssl_verify_mode": "none",
"enable_starttls_auto": true,
"authentication": "plain",
"user_name": "postmaster@example.com",
"password": "SECRET"
},
"to": "default@example.com",
"from": "noreply@example.com",
"timeout": 10
},
"pagerduty": {
"service_key": "examplekey ",
"timeout": 10
}
}Single Handler
Single Handler with a Single Matching Contact
For our first configuration we have a single handler email and a contact support configured. The support contact has a configuration override for email that will change the default to email address from “default@example.com” to “support@sensuapp.com”.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email"
],
"contacts": [
"support"
]
}
}
}Contact configuration:
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.com"
}
}
}
}Because the support contact’s configuration provides overrides matching the handler being used, any event generated for this check will use this contact’s configuration.
Single Handler with a Single Non-Matching Contact
This configuration has a single handler email and a contact support configured. The support contact does not have a configuration override for email.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email"
],
"contacts": [
"support"
]
}
}
}Contact configuration:
{
"contacts": {
"support": {
"pagerduty": {
"service_key": "foobarkey"
}
}
}
}Although the contact support is defined under contacts and specified correctly in the check definition, the contact does not provide configuration for the email integration. This means no change will be made to the default email configuration. In this case, an email will be sent to “default@example.com”.
Single Handler with Multiple Matching Contacts
In this configuration we have a single handler, email, and multiple contacts, support and dev, configured. The support and dev contacts have a configuration override for email handler to send their emails to “support@sensuapp.io” and “dev@sensuapp.io” respectively.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email"
],
"contacts": [
"support",
"dev"
]
}
}
}Contact configurations:
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.io"
}
},
"dev": {
"email": {
"to": "dev@sensuapp.io"
}
}
}
}In this instance, although we are using only one handler email, Sensu runs the handler multiple times to send an email to “support@sensuapp.io” and to “dev@sensuapp.io”.
Single Handler with Multiple Non-Matching Contacts
For this configuration we have a single handler, email, and multiple contacts, support and dev, configured. The support contact has an override configuration for slack and the dev contact has an override configuration for pagerduty, neither contact has an override configuration for email.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email"
],
"contacts": [
"support",
"dev"
]
}
}
}Contact configurations:
{
"contacts": {
"support": {
"slack": {
"channel": "#support"
}
},
"dev": {
"pagerduty": {
"service_key": "foobarkey"
}
}
}
}In this instance, since the contacts defined do not have an override configuration for email, the default configuration for email will be used. In our example an email will be sent to “default@example.com”
Single Handler with Some Matching Contacts
In this configuration we have a single handler, email, and multiple contacts, support, dev and eng, configured. The support contact has an override configuration for email, the dev contact has an override configuration for pagerduty, and the eng contact has an override configuration for slack.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email"
],
"contacts": [
"support",
"dev"
]
}
}
}Contact configurations:
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.io"
}
},
"dev": {
"pagerduty": {
"service_key": "foobarkey"
}
},
"eng": {
"slack": {
"channel": "#engineering"
}
}
}
}In this instance, Sensu generates two emails: one email to the support contact email at “support@sensuapp.io” and the second email to our global configuration email at “default@example.com”.
The reason for this is that since we have one or more contacts that do not have a configuration override for email, Sensu uses the default configuration.
Multiple Handlers
Multiple Handlers with a Matching Contact
In this configuration we have two handlers, email and slack, and a contact, support, configured. The support contact has a configuration override for email that changes the default to email address from “default@example.com” to “support@sensuapp.io” and a configuration override for slack to change the channel from “#alerts” to “#support”
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email",
"slack"
],
"contacts": [
"support"
]
}
}
}Contact configuration:
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.com"
},
"slack": {
"channel": "#support"
}
}
}
}Because the contact configuration matches both handlers being used, any event generated for this check uses this contact’s configuration.
Multiple Handlers with a Single Matching Contact
In this configuration we have two handlers,email and slack, and a contact, support, configured. The support contact only has a configuration override for email that changes the default to email address from “default@example.com” to “support@sensuapp.com”. The support contact does not have a handler override for the slack handler.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email",
"slack"
],
"contacts": [
"support"
]
}
}
}Contact configuration:
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.com"
}
}
}
}Because the contact configuration only has a configuration override for the email handler, Sensu overrides only the email handler and uses the default configuration for slack.
Multiple Handlers with a Single Non-Matching Contact
Similar to the previous configuration, we have two handlers, email and slack, and a contact, support, configured. The support contact has a configuration override for pagerduty that changes the default service_key. The support contact does not have a handler override for the slack or email handler.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email",
"slack"
],
"contacts": [
"support"
]
}
}
}Contact configuration:
{
"contacts": {
"support": {
"pagerduty": {
"service_key": "foobarkey"
}
}
}
}Since the contact does not have a configuration override for email or slack, Sensu uses the default configuration for those handlers instead.
Multiple Handlers with Multiple Matching Contacts
In this example we have two handlers configured for our check, email and slack. We have two contacts being used, support and dev, and both contacts have configuration override for both email and slack.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email",
"slack"
],
"contacts": [
"support",
"dev"
]
}
}
}Contact configurations:
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.io"
},
"slack": {
"channel": "#support"
}
},
"dev": {
"email": {
"to": "dev@sensuapp.io"
},
"slack": {
"channel": "#dev"
}
}
}
}Since both contacts match and both contacts have configuration overrides for both handlers, four handler events are generated. Two emails will be sent, one for the support contact to “support@sensuapp.io” and one to the dev contact to “dev@sensuapp.io”. The same is true for the slack handler with the support contact creating a message in the “#support” channel and the dev contact creating a message in the “#dev” channel.
Multiple Handlers with Multiple Non-Matching Contacts
In this example we have two handlers configured for our check, email and slack. We have two contacts being used, support and dev; both contacts have a configuration override for pagerduty, but do not include any override for slack or email.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email",
"slack"
],
"contacts": [
"support",
"dev"
]
}
}
}Contact configurations:
{
"contacts": {
"support": {
"pagerduty": {
"service_key": "supportfoobarkey"
}
},
"dev": {
"pagerduty": {
"service_key": "devfoobarkey"
}
}
}
}Since both contacts match and neither contact provides configuration overrides for either handler, Sensu generates two handler events and uses the default configuration for both.
Multiple Handlers with Some Matching Contacts
In this example we have two handlers configured for our check, email and slack. We have three contacts being used, support, dev and eng. Support has a contact override for email; dev has a contact override for slack; eng has a contact override for pagerduty which is not a handler that the check is using.
Check configuration:
{
"checks": {
"example_check": {
"command": "example_script.rb",
"handlers": [
"email",
"slack"
],
"contacts": [
"support",
"dev",
"eng"
]
}
}
}Contact configurations:
{
"contacts": {
"support": {
"email": {
"to": "support@sensuapp.io"
}
},
"dev": {
"slack": {
"channel": "#dev"
}
},
"eng": {
"pagerduty": {
"service_key": "foobarkey"
}
}
}
}With some contacts providing at least one applicable settings override, we expect Sensu to generate four different handler events.
For the email handler, Sensu sends an email to the support contact override, “support@sensuapp.io”. Because neither the dev nor eng contacts have an override for email, Sensu sends a single additional email to our default email, “default@example.com”.
For the slack handler, similar to our email handler, Sensu creates a message in the #dev channel for our dev contact. Because neither the support or eng contacts provide an override for slack, Sensu sends a single additional notification to our default slack channel, “#alerts”.
Wrapping Up
Contact Routing and Sensu Event Pipeline
When a check is executed and an event is generated, Sensu sends a copy of of the event to each defined handler, allowing the workflow for each handler to be managed independently.
Each handler evaluates the event to determine which contacts should be notified. For each contact defined in the event, the handler generates a notification using either the contact’s override configuration or the default handler configuration. For events which define multiple contacts without applicable overrides, only a single notification is generated using the default handler configuration.
Resources
For further reference please see our Contact Routing documentation. If you find any issues or question, feel free to reach out in our Community Slack, or open an issue on Github.
