How to send alerts to Slack with handlers
What are Sensu handlers?
Sensu event handlers are actions executed by the Sensu server on events.
Why use a handler?
Handlers can be used for sending an email alert, creating or resolving an incident (in PagerDuty, for example), or storing metrics in a time-series database (InfluxDB, for example).
Using a handler to send alerts to Slack
The purpose of this guide is to help you send alerts to Slack, on the channel
monitoring, by configuring a handler named
slack to a check named
check-cpu. If you don’t already have a check in place, this guide is a
great place to start.
Installing the handler command
The first step is to create an executable script named
slack-handler, which is
responsible for sending the event data to Slack. You can download a release of
this handler from GitHub, then extract it by running:
sudo tar -C /usr/local/bin -xzf REPLACE-WITH-DOWNLOAD-FILENAME
Alternatively, you can compile or cross compile the handler from the source code
using the Go tools. The generated binary will be placed into one of the
$PATH directories, more precisely
# From the local path of the slack-handler repository go build -o /usr/local/bin/slack-handler main.go
Getting a Slack webhook
If you’re already an admin of a Slack, visit
https://YOUR WORKSPACE NAME HERE.slack.com/services/new/incoming-webhook and follow the steps to add the Incoming WebHooks integration, choose a channel, and save the settings.
(If you’re not yet a Slack admin, start here to create a new workspace.)
After saving, you’ll see your webhook URL under Integration Settings.
Creating the handler
Now that our handler command is installed, the second step is to create a
handler that we will call
slack, which is a pipe handler that pipes event
data into our previous script named
slack-handler. We’ll also use environment variables
to pass the Slack webhook URL to this script. Finally, in
order to avoid silenced events from being sent to Slack, we will use the
not_silenced built-in filter, in addition to the
is_incident built-in filter
so zero status events are also discarded.
sensuctl handler create slack \ --type pipe \ --env-vars "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX" \ --command "sensu-slack-handler --channel '#monitoring'" \ --filters is_incident,not_silenced
Assigning the handler to a check
slack handler now created, it can be assigned to a check. Here, since
we want to receive Slack alerts whenever the CPU usage of our systems reach some
specific thresholds, we will apply our handler to the check
sensuctl check set-handlers check-cpu slack
Validating the handler
It might take a few moments, once the handler is assigned to the check, for the check to be scheduled on the entities and the result sent back to Sensu backend, but once an event is handled, you should see the following message in Slack.
Otherwise, you can verify the proper behavior of this handler by using
sensu-backend logs. The default location of these logs varies based on the
platform used, but the installation and configuration documentation
provides this information.
Whenever an event is being handled, a log entry is added with the message
"handler":"slack","level":"debug","msg":"sending event to handler", followed
by a second one with the message
"msg":"pipelined executed event pipe
You now know how to apply a handler to a check and take action on events. From this point, here are some recommended resources: