How to plan maintenance windows using silencing
What is Sensu silencing?
As check results are processed by a Sensu server, the server executes event
handlers to send alerts to personnel or otherwise relay event data to
external services. Sensu’s built-in silencing, along with the built-in
not_silenced filter, provides the means to suppress execution of event
handlers on an ad hoc basis.
When to use silencing
Silencing is used to prevent handlers configured with the
from being triggered based on the check name present in a check result or the
subscriptions associated with the entity that published the check result. This
can be desirable in many scenarios, giving operators the ability to quiet
incoming alerts while coordinating their response.
Sensu silencing entries make it possible to:
- Silence all checks on a specific entity
- Silence a specific check on a specific entity
- Silence all checks on entities with a specific subscription
- Silence a specific check on entities with a specific subscription
- Silence a specific check on every entity
Using silencing to plan maintenance
The purpose of this guide is to help you plan a maintenance window, by creating
a silenced entry for a specific entity named
i-424242 and its check named
check-http, in order to prevent alerts as you restart or redeploy the
services associated with this entity.
Creating the silenced entry
The first step is to create a silenced entry that will silence the check
check-http on an entity named
i-424242, for a planned maintenance window
that starts at 01:00, on Sunday, and ends 1 hour later. Your
username will automatically be added as the creator of the silenced entry.
sensuctl silenced create \ --subscription 'entity:i-424242' \ --check 'check-http' \ --begin '2018-03-16 01:00:00 -04:00' \ --expire 3600 \ --reason 'Server upgrade'
See the sensuctl documentation for the supported time formats in the
Validating the silenced entry
You can verify that the silenced entry against our entity, here named
i-424242, has been properly created, by using
sensuctl silenced info \ --subscription 'entity:i-424242:check-http'
Once the silenced entry starts to take effect, events that are silenced will be
marked as so in
$ sensuctl event list Entity Check Output Status Silenced Timestamp ────────────── ───────── ───────── ──────────── ────────── ─────────────────────────────── scotch.local check-http 0 true 2018-03-16 13:22:16 -0400 EDT
WARNING: By default, a silenced event will be handled unless the handler uses
not_silenced filter to discard silenced events.
You now know how to create silenced entries to plan a maintenance and hopefully avoid false positive. From this point, here are some recommended resources:
- Read the silencing reference for in-depth documentation on silenced entries.