Silencing

• Specification
• Examples
  • Silence all checks on a specific entity
  • Silence a specific check on a specific entity
  • Silence all checks on entities with a specific subscription
  • Silence a specific check on entities with a specific subscription
  • Silence a specific check on every entity
  • Deleting silences

How does silencing work?

Silences are created on an ad-hoc basis via sensuctl. When silencing entries are successfully created, they are assigned a name in the format $SUBSCRIPTION:$CHECK, where $SUBSCRIPTION is the name of a Sensu entity subscription and $CHECK is the name of a Sensu check. Silences can be used to silence checks on specific entities by taking advantage of per-entity subscriptions, for example: entity:$ENTITY_NAME. When the check name and/or subscription described in a silencing entry match an event and a handler use the not_silenced built-in filter, this handler will not be executed.

These silences are persisted in the Sensu data store. When the Sensu server processes subsequent check results, matching silences are retrieved from the store. If one or more matching entries exist, the event is updated with a list of silenced entry names. The presence of silences indicates that the event is silenced.

When creating a silencing entry, a combination of check and subscription can be specified, but only one or the other is strictly required.

For example, when a silencing entry is created specifying only a check, its name will contain an asterisk (or wildcard) in the $SUBSCRIPTION position. This indicates that any event with a matching check name will be marked as silenced, regardless of the originating entities’ subscriptions.

Conversely, a silencing entry which specifies only a subscription will have a name with an asterisk in the $CHECK position. This indicates that any event where the originating entities’ subscriptions match the subscription specified in the entry will be marked as silenced, regardless of the check name.

Silencing specification

Silenced entry names

Silences must contain either a subscription or check name, and are identified by the combination of $SUBSCRIPTION:$CHECK. If a check or subscription is not provided, it will be substituted with a wildcard (asterisk): $SUBSCRIPTION:* or *:$CHECK.

Top-level attributes

"type": "Silenced"

"api_version": "core/v2"

"metadata": {
  "name": "appserver:mysql_status",
  "namespace": "default",
  "labels": {
    "region": "us-west-1"
  }

"spec": {
  "expire": -1,
  "expire_on_resolve": false,
  "creator": "admin",
  "reason": null,
  "check": null,
  "subscription": "entity:i-424242",
  "begin": 1542671205
}

Spec attributes

"check": "haproxy_status"

"subscription": "entity:i-424242"

"begin": 1512512023

"expire": 3600

"expire_on_resolve": true

"creator": "Application Deploy Tool 5.0"

"reason": "rebooting the world"

Metadata attributes

"name": "appserver:mysql_status"

"namespace": "production"

"labels": {
  "environment": "development",
  "region": "us-west-2"
}

 "annotations": {
  "managed-by": "ops",
  "slack-channel": "#monitoring",
  "playbook": "www.example.url"
}

Examples

Silence all checks on a specific entity

Assume a Sensu entity i-424242 which we wish to silence any alerts on. We’ll do this by taking advantage of per-entity subscriptions:

type: Silenced
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: entity:i-424242:*
  namespace: default
spec:
  begin: 1542671205
  check: null
  creator: admin
  expire: -1
  expire_on_resolve: false
  reason: null
  subscription: entity:i-424242

{
  "type": "Silenced",
  "api_version": "core/v2",
  "metadata": {
    "name": "entity:i-424242:*",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "expire": -1,
    "expire_on_resolve": false,
    "creator": "admin",
    "reason": null,
    "check": null,
    "subscription": "entity:i-424242",
    "begin": 1542671205
  }
}

Silence a specific check on a specific entity

Following on the previous example, silence a check named check_ntp on entity i-424242, ensuring the entry is deleted once the underlying issue has been resolved:

check: check_ntp
expire_on_resolve: true
subscription: entity:i-424242

{
  "subscription": "entity:i-424242", 
  "check": "check_ntp", 
  "expire_on_resolve": true 
}

The optional expire_on_resolve attribute used here indicates that when the server processes a matching check from the specified entity with status OK, this silencing entry will automatically be removed.

When used in combination with other attributes (like creator and reason), this provides Sensu operators with a method of acknowledging that they have received an alert, suppressing additional notifications, and automatically clearing the silencing entry when the check status returns to normal.

Silence all checks on entities with a specific subscription

In this case, we’ll completely silence any entities subscribed to appserver. Just as in the example of silencing all checks on a specific entity, we’ll create a silencing entry specifying only the appserver subscription:

subscription: appserver

{
  "subscription": "appserver" 
}

Silence a specific check on entities with a specific subscription

Assume a check mysql_status which we wish to silence, running on Sensu entities with the subscription appserver:

check: mysql_status
subscription: appserver

{
  "subscription": "appserver", 
  "check": "mysql_status"
}

Silence a specific check on every entity

To silence the check mysql_status on every entity in our infrastructure, regardless of subscriptions, we only need to provide the check name:

check: mysql_status

{
  "check": "mysql_status"
}

Deleting silences

To delete a silencing entry, you will need to provide its name. Subscription only silencing entry names will be similar to this:

name: appserver:*

{
  "name": "appserver:*"
}

Check only silencing entry names will be similar to this:

name: '*:mysql_status'

{
  "name": "*:mysql_status"
}