Filters

• Built-in filters
• Building filter expressions
• Specification
• Examples
  • Handling production events
  • Handling non-production events
  • Handling state change only
  • Handling repeated events
  • Handling events during office hours only

How do Sensu filters work?

Sensu filters are applied when event handlers are configured to use one or more filters. Prior to executing a handler, the Sensu server will apply any filters configured for the handler to the event data. If the event is not removed by the filter(s), the handler will be executed. The filter analysis flow performs these steps:

• When the Sensu server is processing an event, it will check for the definition of a handler (or handlers). Prior to executing each handler, the Sensu server will first apply any configured filters for the handler.
• If multiple filters are configured for a handler, they are executed sequentially.
• Filter expressions are compared with event data.
• Filters can be inclusive (only matching events are handled) or exclusive (matching events are not handled).
• As soon as a filter removes an event, no further analysis is performed and the event handler will not be executed.

NOTE: Filters specified in a handler set definition have no effect. Filters must be specified in individual handler definitions.

Inclusive and exclusive filtering

Filters can be inclusive "action": "allow" (replaces "negate": false in Sensu 1) or exclusive "action": "deny" (replaces "negate": true in Sensu 1). Configuring a handler to use multiple inclusive filters is the equivalent of using an AND query operator (only handle events if they match inclusive filter x AND y AND z). Configuring a handler to use multiple exclusive filters is the equivalent of using an OR operator (only handle events if they don’t match x OR y OR z).

• Inclusive filtering: by setting the filter definition attribute "action": "allow", only events that match the defined filter expressions are handled.
• Exclusive filtering: by setting the filter definition attribute "action": "deny", events are only handled if they do not match the defined filter expressions.

Filter expression comparison

Filter expressions are compared directly with their event data counterparts. For inclusive filter definitions (like "action": "allow"), matching expressions will result in the filter returning a true value; for exclusive filter definitions (like "action": "deny"), matching expressions will result in the filter returning a false value, and the event will not pass through the filter. Filters that return a true value will continue to be processed via additional filters (if defined), mutators (if defined), and handlers.

Filter expression evaluation

When more complex conditional logic is needed than direct filter expression comparison, Sensu filters provide support for expression evaluation using Otto. Otto is an ECMAScript 5 (JavaScript) VM, and evaluates javascript expressions that are provided in the filter. There are some caveats to using Otto; most notably, the regular expressions specified in ECMAScript 5 do not all work. See the Otto README for more details.

Filter assets

Sensu filters can have assets that are included in their execution context. When valid assets are associated with a filter, Sensu evaluates any files it finds that have a “.js” extension before executing a filter. The result of evaluating the scripts is cached for a given asset set, for the sake of performance. For an example of how to implement a filter as an asset, see the guide on reducing alert fatigue.

Built-in filters

Sensu includes built-in filters to help you customize event pipelines for metrics and alerts. To start using built-in filters, see the guides to sending Slack alerts and planning maintenances.

Built-in filter: only incidents

The incidents filter is included in every installation of the Sensu backend. You can use the incidents filter to allow only high priority events through a Sensu pipeline. For example, you can use the incidents filter to reduce noise when sending notifications to Slack. When applied to a handler, the incidents filter allows only warning ("status": 1), critical ("status": 2), and resolution events to be processed.

To use the incidents filter, include the is_incident filter in the handler configuration filters array:

type: Handler
api_version: core/v2
metadata:
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel '#monitoring'
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
  filters:
  - is_incident
  handlers: []
  runtime_assets: []
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-slack-handler --channel '#monitoring'",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
    ],
    "filters": [
      "is_incident"
    ],
    "handlers": [],
    "runtime_assets": [],
    "timeout": 0,
    "type": "pipe"
  }
}

The is_incident filter applies the following filtering logic:

Built-in filter: allow silencing

Sensu silencing lets you suppress execution of event handlers on an on-demand basis, giving you the ability to quiet incoming alerts and plan maintenances.

To allow silencing for an event handler, add the not_silenced filter to the handler configuration filters array:

type: Handler
api_version: core/v2
metadata:
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel '#monitoring'
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
  filters:
  - is_incident
  - not_silenced
  handlers: []
  runtime_assets: []
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-slack-handler --channel '#monitoring'",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
    ],
    "filters": [
      "is_incident",
      "not_silenced"
    ],
    "handlers": [],
    "runtime_assets": [],
    "timeout": 0,
    "type": "pipe"
  }
}

When applied to a handler configuration, the not_silenced filter silences events that include the silenced attribute. The handler in the example above uses both the silencing and incidents filters, preventing low priority and silenced events from being sent to Slack.

Built-in filter: has metrics

The metrics filter is included in every installation of the Sensu backend. When applied to a handler, the metrics filter allows only events containing Sensu metrics to be processed. You can use the metrics filter to prevent handlers that require metrics from failing in case of an error in metric collection.

To use the metrics filter, include the has_metrics filter in the handler configuration filters array:

type: Handler
api_version: core/v2
metadata:
  name: influx-db
  namespace: default
spec:
  command: sensu-influxdb-handler -d sensu
  env_vars:
  - INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086
  - INFLUXDB_USER=sensu
  - INFLUXDB_PASSWORD=password
  filters:
  - has_metrics
  handlers: []
  runtime_assets: []
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "influx-db",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-influxdb-handler -d sensu",
    "env_vars": [
      "INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086",
      "INFLUXDB_USER=sensu",
      "INFLUXDB_PASSWORD=password"
    ],
    "filters": [
      "has_metrics"
    ],
    "handlers": [],
    "runtime_assets": [],
    "timeout": 0,
    "type": "pipe"
  }
}

When applied to a handler configuration, the has_metrics filter allows only events that include a metrics scope.

Building filter expressions

You can write custom filter expressions as Sensu query expressions using the event data attributes described in this section. For more information about event attributes, see the event reference.

Syntax quick reference

Event attributes available to filters

Check attributes available to filters

Entity attributes available to filters

Filter specification

Top-level attributes

"type": "EventFilter"

"api_version": "core/v2"

"metadata": {
  "name": "filter-weekdays-only",
  "namespace": "default",
  "labels": {
    "region": "us-west-1"
  },
  "annotations": {
    "slack-channel" : "#monitoring"
  }
}

"spec": {
  "action": "allow",
  "expressions": [
    "event.entity.namespace == 'production'"
  ],
  "runtime_assets": []
}

Spec attributes

"action": "allow"

"expressions": [
  "event.check.team == 'ops'"
]

"runtime_assets": ["underscore"]

Metadata attributes

"name": "filter-weekdays-only"

"namespace": "production"

"labels": {
  "environment": "development",
  "region": "us-west-2"
}

 "annotations": {
  "managed-by": "ops",
  "playbook": "www.example.url"
}

Filter Examples

Minimum required filter attributes

type: EventFilter
api_version: core/v2
metadata:
  name: filter_minimum
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "filter_minimum",
    "namespace": "default"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.occurrences == 1"
    ]
  }
}

Handling production events

The following filter allows only events with a custom entity label "environment": "production" to be handled.

type: EventFilter
api_version: core/v2
metadata:
  name: production_filter
  namespace: default
spec:
  action: allow
  expressions:
  - event.entity.labels.environment == 'production'

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "production_filter",
    "namespace": "default"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.entity.labels.environment == 'production'"
    ]
  }
}

Handling non-production events

The following filter discards events with a custom entity label "environment": "production", allowing only events without an environment label or events with environment set to something other than production to be handled. Note that action is deny, making this an exclusive filter; if evaluation returns false, the event is handled.

type: EventFilter
api_version: core/v2
metadata:
  name: not_production
  namespace: default
spec:
  action: deny
  expressions:
  - event.entity.labels.environment == 'production'

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "not_production",
    "namespace": "default"
  },
  "spec": {
    "action": "deny",
    "expressions": [
      "event.entity.labels.environment == 'production'"
    ]
  }
}

Handling state change only

Some teams migrating to Sensu have asked about reproducing the behavior of their old monitoring system which alerts only on state change. This state_change_only inclusive filter provides such.

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: state_change_only
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "state_change_only",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.occurrences == 1"
    ],
    "runtime_assets": []
  }
}

Handling repeated events

The following example filter definition, entitled filter_interval_60_hourly, will match event data with a check interval of 60 seconds, and an occurrences value of 1 (the first occurrence) -OR- any occurrences value that is evenly divisible by 60 via a modulo operator calculation (calculating the remainder after dividing occurrences by 60).

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: filter_interval_60_hourly
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.interval == 60
  - event.check.occurrences == 1 || event.check.occurrences % 60 == 0
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "filter_interval_60_hourly",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.interval == 60",
      "event.check.occurrences == 1 || event.check.occurrences % 60 == 0"
    ],
    "runtime_assets": []
  }
}

The next example will apply the same logic as the previous example, but for checks with a 30 second interval.

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: filter_interval_30_hourly
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.interval == 30
  - event.check.occurrences == 1 || event.check.occurrences % 120 == 0
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "filter_interval_30_hourly",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.interval == 30",
      "event.check.occurrences == 1 || event.check.occurrences % 120 == 0"
    ],
    "runtime_assets": []
  }
}

Handling events during office hours only

This filter evaluates the event timestamp to determine if the event occurred between 9 AM and 5 PM UTC on a weekday. Remember that action is equal to allow, so this is an inclusive filter. If evaluation returns false, the event will not be handled.

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: nine_to_fiver
  namespace: default
spec:
  action: allow
  expressions:
  - weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5
  - hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "nine_to_fiver",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5",
      "hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17"
    ],
    "runtime_assets": []
  }
}

Using JavaScript libraries with Sensu filters

You can include JavaScript libraries in their filter execution context with assets. For instance, assuming you’ve packaged underscore.js into a Sensu asset, you could then use functions from the underscore library for filter expressions.

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: deny_if_failure_in_history
  namespace: default
spec:
  action: deny
  expressions:
  - _.reduce(event.check.history, function(memo, h) { return (memo || h.status !=
    0); })
  runtime_assets:
  - underscore

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "deny_if_failure_in_history",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "deny",
    "expressions": [
      "_.reduce(event.check.history, function(memo, h) { return (memo || h.status != 0); })"
    ],
    "runtime_assets": ["underscore"]
  }
}