Filters
How do Sensu filters work?
Sensu filters are applied when event handlers are configured to use one or more filters. Prior to executing a handler, the Sensu server will apply any filters configured for the handler to the event data. If the event is not removed by the filter(s), the handler will be executed. The filter analysis flow performs these steps:
- When the Sensu server is processing an event, it will check for the definition
of a
handler
(orhandlers
). Prior to executing each handler, the Sensu server will first apply any configuredfilters
for the handler. - If multiple
filters
are configured for a handler, they are executed sequentially. - Filter
expressions
are compared with event data. - Filters can be inclusive (only matching events are handled) or exclusive (matching events are not handled).
- As soon as a filter removes an event, no further analysis is performed and the event handler will not be executed.
NOTE: Filters specified in a handler set definition have no effect. Filters must be specified in individual handler definitions.
Inclusive and exclusive filtering
Filters can be inclusive "action": "allow"
(replaces "negate": false
in
Sensu 1) or exclusive "action": "deny"
(replaces "negate": true
in Sensu
1). Configuring a handler to use multiple inclusive filters is the equivalent
of using an AND
query operator (only handle events if they match
inclusive filter x AND y AND z
). Configuring a handler to use multiple
exclusive filters is the equivalent of using an OR
operator (only
handle events if they don’t match x OR y OR z
).
- Inclusive filtering: by setting the filter definition attribute
"action": "allow"
, only events that match the defined filter expressions are handled. - Exclusive filtering: by setting the filter definition attribute
"action": "deny"
, events are only handled if they do not match the defined filter expressions.
Filter expression comparison
Filter expressions are compared directly with their event data counterparts. For
inclusive filter definitions (like "action": "allow"
), matching expressions
will result in the filter returning a true
value; for exclusive filter
definitions (like "action": "deny"
), matching expressions will result in the
filter returning a false
value, and the event will not pass through the
filter. Filters that return a true value will continue to be processed via
additional filters (if defined), mutators (if defined), and handlers.
Filter expression evaluation
When more complex conditional logic is needed than direct filter expression comparison, Sensu filters provide support for expression evaluation using Otto. Otto is an ECMAScript 5 (JavaScript) VM, and evaluates javascript expressions that are provided in the filter. There are some caveats to using Otto; most notably, the regular expressions specified in ECMAScript 5 do not all work. See the Otto README for more details.
Filter assets
Sensu filters can have assets that are included in their execution context. When valid assets are associated with a filter, Sensu evaluates any files it finds that have a “.js” extension before executing a filter. The result of evaluating the scripts is cached for a given asset set, for the sake of performance. For an example of how to implement a filter as an asset, see the guide on reducing alert fatigue.
Built-in filters
Sensu includes built-in filters to help you customize event pipelines for metrics and alerts. To start using built-in filters, see the guides to sending Slack alerts and planning maintenances.
Built-in filter: only incidents
The incidents filter is included in every installation of the Sensu backend.
You can use the incidents filter to allow only high priority events through a Sensu pipeline.
For example, you can use the incidents filter to reduce noise when sending notifications to Slack.
When applied to a handler, the incidents filter allows only warning ("status": 1
), critical ("status": 2
), and resolution events to be processed.
To use the incidents filter, include the is_incident
filter in the handler configuration filters
array:
type: Handler
api_version: core/v2
metadata:
name: slack
namespace: default
spec:
command: sensu-slack-handler --channel '#monitoring'
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
filters:
- is_incident
handlers: []
runtime_assets: []
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "slack",
"namespace": "default"
},
"spec": {
"command": "sensu-slack-handler --channel '#monitoring'",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
],
"filters": [
"is_incident"
],
"handlers": [],
"runtime_assets": [],
"timeout": 0,
"type": "pipe"
}
}
The is_incident
filter applies the following filtering logic:
status | allow | discard | ||||
---|---|---|---|---|---|---|
0 | ![]() |
|||||
1 | ![]() |
|||||
2 | ![]() |
|||||
other | ![]() |
|||||
1 –> 0 or 2 –> 0 (resolution event) |
![]() |
Built-in filter: allow silencing
Sensu silencing lets you suppress execution of event handlers on an on-demand basis, giving you the ability to quiet incoming alerts and plan maintenances.
To allow silencing for an event handler, add the not_silenced
filter to the handler configuration filters
array:
type: Handler
api_version: core/v2
metadata:
name: slack
namespace: default
spec:
command: sensu-slack-handler --channel '#monitoring'
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
filters:
- is_incident
- not_silenced
handlers: []
runtime_assets: []
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "slack",
"namespace": "default"
},
"spec": {
"command": "sensu-slack-handler --channel '#monitoring'",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
],
"filters": [
"is_incident",
"not_silenced"
],
"handlers": [],
"runtime_assets": [],
"timeout": 0,
"type": "pipe"
}
}
When applied to a handler configuration, the not_silenced
filter silences events that include the silenced
attribute. The handler in the example above uses both the silencing and incidents filters, preventing low priority and silenced events from being sent to Slack.
Built-in filter: has metrics
The metrics filter is included in every installation of the Sensu backend. When applied to a handler, the metrics filter allows only events containing Sensu metrics to be processed. You can use the metrics filter to prevent handlers that require metrics from failing in case of an error in metric collection.
To use the metrics filter, include the has_metrics
filter in the handler configuration filters
array:
type: Handler
api_version: core/v2
metadata:
name: influx-db
namespace: default
spec:
command: sensu-influxdb-handler -d sensu
env_vars:
- INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086
- INFLUXDB_USER=sensu
- INFLUXDB_PASSWORD=password
filters:
- has_metrics
handlers: []
runtime_assets: []
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "influx-db",
"namespace": "default"
},
"spec": {
"command": "sensu-influxdb-handler -d sensu",
"env_vars": [
"INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086",
"INFLUXDB_USER=sensu",
"INFLUXDB_PASSWORD=password"
],
"filters": [
"has_metrics"
],
"handlers": [],
"runtime_assets": [],
"timeout": 0,
"type": "pipe"
}
}
When applied to a handler configuration, the has_metrics
filter allows only events that include a metrics
scope.
Building filter expressions
You can write custom filter expressions as Sensu query expressions using the event data attributes described in this section. For more information about event attributes, see the event reference.
Syntax quick reference
operator | description |
---|---|
=== / !== |
Identity operator / Nonidentity operator |
== / != |
Equality operator / Inequality operator |
&& / || |
Logical AND / Logical OR |
< / > |
Less than / Greater than |
<= / >= |
Less than or equal to / Greater than or equal to |
Event attributes available to filters
attribute | type | description |
---|---|---|
event.has_check |
boolean | Returns true if the event contains check data |
event.has_metrics |
boolean | Returns true if the event contains metrics |
event.is_incident |
boolean | Returns true for critical alerts (status 2 ), warnings (status 1 ), and resolution events (status 0 transitioning from status 1 or 2 ) |
event.is_resolution |
boolean | Returns true if the event status is OK (0 ) and the previous event was of a non-zero status |
event.is_silenced |
boolean | Returns true if the event matches an active silencing entry |
event.timestamp |
integer | Time that the event occurred in seconds since the Unix epoch |
Check attributes available to filters
attribute | type | description |
---|---|---|
event.check.annotations |
map | Custom annotations applied to the check |
event.check.command |
string | The command executed by the check |
event.check.cron |
string | Check execution schedule using cron syntax |
event.check.discard_output |
boolean | If the check is configured to discard check output from event data |
event.check.duration |
float | Command execution time in seconds |
event.check.env_vars |
array | Environment variables used with command execution |
event.check.executed |
integer | Time that the check was executed in seconds since the Unix epoch |
event.check.handlers |
array | Sensu event handlers assigned to the check |
event.check.high_flap_threshold |
integer | The check’s flap detection high threshold in percent state change |
event.check.history |
array | Check status history for the last 21 check executions |
event.check.hooks |
array | Check hook execution data |
event.check.interval |
integer | The check execution frequency in seconds |
event.check.issued |
integer | Time that the check request was issued in seconds since the Unix epoch |
event.check.labels |
map | Custom labels applied to the check |
event.check.last_ok |
integer | The last time that the check returned an OK status (0 ) in seconds since the Unix epoch |
event.check.low_flap_threshold |
integer | The check’s flap detection low threshold in percent state change |
event.check.max_output_size |
integer | Maximum size, in bytes, of stored check outputs |
event.check.name |
string | Check name |
event.check.occurrences |
integer | The number of preceding events with the same status as the current event |
event.check.occurrences_watermark |
integer | For resolution events, the number of preceding events with a non-OK status |
event.check.output |
string | The output from the execution of the check command |
event.check.output_metric_format |
string | The metric format generated by the check command: nagios_perfdata , graphite_plaintext , influxdb_line , or opentsdb_line |
event.check.output_metric_handlers |
array | Sensu metric handlers assigned to the check |
event.check.proxy_entity_name |
string | The entity name, used to create a proxy entity for an external resource |
event.check.proxy_requests |
map | Proxy request configuration |
event.check.publish |
boolean | If the check is scheduled automatically |
event.check.round_robin |
boolean | If the check is configured to be executed in a round-robin style |
event.check.runtime_assets |
array | Sensu assets used by the check |
event.check.state |
string | The state of the check: passing (status 0 ), failing (status other than 0 ), or flapping |
event.check.status |
integer | Exit status code produced by the check: 0 (OK), 1 (warning), 2 (critical), or other status (unknown or custom status) |
event.check.stdin |
boolean | If the Sensu agent writes JSON-serialized entity and check data to the command process’ STDIN |
event.check.subscriptions |
array | Subscriptions that the check belongs to |
event.check.timeout |
integer | The check execution duration timeout in seconds |
event.check.total_state_change |
integer | The total state change percentage for the check’s history |
event.check.ttl |
integer | The time to live (TTL) in seconds until the event is considered stale |
event.metrics.handlers |
array | Sensu metric handlers assigned to the check |
event.metrics.points |
array | Metric data points including a name, timestamp, value, and tags |
Entity attributes available to filters
attribute | type | description |
---|---|---|
event.entity.annotations |
map | Custom annotations assigned to the entity |
event.entity.deregister |
boolean | If the agent entity should be removed when it stops sending keepalive messages |
event.entity.deregistration |
map | A map containing a handler name, for use when an entity is deregistered |
event.entity.entity_class |
string | The entity type: usually agent or proxy |
event.entity.labels |
map | Custom labels assigned to the entity |
event.entity.last_seen |
integer | Timestamp the entity was last seen, in seconds since the Unix epoch |
event.entity.name |
string | Entity name |
event.entity.redact |
array | List of items to redact from log messages |
event.entity.subscriptions |
array | List of subscriptions assigned to the entity |
event.entity.system |
map | Information about the entity’s system |
event.entity.system.arch |
string | The entity’s system architecture |
event.entity.system.hostname |
string | The entity’s hostname |
event.entity.system.network |
map | The entity’s network interface list |
event.entity.system.os |
string | The entity’s operating system |
event.entity.system.platform |
string | The entity’s operating system distribution |
event.entity.system.platform_family |
string | The entity’s operating system family |
event.entity.system.platform_version |
string | The entity’s operating system version |
event.entity.user |
string | Sensu RBAC username used by the agent entity |
Filter specification
Top-level attributes
type | |
---|---|
description | Top-level attribute specifying the sensuctl create resource type. Filters should always be of type EventFilter . |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | String |
example |
|
api_version | |
---|---|
description | Top-level attribute specifying the Sensu API group and version. For filters in Sensu backend version 5.9, this attribute should always be core/v2 . |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | String |
example |
|
metadata | |
---|---|
description | Top-level collection of metadata about the filter, including the name and namespace as well as custom labels and annotations . The metadata map is always at the top level of the filter definition. This means that in wrapped-json and yaml formats, the metadata scope occurs outside the spec scope. See the metadata attributes reference for details. |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | Map of key-value pairs |
example |
|
spec | |
---|---|
description | Top-level map that includes the filter spec attributes. |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | Map of key-value pairs |
example |
|
Spec attributes
action | |
---|---|
description | Action to take with the event if the filter expressions match. NOTE: see Inclusive and exclusive filtering for more information. |
required | true |
type | String |
allowed values | allow , deny |
example |
|
expressions | |
---|---|
description | Filter expressions to be compared with event data. Note that event metadata can be referenced without including the metadata scope, for example: event.entity.namespace . |
required | true |
type | Array |
example |
|
runtime_assets | |
---|---|
description | Assets to be applied to the filter’s execution context. JavaScript files in the lib directory of the asset will be evaluated. |
required | false |
type | Array of String |
default | [] |
example |
|
Metadata attributes
name | |
---|---|
description | A unique string used to identify the filter. Filter names cannot contain special characters or spaces (validated with Go regex \A[\w\.\-]+\z ). Each filter must have a unique name within its namespace. |
required | true |
type | String |
example |
|
namespace | |
---|---|
description | The Sensu RBAC namespace that this filter belongs to. |
required | false |
type | String |
default | default |
example |
|
labels | |
---|---|
description | Custom attributes you can use to create meaningful collections that can be selected with API filtering and sensuctl filtering. Overusing labels can impact Sensu’s internal performance, so we recommend moving complex, non-identifying metadata to annotations. |
required | false |
type | Map of key-value pairs. Keys can contain only letters, numbers, and underscores, but must start with a letter. Values can be any valid UTF-8 string. |
default | null |
example |
|
annotations | |
---|---|
description | Non-identifying metadata that’s meaningful to people or external tools interacting with Sensu. In contrast to labels, annotations cannot be used in API filtering or sensuctl filtering and do not impact Sensu’s internal performance. |
required | false |
type | Map of key-value pairs. Keys and values can be any valid UTF-8 string. |
default | null |
example |
|
Filter Examples
Minimum required filter attributes
type: EventFilter
api_version: core/v2
metadata:
name: filter_minimum
namespace: default
spec:
action: allow
expressions:
- event.check.occurrences == 1
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "filter_minimum",
"namespace": "default"
},
"spec": {
"action": "allow",
"expressions": [
"event.check.occurrences == 1"
]
}
}
Handling production events
The following filter allows only events with a custom entity label "environment": "production"
to be handled.
type: EventFilter
api_version: core/v2
metadata:
name: production_filter
namespace: default
spec:
action: allow
expressions:
- event.entity.labels.environment == 'production'
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "production_filter",
"namespace": "default"
},
"spec": {
"action": "allow",
"expressions": [
"event.entity.labels.environment == 'production'"
]
}
}
Handling non-production events
The following filter discards events with a custom entity label "environment": "production"
, allowing only events without an environment
label or events with environment
set to something other than production
to be handled.
Note that action
is deny
, making this an exclusive filter; if evaluation
returns false, the event is handled.
type: EventFilter
api_version: core/v2
metadata:
name: not_production
namespace: default
spec:
action: deny
expressions:
- event.entity.labels.environment == 'production'
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "not_production",
"namespace": "default"
},
"spec": {
"action": "deny",
"expressions": [
"event.entity.labels.environment == 'production'"
]
}
}
Handling state change only
Some teams migrating to Sensu have asked about reproducing the behavior of their
old monitoring system which alerts only on state change. This
state_change_only
inclusive filter provides such.
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: state_change_only
namespace: default
spec:
action: allow
expressions:
- event.check.occurrences == 1
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "state_change_only",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"event.check.occurrences == 1"
],
"runtime_assets": []
}
}
Handling repeated events
The following example filter definition, entitled filter_interval_60_hourly
,
will match event data with a check interval
of 60
seconds, and an
occurrences
value of 1
(the first occurrence) -OR- any occurrences
value that is evenly divisible by 60 via a modulo
operator calculation
(calculating the remainder after dividing occurrences
by 60).
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: filter_interval_60_hourly
namespace: default
spec:
action: allow
expressions:
- event.check.interval == 60
- event.check.occurrences == 1 || event.check.occurrences % 60 == 0
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "filter_interval_60_hourly",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"event.check.interval == 60",
"event.check.occurrences == 1 || event.check.occurrences % 60 == 0"
],
"runtime_assets": []
}
}
The next example will apply the same logic as the previous example, but for
checks with a 30 second interval
.
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: filter_interval_30_hourly
namespace: default
spec:
action: allow
expressions:
- event.check.interval == 30
- event.check.occurrences == 1 || event.check.occurrences % 120 == 0
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "filter_interval_30_hourly",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"event.check.interval == 30",
"event.check.occurrences == 1 || event.check.occurrences % 120 == 0"
],
"runtime_assets": []
}
}
Handling events during office hours only
This filter evaluates the event timestamp to determine if the event occurred
between 9 AM and 5 PM UTC on a weekday. Remember that action
is equal to
allow
, so this is an inclusive filter. If evaluation returns false, the event
will not be handled.
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: nine_to_fiver
namespace: default
spec:
action: allow
expressions:
- weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5
- hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "nine_to_fiver",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5",
"hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17"
],
"runtime_assets": []
}
}
Using JavaScript libraries with Sensu filters
You can include JavaScript libraries in their filter execution context with assets. For instance, assuming you’ve packaged underscore.js into a Sensu asset, you could then use functions from the underscore library for filter expressions.
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: deny_if_failure_in_history
namespace: default
spec:
action: deny
expressions:
- _.reduce(event.check.history, function(memo, h) { return (memo || h.status !=
0); })
runtime_assets:
- underscore
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "deny_if_failure_in_history",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "deny",
"expressions": [
"_.reduce(event.check.history, function(memo, h) { return (memo || h.status != 0); })"
],
"runtime_assets": ["underscore"]
}
}