Checks

• Check commands
• Check result specification
• Check scheduling
  • Subscriptions
  • Scheduling
• Proxy checks
• Check token substitution
• Check hooks
• Check specification
  • Top-level attributes
  • Spec attributes
  • Metadata attributes
  • Proxy requests attributes
  • Check output truncation attributes
• Examples

Checks work with Sensu agents to produce monitoring events automatically. You can use checks to monitor server resources, services, and application health as well as collect and analyze metrics. Read the guide to monitoring server resources to get started. You can discover, download, and share Sensu check assets using Bonsai, the Sensu asset index.

Check commands

Each Sensu check definition specifies a command and the schedule at which it should be executed. Check commands are executable commands which are executed by the Sensu agent.

A command may include command line arguments for controlling the behavior of the command executable. Most Sensu check plugins provide support for command line arguments for reusability.

Sensu advises against requiring root privileges to execute check commands or scripts. The Sensu user is not permitted to kill timed out processes invoked by the root user, which could result in zombie processes.

How and where are check commands executed?

All check commands are executed by Sensu agents as the sensu user. Commands must be executable files that are discoverable on the Sensu agent system (for example: installed in a system $PATH directory).

Check result specification

Although Sensu agents attempt to execute any command defined for a check, successful processing of check results requires adherence to a simple specification.

• Result data is output to STDOUT or STDERR
  • For service checks, this output is typically a human-readable message.
  • For metric checks, this output contains the measurements gathered by the check.
• Exit status code indicates state
  • 0 indicates “OK”
  • 1 indicates “WARNING”
  • 2 indicates “CRITICAL”
  • Exit status codes other than 0, 1, or 2 indicate an “UNKNOWN” or custom status

PRO TIP: Those familiar with the Nagios monitoring system may recognize this specification, as it is the same one used by Nagios plugins. As a result, Nagios plugins can be used with Sensu without any modification.

At every execution of a check command – regardless of success or failure – the Sensu agent publishes the check’s result for eventual handling by the event processor (the Sensu backend).

Check scheduling

Checks are scheduled by the Sensu backend, which publishes check execution requests to entities via a publish-subscribe model.

Subscriptions

Checks have a defined set of subscriptions, transport topics to which the Sensu backend publishes check requests. Sensu entities become subscribers to these topics (called subscriptions) via their individual subscriptions attribute. In practice, subscriptions typically correspond to a specific role or responsibility (for example: a webserver or database).

Subscriptions are powerful primitives in the monitoring context because they allow you to effectively monitor for specific behaviors or characteristics corresponding to the function being provided by a particular system. For example, disk capacity thresholds might be more important (or at least different) on a database server as opposed to a webserver; conversely, CPU or memory usage thresholds might be more important on a caching system than on a file server. Subscriptions also allow you to configure check requests for an entire group or subgroup of systems rather than requiring a traditional one-to-one mapping.

To configure subscriptions for a check, use the subscriptions attribute to specify an array of one or more subscription names. Sensu schedules checks once per interval for each agent with a matching subscription. For example, if we have three agents configured with the system subscription, a check configured with the system subscription results in three monitoring events per interval: one check execution per agent per interval. In order for Sensu to execute a check, the check definition must include a subscription that matches the subscription of at least one Sensu agent.

Round-robin checks

By default, Sensu schedules checks once per interval for each agent with a matching subscription: one check execution per agent per interval. Sensu also supports deduplicated check execution when configured with the round_robin check attribute. For checks with round_robin set to true, Sensu executes the check once per interval, cycling through the available agents alphabetically according to agent name.

For example, for three agents configured with the system subscription (agents A, B, and C), a check configured with the system subscription and round_robin set to true results in one monitoring event per interval, with the agent creating the event following the pattern A -> B -> C -> A -> B -> C for the first six intervals.

In the diagram above, the standard check is executed by agents A, B, and C every 60 seconds, while the round-robin check cycles through the available agents, resulting in each agent executing the check every 180 seconds.

PRO TIP: You can use round robin to distribute check execution workload across multiple agents when using proxy checks.

Scheduling

You can schedule checks using the interval, cron, and publish attributes. Sensu requires that checks include either an interval attribute (interval scheduling) or a cron attribute (cron scheduling).

Interval scheduling

You can schedule a check to be executed at regular intervals using the interval and publish check attributes. For example, to schedule a check to execute every 60 seconds, set the interval attribute to 60 and the publish attribute to true.

NOTE: When creating an interval check, Sensu calculates an initial offset to splay the check’s first scheduled request. This helps to balance the load of both the backend and the agent, and may result in a delay before initial check execution.

Example interval check

type: CheckConfig
api_version: core/v2
metadata:
  name: interval_check
  namespace: default
spec:
  command: check-cpu.sh -w 75 -c 90
  handlers:
  - slack
  interval: 60
  publish: true
  subscriptions:
  - system

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "interval_check",
    "namespace": "default"
  },
  "spec": {
    "command": "check-cpu.sh -w 75 -c 90",
    "subscriptions": ["system"],
    "handlers": ["slack"],
    "interval": 60,
    "publish": true
  }
}

Cron scheduling

You can also schedule checks using cron syntax. For example, to schedule a check to execute once a minute at the start of the minute, set the cron attribute to * * * * * and the publish attribute to true.

Example cron check

type: CheckConfig
api_version: core/v2
metadata:
  name: cron_check
  namespace: default
spec:
  command: check-cpu.sh -w 75 -c 90
  cron: '* * * * *'
  handlers:
  - slack
  publish: true
  subscriptions:
  - system

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "cron_check",
    "namespace": "default"
  },
  "spec": {
    "command": "check-cpu.sh -w 75 -c 90",
    "subscriptions": ["system"],
    "handlers": ["slack"],
    "cron": "* * * * *",
    "publish": true
  }
}

Ad-hoc scheduling

In addition to automatic execution, you can create checks to be scheduled manually using the checks API. To create a check with ad-hoc scheduling, set the publish attribute to false in addition to an interval or cron schedule.

Example ad-hoc check

type: CheckConfig
api_version: core/v2
metadata:
  name: ad_hoc_check
  namespace: default
spec:
  command: check-cpu.sh -w 75 -c 90
  handlers:
  - slack
  interval: 60
  publish: false
  subscriptions:
  - system

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "ad_hoc_check",
    "namespace": "default"
  },
  "spec": {
    "command": "check-cpu.sh -w 75 -c 90",
    "subscriptions": ["system"],
    "handlers": ["slack"],
    "interval": 60,
    "publish": false
  }
}

Proxy checks

Sensu supports running proxy checks where the results are considered to be for an entity that isn’t actually the one executing the check, regardless of whether that entity is a Sensu agent entity or a proxy entity. Proxy entities allow Sensu to monitor external resources on systems or devices where a Sensu agent cannot be installed, like a network switch or a website. You can create a proxy check using the proxy_entity_name attribute or the proxy_requests attributes.

Using a proxy check to monitor a proxy entity

When executing checks that include a proxy_entity_name, Sensu agents report the resulting events under the specified proxy entity instead of the agent entity. If the proxy entity doesn’t exist, Sensu creates the proxy entity when the event is received by the backend. To avoid duplicate events, we recommend using the round_robin attribute with proxy checks.

Example proxy check using a proxy_entity_name

The following proxy check runs every 60 seconds, cycling through the agents with the proxy subscription alphabetically according to the agent name, for the proxy entity sensu-site.

type: CheckConfig
api_version: core/v2
metadata:
  name: proxy_check
  namespace: default
spec:
  command: http_check.sh https://sensu.io
  handlers:
  - slack
  interval: 60
  proxy_entity_name: sensu-site
  publish: true
  round_robin: true
  subscriptions:
  - proxy

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "proxy_check",
    "namespace": "default"
  },
  "spec": {
    "command": "http_check.sh https://sensu.io",
    "subscriptions": ["proxy"],
    "handlers": ["slack"],
    "interval": 60,
    "publish": true,
    "round_robin": true,
    "proxy_entity_name": "sensu-site"
  }
}

Using a proxy check to monitor multiple proxy entities

The proxy_requests check attributes allow Sensu to run a check for each entity that matches the definitions specified in the entity_attributes, resulting in monitoring events that represents each matching proxy entity. The entity attributes must match exactly as stated; no variables or directives have any special meaning, but you can still use Sensu query expressions to perform more complicated filtering on the available value, such as finding entities with particular subscriptions.

The proxy_requests attributes are a great way to monitor multiple entities using a single check definition when combined with token substitution. Since checks including proxy_requests attributes need to be executed for each matching entity, we recommend using the round_robin attribute to distribute the check execution workload evenly across your Sensu agents.

Example proxy check using proxy_requests

The following proxy check runs every 60 seconds, cycling through the agents with the proxy subscription alphabetically according to the agent name, for all existing proxy entities with the custom label proxy_type set to website.

This check uses token substitution to import the value of the custom entity label url to complete the check command. See the entity reference for information about using custom labels.

type: CheckConfig
api_version: core/v2
metadata:
  name: proxy_check_proxy_requests
  namespace: default
spec:
  command: http_check.sh {{ .labels.url }}
  handlers:
  - slack
  interval: 60
  proxy_requests:
    entity_attributes:
    - entity.labels.proxy_type == 'website'
  publish: true
  round_robin: true
  subscriptions:
  - proxy

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "proxy_check_proxy_requests",
    "namespace": "default"
  },
  "spec": {
    "command": "http_check.sh {{ .labels.url }}",
    "subscriptions": ["proxy"],
    "handlers": ["slack"],
    "interval": 60,
    "publish": true,
    "proxy_requests": {
      "entity_attributes": [
        "entity.labels.proxy_type == 'website'"
      ]
    },
    "round_robin": true
  }
}

Fine-tuning proxy check scheduling with splay

Sensu supports distributing proxy check executions across an interval using the splay and splay_coverage attributes. For example, if we assume that the proxy_check_proxy_requests check in the example above matches three proxy entities, we’d expect to see a burst of three events every 60 seconds. If we add the splay attribute (set to true) and the splay_coverage attribute (set to 90) to the proxy_requests scope, Sensu distributes the three check executions over 90% of the 60-second interval, resulting in three events splayed evenly across a 54-second period.

Check token substitution

Sensu check definitions may include attributes that you may wish to override on an entity-by-entity basis. For example, check commands – which may include command line arguments for controlling the behavior of the check command – may benefit from entity-specific thresholds, etc. Sensu check tokens are check definition placeholders that will be replaced by the Sensu agent with the corresponding entity definition attributes values (including custom attributes).

Learn how to use check tokens with the Sensu tokens reference documentation.

NOTE: Check tokens are processed before check execution, therefore token substitutions will not apply to check data delivered via the local agent socket input.

Check hooks

Check hooks are commands run by the Sensu agent in response to the result of check command execution. The Sensu agent will execute the appropriate configured hook command, depending on the check execution status (ex: 0, 1, 2).

Learn how to use check hooks with the Sensu hooks reference documentation.

Check specification

Top-level attributes

"type": "CheckConfig"

"api_version": "core/v2"

"metadata": {
  "name": "collect-metrics",
  "namespace": "default",
  "labels": {
    "region": "us-west-1"
  },
  "annotations": {
    "slack-channel" : "#monitoring"
  }
}

"spec": {
  "command": "/etc/sensu/plugins/check-chef-client.go",
  "interval": 10,
  "publish": true,
  "subscriptions": [
    "production"
  ]
}

Spec attributes

"command": "/etc/sensu/plugins/check-chef-client.go"

"subscriptions": ["production"]

"handlers": ["pagerduty", "email"]

"interval": 60

"cron": "0 0 * * *"

"publish": false

"timeout": 30

"ttl": 100

"stdin": true

"low_flap_threshold": 20

"high_flap_threshold": 60

"runtime_assets": ["ruby-2.5.0"]

"check_hooks": [
  {
    "0": [
      "passing-hook","always-run-this-hook"
    ]
  },
  {
    "critical": [
      "failing-hook","collect-diagnostics","always-run-this-hook"
    ]
  }
]

"proxy_entity_name": "switch-dc-01"

"proxy_requests": {
  "entity_attributes": [
    "entity.entity_class == 'proxy'",
    "entity.labels.proxy_type == 'website'"
  ],
  "splay": true,
  "splay_coverage": 90
}

"silenced": ["*:routers"]

"env_vars": ["RUBY_VERSION=2.5.0", "CHECK_HOST=my.host.internal"]

"output_metric_format": "graphite_plaintext"

"output_metric_handlers": ["influx-db"]

"round_robin": true

"subdue": null

Metadata attributes

"name": "check-cpu"

"namespace": "production"

"labels": {
  "environment": "development",
  "region": "us-west-2"
}

 "annotations": {
  "managed-by": "ops",
  "playbook": "www.example.url"
}

Proxy requests attributes

"entity_attributes": [
  "entity.entity_class == 'proxy'",
  "entity.labels.proxy_type == 'website'"
]

"splay": true

"splay_coverage": 90

Check output truncation attributes

"max_output_size": 1024

"discard_output": true

Examples

Minimum recommended check attributes

NOTE: The attribute interval is not required if a valid cron schedule is defined.

type: CheckConfig
api_version: core/v2
metadata:
  name: check_minimum
  namespace: default
spec:
  command: collect.sh
  handlers:
  - slack
  interval: 10
  publish: true
  subscriptions:
  - system

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default",
    "name": "check_minimum"
  },
  "spec": {
    "command": "collect.sh",
    "subscriptions": [
      "system"
    ],
    "handlers": [
      "slack"
    ],
    "interval": 10,
    "publish": true
  }
}

Metric check

type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    slack-channel: '#monitoring'
  labels:
    region: us-west-1
  name: collect-metrics
  namespace: default
spec:
  check_hooks: null
  command: collect.sh
  discard_output: true
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: graphite_plaintext
  output_metric_handlers:
  - influx-db
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets: null
  stdin: false
  subscriptions:
  - system
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "collect-metrics",
    "namespace": "default",
    "labels": {
      "region": "us-west-1"
    },
    "annotations": {
      "slack-channel" : "#monitoring"
    }
  },
  "spec": {
    "command": "collect.sh",
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 10,
    "low_flap_threshold": 0,
    "publish": true,
    "runtime_assets": null,
    "subscriptions": [
      "system"
    ],
    "proxy_entity_name": "",
    "check_hooks": null,
    "stdin": false,
    "ttl": 0,
    "timeout": 0,
    "round_robin": false,
    "output_metric_format": "graphite_plaintext",
    "output_metric_handlers": [
      "influx-db"
    ],
    "env_vars": null,
    "discard_output": true
  }
}