Handlers

Reference documentation

What is a Sensu event handler?

Sensu event handlers are actions executed by the Sensu server on events, such as sending an email alert, creating or resolving an incident (e.g. in PagerDuty, ServiceNow, etc), or storing metrics in a time-series database (e.g. Graphite).

Handler types

There are several types of handlers. The most common handler type is the pipe handler, which works very similarly to how checks work, enabling Sensu to interact with almost any computer program via standard streams.

  • Pipe handlers. Pipe handlers pipe event data into arbitrary commands via STDIN.
  • TCP/UDP handlers. TCP and UDP handlers send event data to a remote socket (e.g. external API).
  • Transport handlers. Transport handlers publish event data to the Sensu transport.
  • Handler sets. Handler sets (also called “set handlers”) are used to group event handlers, making it easy to manage groups of actions that should be executed for certain types of events.

The default handler

Sensu expects all events to have a corresponding handler. Event handler(s) may be configured in check definitions, however if no handler or handlers have been configured, Sensu will attempt to handle the event using a handler named default. The default handler is only a reference (i.e. Sensu does not provide a built-in default handler), so if no handler definition exists for a handler named default, Sensu will log an error indicating that the event was not handled because a default handler definition does not exist. To use one or more existing handlers as the default, you can create a Set handler called default and include the existing handler(s) in the set.

Keepalive

Sensu Client keepalives are the heartbeat mechanism used by Sensu to ensure that all registered Sensu clients are still operational and able to reach the Sensu transport. By default keepalive events are handled by the default handler.

The keepalive scope can be configured to use specific handlers on the client, as well as overriding the default threshold values.

For example:

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "keepalive": {
      "handlers": [
        "pagerduty",
        "email"
      ],
      "thresholds": {
        "warning": 40,
        "critical": 60
      }
    }
  }
}

Pipe handlers

Pipe handlers are external commands that can consume event data via STDIN.

Example pipe handler definition

{
  "handlers": {
    "example_pipe_handler": {
      "type": "pipe",
      "command": "do_something_awesome.rb -o options"
    }
  }
}

Pipe handler commands

What is a pipe handler command?

Pipe handler definitions include a command attribute which are literally executable commands which will be executed on a Sensu server as the sensu user.

Pipe handler command arguments

Pipe handler command attributes may include command line arguments for controlling the behavior of the command executable. Most Sensu handler plugins provide support for command line arguments for reusability.

How and where are pipe handler commands executed?

As mentioned above, all pipe handlers are executed by a Sensu server as the sensu user. Commands must be executable files that are discoverable on the Sensu server system (i.e. installed in a system $PATH directory).

NOTE: By default, the Sensu installer packages will modify the system $PATH for the Sensu processes to include /etc/sensu/plugins. As a result, executable scripts (e.g. plugins) located in /etc/sensu/plugins will be valid commands. This allows command attributes to use “relative paths” for Sensu plugin commands;

e.g.: "command": "handler-irc.rb"

TCP/UDP handlers

TCP and UDP handlers enable Sensu to forward event data to arbitrary [TCP or UDP sockets][t] for external services to consume (e.g. third-party APIs).

Example TCP handler definition

The following example TCP handler definition will forward event data to a TCP socket (i.e. 10.0.1.99:4444) and will timeout if an acknowledgement (ACK) is not received within 30 seconds.

{
  "handlers": {
    "example_tcp_handler": {
      "type": "tcp",
      "timeout": 30,
      "socket": {
        "host": "10.0.1.99",
        "port": 4444
      }
    }
  }
}

The following example UDP handler definition will forward event data to a UDP socket (i.e. 10.0.1.99:444).

{
  "handlers": {
    "example_udp_handler": {
      "type": "udp",
      "socket": {
        "host": "10.0.1.99",
        "port": 4444
      }
    }
  }
}

Transport handlers

Transport handlers enable Sensu to publish event data to named queues on the Sensu transport for external services to consume.

Example transport handler definition

The following example transport handler definition will publish event data to the Sensu transport on a pipe (e.g. a “queue” or “channel”, etc) named example_handler_queue. One or more instances of an external process or third-party application would need to subscribe to the named pipe to process the events.

{
  "handlers": {
    "example_transport_handler": {
      "type": "transport",
      "pipe": {
        "type": "direct",
        "name": "example_handler_queue"
      }
    }
  }
}

Handler sets

Handler set definitions allow groups of handlers (i.e. individual collections of actions to take on event data) to be referenced via a single named handler set.

NOTE: Attributes defined on handler sets do not apply to the handlers they include. For example, filter, filters, and mutator attributes defined in a handler set will have no effect.

Example handler set definition

The following example handler set definition will execute three handlers (i.e. email, slack, and pagerduty) for every event.

{
  "handlers": {
    "notify_all_the_things": {
      "type": "set",
      "handlers": [
        "email",
        "slack",
        "pagerduty"
      ]
    }
  }
}

Handler configuration

Example handler definition

The following is an example Sensu handler definition, a JSON configuration file located at /etc/sensu/conf.d/mail_handler.json. This handler definition uses the mailx unix command, to email the event data to example@address.com, with the email subject sensu event. The handler is named mail.

{
  "handlers": {
    "mail": {
      "type": "pipe",
      "command": "mailx -s 'sensu event' example@address.com"
    }
  }
}

Handler definition specification

Handler name(s)

Each handler definition has a unique handler name, used for the definition key. Every handler definition is within the "handlers": {} configuration scope.

  • A unique string used to name/identify the check
  • Cannot contain special characters or spaces
  • Validated with Ruby regex /^[\w\.-]+$/.match("handler-name")

HANDLER attributes

The following attributes are configured within the {"handlers": { "HANDLER": {} } } configuration scope (where HANDLER is a valid handler name).

type
description The handler type.
required true
type String
allowed values pipe, tcp, udp, transport, set
example
"type": "pipe"
filter
description The Sensu event filter (name) to use when filtering events for the handler.
required false
type String
example
"filter": "occurrences"
filters
description An array of Sensu event filters (names) to use when filtering events for the handler. Each array item must be a string.
required false
type Array
example
"filters": ["occurrences", "production"]
severities
description An array of check result severities the handler will handle.NOTE: event resolution bypasses this filtering.
required false
type Array
allowed values warning, critical, unknown
example
"severities": ["critical", "unknown"]
mutator
description The Sensu event mutator (name) to use to mutate event data for the handler.
required false
type String
example
"mutator": "only_check_output"
timeout
description The handler execution duration timeout in seconds (hard stop). Only used by pipe and tcp handler types.
required false
type Integer
example
"timeout": 30
handle_silenced
description If events matching one or more silence entries should be handled.
required false
type Boolean
default false
example
"handle_silenced": true
handle_flapping
description If events in the flapping state should be handled.
required false
type Boolean
default false
example
"handle_flapping": true
command
description The handler command to be executed. The event data is passed to the process via STDIN.NOTE: the command attribute is only supported for Pipe handlers (i.e. handlers configured with "type": "pipe").
required true (if type == pipe)
type String
example
"command": "/etc/sensu/plugins/pagerduty.rb"
socket
description The socket definition scope, used to configure the TCP/UDP handler socket.NOTE: the socket attribute is only supported for TCP/UDP handlers (i.e. handlers configured with "type": "tcp" or "type": "udp").
required true (if type == tcp or udp)
type Hash
example
"socket": {}
pipe
description The pipe definition scope, used to configure the Sensu transport pipe.NOTE: the pipe attribute is only supported for Transport handlers (i.e. handlers configured with "type": "transport").
required true (if type == transport)
type Hash
example
"pipe": {}
handlers
description An array of Sensu event handlers (names) to use for events using the handler set. Each array item must be a string.NOTE: the handlers attribute is only supported for handler sets (i.e. handlers configured with "type": "set").
required true (if type == set)
type Array
example
"handlers": ["pagerduty", "email", "ec2"]

socket attributes

The following attributes are configured within the {"handlers": { "HANDLER": { "socket": {} } } } configuration scope (where HANDLER is a valid handler name).

NOTE: socket attributes are only supported for TCP/UDP handlers (i.e. handlers configured with "type": "tcp" or "type": "udp").

EXAMPLE
{
  "handlers": {
    "example_handler": {
      "type": "tcp",
      "socket": {
        "host": "10.0.5.100",
        "port": 8000
      }
    }
  }
}
ATTRIBUTES
host
description The socket host address (IP or hostname) to connect to.
required true
type String
example
"host": "8.8.8.8"
port
description The socket port to connect to.
required true
type Integer
example
"port": 4242

pipe attributes

The following attributes are configured within the {"handlers": { "HANDLER": { "pipe": {} } } } configuration scope (where HANDLER is a valid handler name).

NOTE: pipe attributes are only supported for Transport handlers (i.e. handlers configured with "type": "transport").

EXAMPLE
{
  "handlers": {
    "example_handler": {
      "type": "transport",
      "pipe": {
        "type": "topic",
        "name": "example_transport_handler"
      }
    }
  }
}
ATTRIBUTES
type
description The Sensu transport pipe type.
required true
type String
allowed values direct, fanout, topic
example
"type": "direct"

NOTE: types direct, fanout and topic are supported by the default RabbitMQ transport. Redis and other transports may only implement a subset of these.

name
description The Sensu transport pipe name.
required true
type String
example
"name": "graphite_plaintext"
options
description The Sensu transport pipe options. These options may be specific to the Sensu transport in use.
required false
type Hash
default {}
example
"options": {"durable": true}