Sensu Configuration

Reference documentation

• How does Sensu load configuration?
  • Sensu configuration sources
  • Configuration load order
  • Configuration merging
  • Configuration logging
  • Configuration scopes
    • Configuration scopes are relative
    • Configuration scope examples
• Sensu service init configuration
  • Sensu service init configuration variables
• Sensu command line interfaces and arguments
• Sensu environment variables
• Sensu configuration specification
  • Example sensu configuration
  • Top-level configuration scopes
• Sensu definition specification

How does Sensu load configuration?

Sensu configuration sources

By default, the main configuration file for the Sensu platform is located at /etc/sensu/config.json. However, Sensu also provides support for loading configuration from a directory (containing multiple/disparate configuration files), and/or environment variables. Sensu merges configuration parameters provided from these three distinct configuration sources (environment variables, configuration file, and configuration directories) into a single Hash. This type of Hash merging is often called “deep merging”, and is probably the most important concept to understand when learning how to configure Sensu.

Configuration load order

As previously mentioned, Sensu configuration can be provided via three distinct sources: environment variables, a configuration file, and one or more directories containing configuration files. Sensu loads configuration from these sources in the following order:

1. The Sensu service scripts provide command line arguments for starting the Sensu services (e.g. the location of the configuration file (-c), the location of configuration directories (-d), etc).

2. Sensu will load configuration from environment variables.

3. Sensu loads configuration from the configuration file (by default, this is located at /etc/sensu/config.json).

4. Sensu loads configuration snippets from configuration files located in a Sensu configuration directory (by default, this is /etc/sensu/conf.d/, however it is possible to configure Sensu to load from multiple configuration directories; see: command line arguments, below).

   NOTE: configuration file load order is dictated by a *.json glob of the configuration directory; as such it is strongly recommended to avoid a dependency on configuration directory file load order (e.g. if you’re attempting to name configuration files in the configuration directory to control load order, you’re doing it wrong).

Configuration merging

To explain how Sensu merges configuration parameters from the various disparate configuration sources, please note the following example scenario:

1. The Sensu runtime configuration is stored as an in-memory Hash object. For the purposes of providing a visual example, let’s imagine that this Hash object is actually a JSON document, which begins life (as Sensu is started) as an empty JSON document ({}).

   When Sensu is started, it will begin to collect configuration from environment variables, a configuration file, and one or more configuration directories, which configuration parameters will be used to build up this configuration Hash.

2. For the purposes of this example, let’s assume that the first configuration snippet that Sensu encounters is a configuration file, located on disk at /etc/sensu/config.json with the following contents:

   {
    "rabbitmq": {
      "host": "10.0.1.10",
      "vhost": "/sensu",
      "user": "sensu",
      "password": "secret"
    },
    "redis": {
      "host": "10.0.1.30",
      "port": 6379,
      "password": "secret"
    }
   }

   At this time, the updated Sensu configuration Hash (in memory) will look like the following example:

   {
    "rabbitmq": {
      "host": "localhost",
      "vhost": "/sensu",
      "user": "sensu",
      "password": "secret"
    },
    "redis": {
      "host": "localhost",
      "port": 6379,
      "password": "secret"
    }
   }

3. Now let’s see what happens when Sensu encounters another configuration snippet (e.g. a file located on disk in a Sensu configuration directory, such as /etc/sensu/conf.d/rabbitmq.json):

   {
    "rabbitmq": {
      "host": "10.0.1.10",
      "user": "sensu01",
      "password": "newsecret"
    }
   }

   The second configuration snippet provided configuration for the rabbitmq scope, some of which already exists in the Sensu configuration Hash (in memory) - but also missing some attributes which already exist in the Sensu configuration Hash (i.e. vhost). The result of merging this configuration snippet into the Sensu configuration Hash (in memory) is as follows:

   {
    "rabbitmq": {
      "host": "10.0.1.10",
      "vhost": "/sensu",
      "user": "sensu01",
      "password": "newsecret"
    },
    "redis": {
      "host": "localhost",
      "port": 6379,
      "password": "secret"
    }
   }

   The result of the deep merge is that the configuration snippet provided by /etc/sensu/conf.d/rabbitmq.json was overlaid on the Sensu configuration Hash (in memory), essentially overwriting the previously existing values provided by the configuration snippet, while not discarding configuration attributes that already existed in the rabbitmq configuration scope - even though they weren’t provided by the configuration snippet.

Configuration logging

As configuration snippets are applied to the Sensu configuration Hash (i.e. during “deep merge”), all configuration changes are logged to the corresponding log file (e.g. the Sensu server, API, or Client logs).

Configuration scopes

Because Sensu configuration can be provided in so many different sources, it is important to understand that – regardless of the physical location of the configuration data (e.g. from the main configuration file, or from a configuration file in a configuration directory) – all configuration must be placed in the appropriate “scope” in the JSON file (i.e. the named “level” that attributes should be defined in).

For example, the “root” or scope of the Sensu configuration would be any attributes defined at the top “level” of a JSON configuration file, such as the configuration attributes for rabbitmq, redis, or the api:

{
  "rabbitmq": {},
  "redis": {},
  "api": {}
}

Attributes defined in the root scope (or top “level”) provide the corresponding scope(s) for additional configuration settings (e.g. the rabbitmq attribute defined above provides the rabbitmq scope, a JSON Hash, for the actual RabbitMQ configuration settings).

Configuration scopes are relative

Throughout the Sensu documentation whenever a configuration scope is mentioned, it is describing the named “level” that the corresponding configuration attributes should be defined within, which may be relative to any potentially related scopes. Please note the following examples:

Configuration scope examples

The client scope ("client": {})

In the Sensu Client reference documentation it explains that:

“The client definition uses the "client": {} definition scope.”

Which means that, regardless where you might store a configuration file containing Sensu client configuration on disk (assuming it is in a location that will be loaded by Sensu), the file should contain a top “level” attribute called "client":

{
  "client": {}
}

The client socket scope ("socket": {})

The Sensu Client reference documentation continues to explain that Sensu clients may have a "socket" attribute, and that there are additional Client Socket attributes which should be defined within the "socket" scope:

“The following attributes are configured within the "socket": {} client definition attribute scope.”

Which means that, regardless where you might store a configuration file containing Sensu Client Socket configuration on disk (assuming it is in a location that will be loaded by Sensu), the file should contain a top “level” attribute called "client", and another attribute defined within the "client" scope (or level) called "socket":

{
  "client": {
    "socket": {}
  }
}

Thus, when the Client Socket reference documentation continues to explain that the bind and port attributes should be defined in the Client Socket scope, it means they should live under the "socket" “level” of the JSON file, regardless of where you might store a configuration file containing said configuration attributes on disk (assuming it is in a location that will be loaded by Sensu).

{
  "client": {
    "socket": {
      "bind": "0.0.0.0",
      "port": 3031
    }
  }
}

Sensu service service configuration

The Sensu services are managed by service scripts that are provided in the Sensu packages. The Sensu service scripts are able to start/stop/restart the corresponding Sensu services (e.g. the Sensu server, API, client, etc).

The default Sensu service scripts and related configuration file(s) (containing configuration variables) are located as follows:

• /etc/default/sensu (sourced by the init scripts to provide configuration variables)
• /etc/init.d/sensu-service (shared init script used by the Sensu service init scripts)
• /etc/init.d/sensu-server
• /etc/init.d/sensu-api
• /etc/init.d/sensu-client
• /etc/init.d/sensu-enterprise
• /etc/init.d/sensu-enterprise-dashboard

Sensu service script configuration variables

The following configuration variables can be set in the service script(s) for the platform.

EMBEDDED_RUBY=true

CONFIG_FILE=/etc/sensu/config.json

CONFD_DIR=/etc/sensu/conf.d

EXTENSION_DIR=/etc/sensu/extensions

PLUGINS_DIR=/etc/sensu/plugins

HANDLERS_DIR=/etc/sensu/handlers

LOG_DIR=/var/log/sensu

LOG_LEVEL=info

PID_DIR=/var/run/sensu

USER=sensu

SERVICE_MAX_WAIT=10

Sensu command line interfaces and arguments

The Sensu services can be run from the command line with the following command line options:

NOTE: these options will work with ALL of the Sensu services (sensu-server, sensu-api, sensu-client, and sensu-enterprise).

$ /opt/sensu/bin/sensu-server -h
Usage: sensu-server [options]
  -h, --help                       Display this message
  -V, --version                    Display version
  -c, --config FILE                Sensu JSON config FILE
  -d, --config_dir DIR[,DIR]       DIR or comma-delimited DIR list for Sensu JSON config files
  -P, --print_config               Print the compiled configuration and exit
  -e, --extension_dir DIR          DIR for Sensu extensions
  -l, --log FILE                   Log to a given FILE. Default: STDOUT
  -L, --log_level LEVEL            Log severity LEVEL
  -v, --verbose                    Enable verbose logging
  -b, --background                 Fork into the background
  -p, --pid_file FILE              Write the PID to a given FILE

$ /opt/sensu/bin/sensu-install -h
Usage: sensu-install [options]
  -h, --help                       Display this message
  -v, --verbose                    Enable verbose logging
  -p, --plugin PLUGIN              Install a Sensu PLUGIN
  -P, --plugins PLUGIN[,PLUGIN]    PLUGIN or comma-delimited list of Sensu plugins to install
  -s, --source SOURCE              Install Sensu plugins from a custom SOURCE

$ /opt/sensu/bin/sensu-client -V
1.0.0

/opt/sensu/bin/sensu-api -c /etc/sensu/config.json

/opt/sensu/bin/sensu-server -d /etc/sensu/conf.d,/path/to/more/configuration

$ sudo -u sensu /opt/sensu/bin/sensu-client -P -c /etc/sensu/config.json -d /etc/sensu/conf.d/
{"timestamp":"2016-04-08T16:41:35.699200-0700","level":"warn","message":"loading config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-04-08T16:41:35.699411-0700","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d/"}
{"timestamp":"2016-04-08T16:41:35.699456-0700","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/check-sensu-website.json"}
{"timestamp":"2016-04-08T16:41:35.699551-0700","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/check-sensu-website.json","changes":{"checks":{"sensu_website":[null,{"command":"check-http.rb -u https://sensuapp.org","subscribers":["production"],"interval":60,"publish":false}]}}}
{"timestamp":"2016-04-08T16:41:35.699597-0700","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/client.json"}
{"timestamp":"2016-04-08T16:41:35.699673-0700","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/client.json","changes":{"client":[null,{"name":"client-01","address":"127.0.0.1","environment":"development","subscriptions":["production"],"socket":{"bind":"127.0.0.1","port":3030}}]}}
{"timestamp":"2016-04-08T16:41:35.700246-0700","level":"warn","message":"outputting compiled configuration and exiting"}
{
  "transport":{
    "name":"redis",
    "reconnect_on_error":true
  },
  "checks":{
    "sensu_website":{
      "command":"check-http.rb -u https://sensuapp.org",
      "subscribers":[
        "production"
      ],
      "interval":60,
      "publish":false
    }
  },
  "filters":{},
  "mutators":{},
  "handlers":{},
  "redis":{
    "host":"127.0.0.1"
  },
  "api":{
    "host":"127.0.0.1",
    "port":4567
  },
  "client":{
    "name":"client-01",
    "address":"127.0.0.1",
    "environment":"development",
    "subscriptions":[
      "production"
    ],
    "socket":{
      "bind":"127.0.0.1",
      "port":3030
    }
  }
}

$ sudo -u sensu /opt/sensu/bin/sensu-client --validate_config -c /etc/sensu/config.json -d /etc/sensu/conf.d/`
`configuration is valid

/opt/sensu/bin/sensu-server -e /etc/sensu/extensions

/opt/sensu/bin/sensu-server -l /var/log/sensu/sensu-server.log

/opt/sensu/bin/sensu-server -L warn

/opt/sensu/bin/sensu-client -v

/opt/sensu/bin/sensu-server -b

/opt/sensu/bin/sensu-server -p /var/run/sensu-server.pid

Sensu environment variables

The Sensu services are aware of the following environment variables. Configuration provided via environment variables will only be used if no corresponding configuration is provided via the Sensu configuration file, or configuration directories. Providing configuration via environment variables is primarily beneficial in environments where configuration management tools (e.g. Chef or Puppet) are not used – for example, in container-based environments.

SENSU_CLIENT_NAME="container-01"

SENSU_CLIENT_ADDRESS="8.8.8.8"

SENSU_CLIENT_SUBSCRIPTIONS="production,webserver,nginx,memcached,all"

SENSU_TRANSPORT_NAME="redis"

RABBITMQ_URL="amqp://user:password@hostname:5672/%2Fvhost"

REDIS_URL="redis://hostname:6379/0"

REDIS_SENTINEL_URLS="redis://sentinel1:26379,redis://sentinel2:26379,redis://sentinel3:26379/"

SENSU_API_PORT=8080

Sensu configuration specification

Example Sensu configuration

The following is an example Sensu configuration, a JSON configuration file located at /etc/sensu/config.json. This Sensu configuration provides Sensu with information it needs to communicate with RabbitMQ and Redis:

{
  "rabbitmq": {
    "host": "10.0.1.10",
    "vhost": "/sensu",
    "user": "sensu",
    "password": "secret"
  },
  "redis": {
    "host": "10.0.1.30",
    "port": 6379,
    "password": "secret"
  },
  "api": {
    "host": "10.0.1.30",
    "bind": "0.0.0.0",
    "port": "4242"
  }
}

Top-level configuration scopes

The top-level Sensu configuration scopes are as follows (these attributes live at the top level of their respective JSON documents):

{
  "rabbitmq": {
    "host": "10.0.1.10",
    "vhost": "/sensu",
    "user": "sensu",
    "password": "secret"
  }
}

{
  "redis": {
    "host": "10.0.1.30",
    "port": 6379,
    "password": "secret"
  }
}

{
  "transport": {
    "name": "rabbitmq"
  }
}

{
  "api": {
    "host": "10.0.1.30",
    "bind": "0.0.0.0",
    "port": 4242
  }
}

{
  "sensu": {
     "spawn": 24
  }
}

{
  "client": {
    "name": "i-424242",
    "address": "8.8.8.8",
    "subscriptions": [
      "production",
      "webserver",
      "mysql"
    ]
  }
}

{
  "checks": {
    "example_check": {},
    "another_check": {}
  }
}

{
  "handlers": {
    "example_handler": {},
    "another_handler": {}
  }
}

{
  "filters": {
    "example_filter": {},
    "another_filter": {}
  }
}

{
  "mutators": {
    "example_mutator": {},
    "another_mutator": {}
  }
}

{
  "deregistration": {
    "host": "10.0.0.1",
    "port": 14567
  }
}

Sensu definition specification

Sensu uses the "sensu": {} definition scope.

Sensu attributes

The following attributes are defined within the "sensu": {} definition scope.

{
  "sensu": {
    "global_error_handler": true
  }
}

{
  "sensu": {
    "spawn": 24
  }
}