Clients

Reference documentation

• What is a Sensu client?
• Client keepalives
  • What is a client keepalive?
  • Client registration & the client registry
    • Registration and deregistration events
    • Proxy clients
  • How are keepalive events created?
  • Client keepalive configuration
• Client signature
• Client subscriptions
  • What is a client subscription?
  • Round-robin client subscriptions
  • Client subscription configuration
• Client socket input
  • What is the Sensu client socket
  • Example client socket usage
  • Client socket configuration
• Standalone check execution scheduler
• Managing clients
• Client configuration
  • Example client definition
  • Client definition specification
    • client attributes
    • socket attributes
    • http-socket attributes
    • keepalive attributes
    • thresholds attributes
    • registration attributes
    • deregistration attributes
    • ec2 attributes
    • chef attributes
    • puppet attributes
    • servicenow attributes
    • influxdb attributes
    • opsgenie attributes
    • Custom attributes

What is a Sensu client?

Sensu clients are monitoring agents, which are installed and run on every system (e.g. server, container, etc) that needs to be monitored. The client is responsible for registering the system with Sensu, sending client keepalive messages (the Sensu heartbeat mechanism), and executing monitoring checks. Each client is a member of one or more subscriptions – a list of roles and/or responsibilities assigned to the system (e.g. a webserver, database, etc). Sensu clients will “subscribe” to (or watch for) check requests published by the Sensu server (via the Sensu Transport), execute the corresponding requests locally, and publish the results of the check back to the transport (to be processed by a Sensu server).

Client keepalives

What is a client keepalive?

Sensu Client keepalives are the heartbeat mechanism used by Sensu to ensure that all registered Sensu clients are still operational and able to reach the Sensu Transport. Sensu clients publish keepalive messages containing client configuration data to the Sensu transport every 20 seconds. If a Sensu client fails to send keepalive messages over a period of 120 seconds (by default), the Sensu server (or Sensu Enterprise) will create a keepalive event. Keepalives can be used to identify unhealthy systems and network partitions (among other things), and keepalive events can trigger email notifications and other useful actions.

Among those actions is communicating with external sources of truth to determine if the client should still be active. Those sources of truth may be something like a Chef server, a Puppet master, or AWS. This is especially useful when combined with deregistration events. Adopting this pattern results in clients being deregistered only when a client keepalive threshold has been exceeded.

Client registration & the client registry

In practice, client registrations happens when a Sensu server processes a client keepalive message for a client that is not already registered in the Sensu client registry (based on the configured client name or source attribute). This client registry is stored in the Sensu data store, and is accessible via the Sensu Clients API.

All Sensu client data provided in client keepalive messages gets stored in the client registry, which data is used to add context to Sensu Events and to detect Sensu clients in an unhealthy state.

Registration events

If a Sensu event handler named registration is configured, or if a Sensu client definition includes a registration attribute, the Sensu server will create and process a Sensu event for the client registration, applying any configured filters and mutators before executing the configured handler(s).

Registration events are useful for executing one-time handlers for new Sensu clients. For example, registration event handlers can be used to update external Configuration Management Databases (CMDBs) such as ServiceNow, etc.

To configure a registration event handler, please refer to the Sensu event handler documentation for instructions on creating a handler named registration. Alternatively, please see Client definition registration attributes, below.

WARNING: registration events are not stored in the event registry (in the Sensu data store), so they are not accessible via the Sensu API; however, all registration events are logged in the Sensu server log.

Deregistration events

Similarly to registration events, Sensu can create and process a deregistration event when the Sensu client process stops. You can use deregistration events to trigger a handler that updates external CMDBs or performs an action to update ephemeral infrastructures.

By default, deregistration events trigger the built-in deregistration extension. To enable deregistration events, set the client deregister flag to true. This results in the client being immediately depopulated from the client registry in the event that the sensu-client process is administratively stopped. If you require additional logic or actions to be applied when deregistration events are received, specify your custom deregistration event handler using the client deregistration definition scope.

NOTE: To ensure visibility into stopped clients, we recommend implementing a handler inside the keepalive scope to deregister clients based on a keepalive exceeding a threshold. For the vast majority of Sensu use cases, having clients deregister immediately does not produce the desired effect.

Proxy clients

Sensu proxy clients (formerly known as “Just-in-time” or “JIT” clients) are dynamically created clients, added to the client registry if a client does not already exist for a check result.

Proxy client registration differs from keepalive-based registration because the registration event happens while processing a check result (not a keepalive message). If a check result includes a source attribute, a proxy client will be created for the source, and the check result will be stored under the newly created client. Sensu proxy clients allow Sensu to monitor external resources (e.g. on systems and/or devices where a sensu-client cannot be installed, such a network switches), using the defined check source to create a proxy clients for the external resource. Once created, proxy clients work much in the same way as any other Sensu client; e.g. they are used to store check execution history and provide context within event data.

NOTE: keepalive monitoring is not supported for proxy clients, as they are inherently unable to run a sensu-client.

Proxy client example

Proxy clients are created when a check result includes a source attribute. See the example check definition below:

{
  "check": {
    "command": "check-http.rb -u https://sensu.io",
    "subscribers": [
      "demo"
    ],
    "interval": 60,
    "name": "sensu-website",
    "source": "sensuapp.org"
  }
}

NOTE: This source attribute can be provided in a check definition, or included in a check result published to the Sensu client input socket.

By default, proxy client data includes a minimal number of attributes. The following is an example of proxy client data that is added to the registry.

{
  "name": "switch-x",
  "address": "unknown",
  "subscriptions": [],
  "keepalives": false
}

The Sensu API can be used to update proxy client data in the client registry. To update proxy client data, please refer to the Client API reference documentation.

Create a proxy client via the client socket

The following is an example of how to create a proxy client payload via the client socket, using netcat:

echo '{"source": "mysql_01", "name": "app_01", "output": "could not connect to mysql", "status": 1}' | nc localhost 3030

How are keepalive events created?

Sensu servers (including Sensu Enterprise) monitor the Sensu client registry for stale client data, detecting clients that have failed to send client keepalive messages for more than 120 seconds (by default). When a Sensu server detects that a client hasn’t sent a keepalive message within the configured threshold, the Sensu server (or Sensu Enterprise) will create an event (this is different from how events are created for monitoring checks; see “How are Sensu events created?”).

Client keepalive configuration

For more information on configuring client keepalives, please see the client keepalive attribute reference documentation (below).

Sensu client keepalives are published to the Sensu transport every 20 seconds. Client keepalive behavior can be configured per Sensu client, allowing each Sensu client to set its own alert thresholds and keepalive event handlers. By default, client data is considered stale if a keepalive hasn’t be received in 120 seconds (WARNING). By default, keepalive events will be sent to the Sensu handler named keepalive if defined, or the default handler will be used.

To configure the keepalive check for a Sensu client, please refer to the client keepalive attributes reference documentation.

Client signature

Sensu client definitions may specify a unique string identifier as their signature, which is included in each keepalive message.

Once a client advertises a signature via keepalives, the server will expect that signature to be present in any keepalives or check results originating from the client. Any keepalives or check results which do not contain a matching signature will be dropped with an “invalid client signature” warning in the log.

A client can begin to use a signature if one was not previously configured, but removing a client signature requires deleting the client.

Providing a unique client signature prevents other clients from accidentally or maliciously submitting keepalives or check results using the same client name.

Client subscriptions

What is a client subscription?

Sensu’s use of the publish/subscribe pattern of communication allows for automated registration & de-registration of ephemeral systems. At the core of this model are Sensu client subscriptions.

Each Sensu client has a defined set of subscriptions, a list of roles and/or responsibilities assigned to the system (e.g. a webserver, database, etc). These subscriptions determine which monitoring checks are executed by the client. Client subscriptions allow Sensu to request check executions on a group of systems at a time, instead of a traditional 1:1 mapping of configured hosts to monitoring checks. Sensu checks target Sensu client subscriptions, using the check definition attribute subscribers.

Round-robin client subscriptions

Round-robin client subscriptions allow checks to be executed on a single client within a subscription, in a round-robin fashion. To create a round-robin client subscription, prepend the subscription name with roundrobin:, e.g. roundrobin:elasticsearch. Any check that targets the roundrobin:elasticsearch subscription will have its check requests sent to clients in a round-robin fashion, meaning only one member (client) in the subscription will execute a roundrobin check each time it is published.

The following is a Sensu client definition that includes a round-robin subscription.

{
  "client": {
    "name": "i-424242",
    "address": "8.8.8.8",
    "subscriptions": [
      "production",
      "webserver",
      "roundrobin:webserver"
    ]
  }
}

The following is a Sensu check definition that targets a round-robin subscription.

{
  "checks": {
    "web_application_api": {
      "command": "check-http.rb -u https://localhost:8080/api/v1/health",
      "subscribers": [
        "roundrobin:webserver"
      ],
      "interval": 20
    }
  }
}

Client subscription configuration

To configure Sensu client subscriptions for a client, please refer to the client subscriptions attribute reference documentation.

In addition to the subscriptions defined in the client configuration, Sensu clients also subscribe automatically to a subscription matching their client name. For example, a client named i-424242 will subscribe to check requests for the subscription client:i-424242. This makes it possible to generate ad-hoc check requests targeting specific clients via the /request API.

Client socket input

What is the Sensu client socket?

Every Sensu client has a TCP, UDP and HTTP socket listening for external check result input. The Sensu client socket(s) listen by default on localhost port 3030 for TCP/UDP and 3031 for HTTP and expect JSON formatted check results, allowing external sources (e.g. your web application, backup scripts, etc.) to push check results without needing to know anything about Sensu’s internal implementation. An excellent client socket use case example is a web application pushing check results to indicate database connectivity issues.

To configure the Sensu client socket for a client, please refer to the client socket attributes.

HTTP Socket

The HTTP socket, just like the TCP and UDP sockets, accepts check results, but it requires a well-formed HTTP request and exposes other functionality that is not possible with the raw TCP/UDP sockets. In exchange for a bit more complexity, the HTTP socket interface has the advantage of being more expressive than a raw TCP/UDP socket, both in the requests that it accepts and how it responds, and so exposes more functionality. The following endpoints are available for the HTTP socket:

GET /info

This endpoint returns 200 OK with some basic Sensu information like the version and transport metrics.

POST /results This endpoint expects an application/json body with a check result

GET /settings

This endpoint responds with 200 OK and the sensu configuration. Due to the possible sensitive nature of the client settings (e.g passwords might be in the config) this endpoint is protected using HTTP basic authentication and by default the information returned is redacted (e.g common config keys like ‘password’ have their values replaced by ‘REDACTED’). See the ‘redact’ configuration option if you need control over what is redacted. This endpoint accepts the optional query string ?redacted=false to disable redaction.

Refer to the http client socket attributes for details on configuring the HTTP basic authentication details that this endpoint requires.

GET /brew

This endpoint gets you some fresh coffee. Try it!

Any requests for unknown endpoints will get a 404 with some help information in the body.

At the moment only unsecure HTTP (no HTTPS) is supported.

To configure the Sensu HTTP client socket, please refer to the http client socket attributes.

Example client socket usage

The following is an example demonstrating external check result input via the Sensu client TCP socket. The example uses Bash’s built-in /dev/tcp file to communicate with the Sensu client socket.

echo '{"name": "app_01", "output": "could not connect to mysql", "status": 1}' > /dev/tcp/localhost/3030

Netcat can also be used, instead of the TCP file:

echo '{"name": "app_01", "output": "could not connect to mysql", "status": 1}' | nc localhost 3030

You can do the same using the HTTP socket:

curl -v -H "Content-Type: application/json" -X POST -d '{"name": "app_01", "output": "could not connect to mysql", "status": 1}' localhost:3031/results

Creating a “dead man’s switch”

The Sensu client socket(s) in combination with check TTLs can be used to create what’s commonly referred to as “dead man’s switches”. Outside of the software industry, a dead man’s switch is a switch that is automatically triggered if a human operator becomes incapacitated (source: Wikipedia). Sensu is more interested in detecting silent failures than incapacitated human operators. By using Check TTLs, Sensu is able to set an expectation that a Sensu client will continue to publish results for a check at a regular interval. If a Sensu client fails to publish a check result and the check TTL expires, Sensu will create an event to indicate the silent failure. For more information on check TTLs, please refer to the check attributes reference documentation.

A great use case for the Sensu client socket is to create a dead man’s switch for backup scripts, to ensure they continue to run successfully at regular intervals. If an external source sends a Sensu check result with a check TTL to the Sensu client socket, Sensu will expect another check result from the same external source before the TTL expires.

The following is an example of external check result input via the Sensu client TCP socket, using a check TTL to create a dead man’s switch for MySQL backups. The example uses a check TTL of 25200 seconds (or 7 hours). A MySQL backup script using the following code would be expected to continue to send a check result at least once every 7 hours or Sensu will create an event to indicate the silent failure.

echo '{"name": "backup_mysql", "ttl": 25200, "output": "backed up mysql successfully | size_mb=568", "status": 0}' | nc localhost 3030

echo '{"name": "backup_mysql", "ttl": 25200, "output": "failed to backup mysql", "status": 1}' | nc localhost 3030

It is also worth noting that you can set the attribute ttl_status, which will change the exit code from its default (“1”) to a different exit code. You can see an example of this in the check attributes reference documentation linked above.

Standalone check execution scheduler

In addition to subscribing to client subscriptions and executing check requests published by the Sensu server, the Sensu client is able to maintain its own/separate schedule for standalone checks.

Because the Sensu client shares the same check scheduling algorithm as the Sensu server, it is not only possible to have consistency between subscription checks and standalone checks — it’s also possible to maintain consistency between standalone checks across an entire infrastructure (assuming that system clocks are synchronized via NTP).

Managing clients

Delete a client

While shutting down a client stops all check results and keepalives, it doesn’t affect the history of the client or remove existing keepalive results. To delete a client’s history from Sensu, use the Clients API DELETE endpoint or the x icon in the dashboard client view.

Rename a client

To rename a client, modify the name attribute in the client configuration, then restart the client and use the Clients API DELETE endpoint or the x icon in the dashboard client view to remove the history and keepalive results tied to the former client name. See the appropriate platform page for instructions on restarting the Sensu client.

Client configuration

Example client definition

The following is an example Sensu client definition, a JSON configuration file located at /etc/sensu/conf.d/client.json. This client definition provides Sensu with information about the system on which it resides. This is a production system, running a web server and a MySQL database. The client ’name’ attribute is required in the definition, and must be unique.

{
  "client": {
    "name": "i-424242",
    "address": "8.8.8.8",
    "subscriptions": [
      "production",
      "webserver",
      "mysql"
    ],
    "socket": {
      "bind": "127.0.0.1",
      "port": 3030
    }
  }
}

Client definition specification

The client definition uses the { "client": {} } configuration scope.

client attributes

"name": "i-424242"

"address": "8.8.8.8"

"subscriptions": ["production", "webserver"]

"safe_mode": true

[
  "password", "passwd", "pass",
  "api_key", "api_token", "access_key",
  "secret_key", "private_key","secret"
]

"redact": [
  "password",
  "ec2_access_key",
  "ec2_secret_key"
]

"socket": {}

"keepalives": false

"keepalive": {}

"signature": "yVNxtPbRGwCYFYEr3V"

"registration": {}

"deregister": true

"deregistration": {}

"ec2": {}

"chef": {}

"puppet": {}

"servicenow": {}

"influxdb": {}

"opsgenie": {}

socket attributes

The following attributes are configured within the { "client": { "socket": {} } } configuration scope.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "socket": {
      "bind": "127.0.0.1",
      "port": 3030
    }
  }
}

ATTRIBUTES

"enabled": false

"bind": "0.0.0.0"

"port": 3032

http socket attributes

The following attributes are configured within the { "client": { "http_socket": {} } } configuration scope.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "http_socket": {
      "bind": "127.0.0.1",
      "port": 3031,
      "user": "http-auth-user-name",
      "password": "use-somethign-secure-here"
    }
  }
}

ATTRIBUTES

"bind": "0.0.0.0"

"port": 8000

"user": "sensu"

"password": "F76639PML6c7sk5nI46N"

"protect_all_endpoints": true

keepalive attributes

The following attributes are configured within the { "client": { "keepalive": {} } } configuration scope.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "keepalive": {
      "handler": "pagerduty",
      "thresholds": {
        "warning": 40,
        "critical": 60
      }
    }
  }
}

ATTRIBUTES

"handler": "pagerduty"

"handlers": ["pagerduty", "chef"]

"thresholds": {}

thresholds attributes (for client keepalives)

The following attributes are configured within the { "client": { "keepalive": { "thresholds": {} } } } configuration scope.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "keepalive": {
      "...": "...",
      "thresholds": {
        "warning": 40,
        "critical": 60
      }
    }
  }
}

ATTRIBUTES

"warning": 60

"critical": 90

registration attributes

The following attributes are configured within the { "client": { "registration": {} } } configuration scope.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "registration": {
      "handler": "servicenow"
    }
  }
}

ATTRIBUTES

"handler": "registration_cmdb"

NOTE: client registration attributes are used to generate check result data for the registration event. Client registration attributes are merged with some default check definition attributes by the Sensu server during client registration, so any valid check definition attributes – including custom check definition attributes – may be used as registration attributes. The following attributes are provided as recommendations for controlling client registration behavior.

deregistration attributes

The following attributes are configured within the { "client": { "deregistration": {} } } configuration scope.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "deregister": true,
    "deregistration": {
      "handler": "deregister_client"
    }
  }
}

ATTRIBUTES

NOTE: client deregistration attributes are used to generate check result data for the de-registration event. Client deregistration attributes are merged with some default check definition attributes by the Sensu server during client deregistration, so any valid check definition attributes – including custom check definition attributes – may be used as deregistration attributes, with the following exceptions (which are used to ensure the check result is valid): check name, output, status, and issued timestamp. The following attributes are provided as recommendations for controlling client deregistration behavior.

"handler": "cmdb_deregistration"

ec2 attributes

The following attributes are configured within the { "client": { "ec2": {} } } configuration scope.

ENTERPRISE: This configuration is provided for using the built-in Sensu Enterprise AWS EC2 integration.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "keepalive": {
      "handlers": [
        "ec2"
      ]
    },
    "ec2": {
      "instance_id": "i-424242",
      "account": "sensu-testing",
      "allowed_instance_states": [
        "running",
        "rebooting"
      ]
    }
  }
}

NOTE: In order for the EC2 integration to perform automatic deregistration, the ec2 handler must be specified in the keepalive scope as in the example above.

ATTRIBUTES

"instance_id": "i-424242"

"allowed_instance_states": [
  "pending",
  "running",
  "rebooting"
]

"region": "us-west-1"

"access_key_id": "AlygD0X6Z4Xr2m3gl70J"

"secret_access_key": "y9Jt5OqNOqdy5NCFjhcUsHMb6YqSbReLAJsy4d6obSZIWySv"

"timeout": 30

"account": "sensu-testing"

chef attributes

The following attributes are configured within the { "client": { "chef": {} } } configuration scope.

ENTERPRISE: This configuration is provided for using the built-in Sensu Enterprise Chef integration.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "chef": {
      "node_name": "webserver01"
    }
  }
}

ATTRIBUTES

"node_name": "webserver01"

"endpoint": "https://api.chef.io/organizations/example"

"flavor": "enterprise"

"client": "sensu-server"

"key": "/etc/chef/i-424242.pem"

"ssl_pem_file": "/etc/chef/ssl.pem"

"ssl_verify": false

"proxy_address": "proxy.example.com"

"proxy_port": 8080

"proxy_username": "chef"

"proxy_password": "secret"

"timeout": 30

puppet attributes

The following attributes are configured within the { "client": { "puppet": {} } } configuration scope.

ENTERPRISE: This configuration is provided for using the built-in Sensu Enterprise Puppet integration.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "puppet": {
      "node_name": "webserver01"
    }
  }
}

ATTRIBUTES

"node_name": "webserver01"

servicenow attributes

The following attributes are configured within the { "client": { "servicenow": {} } } configuration scope.

ENTERPRISE: this configuration is provided for using the built-in Sensu Enterprise ServiceNow integration.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "servicenow": {
      "configuration_item": {
        "name": "webserver01"
      },
      "incident": {
        "product_team": "onboarding"
      }
    }
  }
}

ATTRIBUTES

"configuration_item": {
  "name": "webserver01"
}

"incident": {
  "product_team": "onboarding"
}

influxdb attributes

The following attributes are configured within the { "client": { "influxdb": {} } } configuration scope.

ENTERPRISE: This configuration is provided for using the built-in Sensu Enterprise InfluxDB integration.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "influxdb": {
      "tags": {
        "dc": "us-central-1"
      }
    }
  }
}

ATTRIBUTES

{}

"tags": {
  "dc": "us-central-1"
}

opsgenie attributes

The following attributes are configured within the { "client": { "opsgenie": {} } } configuration scope.

ENTERPRISE: This configuration is provided for using the built-in Sensu Enterprise OpsGenie integration.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "opsgenie": {
      "tags": ["production"]
    }
  }
}

ATTRIBUTES

"tags": ["production"]

configuration_item attributes

The following attributes are configured within the { "client": { "servicenow": { "configuration_item": {} } } } configuration scope.

EXAMPLE

{
  "client": {
    "name": "i-424242",
    "...": "...",
    "servicenow": {
      "configuration_item": {
        "name": "webserver01",
        "os_version": "14.04"
      }
    }
  }
}

PRO TIP: ServiceNow users may provide custom Configuration Item (CI) field values via the configuration_item configuration scope. In this example, the CI field os_version is being set to 14.04.

ATTRIBUTES

"name": "webserver01.example.com"

Custom attributes

Because Sensu configuration is just JSON data, it is possible to define configuration attributes that are not part of the Sensu client specification. Custom client definition attributes may be defined to provide context about the Sensu client and the services that run on its system. Custom client attributes will be included in client keepalives, and event data and can be used by Sensu filters (e.g. only alert on events in the “production” environment), and accessed via check token substitution.

EXAMPLE

The following is an example Sensu client definition that has custom attributes for the environment it is running in, a mysql attribute containing information about a local database, and a link to an operational playbook.

{
  "client": {
    "name": "i-424242",
    "address": "10.0.2.101",
    "environment": "production",
    "subscriptions": [
      "production",
      "webserver",
      "mysql"
    ],
    "mysql": {
      "host": "10.0.2.101",
      "port": 3306,
      "user": "app",
      "password": "secret"
    },
    "playbook": "https://wiki.example.com/ops/mysql-playbook"
  }
}

NOTE: Because client data is included in alerts created by Sensu, custom attributes that only exist for the purpose of providing troubleshooting information for operations teams can be extremely valuable.