How to run a Sensu cluster

What is a Sensu cluster?
Why use clustering?
Configuring a cluster
Adding sensu agents to the cluster
Cluster health
Managing cluster members
Security
Using an external etcd cluster
Troubleshooting

What is a Sensu cluster?

A Sensu cluster is a group of at least three sensu-backend nodes, each connected to a shared etcd cluster, using Sensu’s embedded etcd or an external etcd cluster. Creating a Sensu cluster ultimately configures an etcd cluster.

Why use clustering?

Clustering is important to make Sensu more highly available, reliable, and durable. It will help you cope with the loss of a backend node, prevent data loss, and distribute the network load of agents.

NOTE: We recommend using a load balancer to evenly distribute agent connections across the cluster.

Configuring a cluster

The sensu-backend arguments for its store mirror the etcd configuration flags, however the Sensu flags are prefixed with etcd. For more detailed descriptions of the different arguments, you can refer to the etcd docs or the Sensu backend reference.

You can configure a Sensu cluster in a couple different ways (we’ll show you a few below) but it’s recommended to adhere to some etcd cluster guidelines as well.

The recommended etcd cluster size is 3, 5 or 7, which is decided by the fault tolerance requirement. A 7-member cluster can provide enough fault tolerance in most cases. While a larger cluster provides better fault tolerance, the write performance reduces since data needs to be replicated to more machines. It is recommended to have an odd number of members in a cluster. Having an odd cluster size doesn’t change the number needed for majority, but you gain a higher tolerance for failure by adding the extra member (Core OS).

We also recommend using stable platforms to support your etcd instances (see etcd’s supported platforms).

Docker

If you’d prefer to stand up your Sensu cluster within Docker containers, check out the Sensu Go docker configuration. This configuration defines three sensu-backend containers and three sensu-agent containers.

Traditional computer instance

NOTE: The remainder of this guide uses on disk configuration. If you are using an ephemeral computer instance, you can use sensu-backend start --help to see examples of etcd command line flags. The configuration file entries below translate to sensu-backend flags.

Sensu backend configuration

Below are example configuration snippets from /etc/sensu/backend.yml using a three node cluster. The nodes are named backend-1, backend-2 and backend-3 with IP addresses 10.0.0.1, 10.0.0.2 and 10.0.0.3, respectively.

NOTE: This backend configuration assumes you have set up and installed the sensu-backend on all the nodes used in your cluster. You can use our installation and configuration guide guide if you have not done so.

backend-1

##
# store configuration for backend-1/10.0.0.1
##
etcd-advertise-client-urls: "http://10.0.0.1:2379"
etcd-listen-client-urls: "http://10.0.0.1:2379"
etcd-listen-peer-urls: "http://0.0.0.0:2380"
etcd-initial-cluster: "backend-1=http://10.0.0.1:2380,backend-2=http://10.0.0.2:2380,backend-3=http://10.0.0.3:2380"
etcd-initial-advertise-peer-urls: "http://10.0.0.1:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: ""
etcd-name: "backend-1"

backend-2

##
# store configuration for backend-2/10.0.0.2
##
etcd-advertise-client-urls: "http://10.0.0.2:2379"
etcd-listen-client-urls: "http://10.0.0.2:2379"
etcd-listen-peer-urls: "http://0.0.0.0:2380"
etcd-initial-cluster: "backend-1=http://10.0.0.1:2380,backend-2=http://10.0.0.2:2380,backend-3=http://10.0.0.3:2380"
etcd-initial-advertise-peer-urls: "http://10.0.0.2:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: ""
etcd-name: "backend-2"

backend-3

##
# store configuration for backend-3/10.0.0.3
##
etcd-advertise-client-urls: "http://10.0.0.3:2379"
etcd-listen-client-urls: "http://10.0.0.3:2379"
etcd-listen-peer-urls: "http://0.0.0.0:2380"
etcd-initial-cluster: "backend-1=http://10.0.0.1:2380,backend-2=http://10.0.0.2:2380,backend-3=http://10.0.0.3:2380"
etcd-initial-advertise-peer-urls: "http://10.0.0.3:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: ""
etcd-name: "backend-3"

Once each node has the configuration described above, start each sensu-backend:

sudo systemctl start sensu-backend

Adding sensu agents to the cluster

Each Sensu agent should have the following entries in /etc/sensu/agent.yml to ensure they are aware of all cluster members. This allows the agent to reconnect to a working backend if the backend it is currently connected to goes into an unhealthy state.

##
# backend-url configuration for all agents connecting to cluster over ws
##

backend-url:
  - "ws://10.0.0.1:8081"
  - "ws://10.0.0.2:8081"
  - "ws://10.0.0.3:8081"

You should now have a highly available Sensu cluster! You can verify its health and try other cluster management commands using sensuctl.

Sensuctl

Sensuctl has several commands to help you manage and monitor your cluster. See sensuctl cluster -h for additional help usage.

Cluster health

Get cluster health status and etcd alarm information.

sensuctl cluster health

       ID            Name                          Error                           Healthy  
────────────────── ─────────── ─────────────────────────────────────────────────── ─────────
a32e8f613b529ad4   backend-1                                                        true
c3d9f4b8d0dd1ac9   backend-2  dial tcp 10.0.0.2:2379: connect: connection refused   false
c8f63ae435a5e6bf   backend-3                                                        true

Add a cluster member

Add a new member node to an existing cluster.

sensuctl cluster member-add backend-4 https://10.0.0.4:2380

added member 2f7ae42c315f8c2d to cluster

ETCD_NAME="backend-4"
ETCD_INITIAL_CLUSTER="backend-4=https://10.0.0.4:2380,backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

List cluster members

List the ID, name, peer urls, and client urls of all nodes in a cluster.

sensuctl cluster member-list

       ID            Name             Peer URLs                Client URLs
────────────────── ─────────── ───────────────────────── ─────────────────────────
a32e8f613b529ad4   backend-1    https://10.0.0.1:2380     https://10.0.0.1:2379  
c3d9f4b8d0dd1ac9   backend-2    https://10.0.0.2:2380     https://10.0.0.2:2379
c8f63ae435a5e6bf   backend-3    https://10.0.0.3:2380     https://10.0.0.3:2379
2f7ae42c315f8c2d   backend-4    https://10.0.0.4:2380     https://10.0.0.4:2379

Remove a cluster member

Remove a faulty or decommissioned member node from a cluster.

sensuctl cluster member-remove 2f7ae42c315f8c2d

Removed member 2f7ae42c315f8c2d from cluster

Replace a faulty cluster member

Here’s how to replace a faulty cluster member to restore a cluster’s health.

First, run sensuctl cluster health to identify the faulty cluster member. For a faulty cluster member, the Error column will include an error message and the Healthy column will list false.

In this example, cluster member backend-4 is faulty:

sensuctl cluster health

       ID            Name                          Error                           Healthy  
────────────────── ─────────── ─────────────────────────────────────────────────── ─────────
a32e8f613b529ad4   backend-1                                                        true
c3d9f4b8d0dd1ac9   backend-2                                                        true
c8f63ae435a5e6bf   backend-3                                                        true
2f7ae42c315f8c2d   backend-4  dial tcp 10.0.0.4:2379: connect: connection refused   false

Second, delete the faulty cluster member. To continue this example, you will delete cluster member backend-4 using its ID field:

sensuctl cluster member-remove 2f7ae42c315f8c2d

Removed member 2f7ae42c315f8c2d from cluster

Third, add a newly created member to the cluster. You can use the same name and IP address as the faulty member you deleted, with one change to the configuration: specify the etcd-initial-cluster-state as existing.

etcd-advertise-client-urls: "http://10.0.0.4:2379"
etcd-listen-client-urls: "http://10.0.0.4:2379"
etcd-listen-peer-urls: "http://0.0.0.0:2380"
etcd-initial-cluster: "backend-1=http://10.0.0.1:2380,backend-2=http://10.0.0.2:2380,backend-3=http://10.0.0.3:2380,backend-4=http://10.0.0.4:2380"
etcd-initial-advertise-peer-urls: "http://10.0.0.4:2380"
etcd-initial-cluster-state: "existing"
etcd-initial-cluster-token: ""
etcd-name: "backend-4"

If replacing the faulty cluster member does not resolve the problem, please see the etcd operations guide for more information.

Update a cluster member

Update the peer URLs of a member in a cluster.

sensuctl cluster member-update c8f63ae435a5e6bf https://10.0.0.4:2380

Updated member with ID c8f63ae435a5e6bf in cluster

Security

Creating self-signed certificates

We will use the cfssl tool to generate our self-signed certificates.

The first step is to create a Certificate Authority (CA). In order to keep things simple, we will generate all our clients and peer certificates using this CA, but you might eventually want to create distinct CA.

echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json

Then, using that CA, we can generate certificates and keys for each peer (backend server) by specifying their Common Name (CN) and their hosts. A *.pem, *.csr and *.pem will be created for each backend.

export ADDRESS=10.0.0.1,backend-1
export NAME=backend-1
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" -profile=peer - | cfssljson -bare $NAME

export ADDRESS=10.0.0.2,backend-2
export NAME=backend-2
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" -profile=peer - | cfssljson -bare $NAME

export ADDRESS=10.0.0.3,backend-3
export NAME=backend-3
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" -profile=peer - | cfssljson -bare $NAME

We will also create generate a client certificate that can be used by clients to connect to the etcd client URL. This time, we don’t need to specify an address but simply a Common Name (CN) (here client). The files client-key.pem, client.csr and client.pem will be created.

export NAME=client
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="" -profile=client - | cfssljson -bare $NAME

See etcd’s guide to generating self signed certificates for detailed instructions.

Once done, you should have the following files created. The *.csr files will not be used in this guide.

backend-1-key.pem
backend-1.csr
backend-1.pem
backend-2-key.pem
backend-2.csr
backend-2.pem
backend-3-key.pem
backend-3.csr
backend-3.pem
ca-config.json
ca-key.pem
ca.csr
ca.pem
client-key.pem
client.csr
client.pem

Client-to-server transport security with HTTPS

Below are example configuration snippets from /etc/sensu/backend.yml on three Sensu backends named backend-1, backend-2 and backend-3 with IP addresses 10.0.0.1, 10.0.0.2 and 10.0.0.3 respectively. This configuration assumes that your client certificates are in /etc/sensu/certs/ and your CA certificate is in /usr/local/share/ca-certificates/sensu/.

##
# etcd peer ssl configuration for backend-1/10.0.0.1
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-1.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-1-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"

##
# etcd peer ssl configuration for backend-2/10.0.0.2
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-2.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-2-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"

##
# etcd peer ssl configuration for backend-3/10.0.0.3
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-3.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-3-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"

Validating with curl:

curl --cacert /usr/local/share/ca-certificates/sensu/ca.pem \
https://127.0.0.1:2379/v2/keys/foo -XPUT -d value=bar

Client-to-server authentication with HTTPS client certificates

Below are example configuration snippets from /etc/sensu/backend.yml on three Sensu backends named backend-1, backend-2 and backend-3 with IP addresses 10.0.0.1, 10.0.0.2 and 10.0.0.3 respectively. This configuration assumes your client certificates are in /etc/sensu/certs/ and your CA certificate is in /usr/local/share/ca-certificates/sensu/.

##
# etcd peer ssl configuration for backend-1/10.0.0.1
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-1.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-1-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"
etcd-client-cert-auth: true

##
# etcd peer ssl configuration for backend-2/10.0.0.2
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-2.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-2-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"
etcd-client-cert-auth: true

##
# etcd peer ssl configuration for backend-3/10.0.0.3
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-3.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-3-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"
etcd-client-cert-auth: true

Validating with curl, with a different certificate and key:

curl --cacert /usr/local/share/ca-certificates/sensu/ca.pem \
--cert /etc/sensu/certs/client.pem \
--key /etc/sensu/certs/client-key.pem \
-L https://127.0.0.1:2379/v2/keys/foo -XPUT -d value=bar

Peer communication authentication with HTTPS client certificates

NOTE: If you ran through the first part of the guide, you will need to update the store configuration for all backends to use https instead of http.

backend-1

##
# store configuration for backend-1/10.0.0.1
##

etcd-listen-client-urls: "https://10.0.0.1:2379"
etcd-listen-peer-urls: "https://0.0.0.0:2380"
etcd-initial-cluster: "backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380"
etcd-initial-advertise-peer-urls: "https://10.0.0.1:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: "sensu"
etcd-name: "backend-1"

##
# etcd peer ssl configuration for backend-1/10.0.0.1
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-1.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-1-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"
etcd-peer-client-cert-auth: true

backend-2

##
# store configuration for backend-2/10.0.0.2
##

etcd-listen-client-urls: "https://10.0.0.2:2379"
etcd-listen-peer-urls: "https://0.0.0.0:2380"
etcd-initial-cluster: "backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380"
etcd-initial-advertise-peer-urls: "https://10.0.0.2:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: "sensu"
etcd-name: "backend-2"

##
# etcd peer ssl configuration for backend-2/10.0.0.2
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-2.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-2-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"
etcd-peer-client-cert-auth: true

backend-3

##
# store configuration for backend-3/10.0.0.3
##

etcd-listen-client-urls: "https://10.0.0.3:2379"
etcd-listen-peer-urls: "https://0.0.0.0:2380"
etcd-initial-cluster: "backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380"
etcd-initial-advertise-peer-urls: "https://10.0.0.3:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: "sensu"
etcd-name: "backend-3"

##
# etcd peer ssl configuration for backend-3/10.0.0.3
##

etcd-peer-cert-file: "/etc/sensu/certs/backend-3.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-3-key.pem"
etcd-peer-trusted-ca-file: "/usr/local/share/ca-certificates/sensu/ca.pem"
etcd-peer-client-cert-auth: true

Sensu agent with HTTPS

Below is a sample configuration for an agent that would connect to the cluster using wss from /etc/sensu/agent.yml.

##
# backend-url configuration for all agents connecting to cluster over wss
##

backend-url:
  - "wss://10.0.0.1:8081"
  - "wss://10.0.0.2:8081"
  - "wss://10.0.0.3:8081"

Using an external etcd cluster

Using Sensu with an external etcd cluster requires etcd 3.3.2 or newer. To stand up an external etcd cluster, you can follow etcd’s clustering guide using the same store configuration.

In this example, we will enable client-to-server and peer communication authentication using self-signed TLS certificates. Below is how you would start etcd for backend-1 from our three node configuration example above.

etcd \
--listen-client-urls "https://10.0.0.1:2379" \
--advertise-client-urls "https://10.0.0.1:2379" \
--listen-peer-urls "https://10.0.0.1:2380" \
--initial-cluster "backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380" \
--initial-advertise-peer-urls "https://10.0.0.1:2380" \
--initial-cluster-state "new" \
--name "backend-1" \
--trusted-ca-file=./ca.pem \
--cert-file=./backend-1.pem \
--key-file=./backend-1-key.pem \
--client-cert-auth \
--peer-trusted-ca-file=./ca.pem \
--peer-cert-file=./backend-1.pem \
--peer-key-file=./backend-1-key.pem \
--peer-client-cert-auth \
--auto-compaction-mode revision \
--auto-compaction-retention 2

NOTE: The auto-compaction-mode and auto-compaction-retention flags are of particular significance. Without these settings your database may quickly reach etcd’s maximum database size limit.

In order to inform Sensu that you’d like to use this external etcd data source, add the sensu-backend flag --no-embed-etcd to the original configuration, along with the path to a client certificate created using our CA.

sensu-backend start \
--etcd-trusted-ca-file=./ca.pem \
--etcd-cert-file=./client.pem \
--etcd-key-file=./client-key.pem \
--etcd-advertise-client-urls=https://10.0.0.1:2379,https://10.0.0.2:2379,https://10.0.0.3:2379 \
--no-embed-etcd