Monitor containers and applications with Sensu

In this tutorial, you’ll deploy a sample app with Kubernetes and monitor it with Sensu. In the sample app, / returns the local hostname. The sample app has three endpoints: /metrics returns Prometheus metric data, /healthz returns the Boolean health state, and POST /healthz toggles the health state.

Prerequisites

The sample app requires Kubernetes and a Kubernetes ingress controller. Most hosted Kubernetes offerings (like GKE) include a Kubernetes ingress controller.

This tutorial uses Minikube, a cross-platform application for running a local single-node Kubernetes cluster. After you install and start Minikube, proceed through the rest of this tutorial.

Set up

1. Clone the sample app.

git clone https://github.com/sensu/sensu-kube-demo && cd sensu-kube-demo

2. Create the Kubernetes ingress resources.

minikube start

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

minikube addons enable ingress

kubectl create -f go/ingress-nginx/ingress/sensu-go.yaml

3. Deploy kube-state-metrics.

git clone https://github.com/kubernetes/kube-state-metrics

kubectl apply -f kube-state-metrics/kubernetes

4. Open your /etc/hosts file and add the hostnames.

NOTE: You’ll use the IP address for the Minikube VM in the hosts file. To view the address, use the command minikube ip.

192.168.99.100       sensu.local webui.sensu.local sensu-enterprise.local dashboard.sensu-enterprise.local
192.168.99.100       influxdb.local grafana.local dummy.local

5. Install sensuctl.

Follow the sensuctl installation guide instructions to install sensuctl on Linux, macOS, or Windows.

6. Deploy two instances of the sample app (dummy) behind a load balancer.

kubectl apply -f go/deploy/dummy.yaml

You can test the dummy app using the API.

# Linux/macOS
curl -i http://dummy.local
# Windows
Invoke-WebRequest -Uri http://dummy.local -Method GET

A 200 response indicates that the dummy app is working correctly.

7. Deploy the Sensu backend

kubectl create -f go/deploy/sensu-backend.yaml

Multitenancy

Use Sensu role-based access control (RBAC) to create a demo namespace and a demo user.

1. Configure sensuctl to use the admin user.

When you installed the Sensu backend, during the initialization step, you created an admin username and password for a default namespace. Use that username and password to configure sensuctl in this step.

sensuctl configure
? Sensu Backend URL: http://sensu.local
? Username: YOUR_USERNAME
? Password: YOUR_PASSWORD
? Namespace: default
? Preferred output format: tabular

2. Create a demo namespace.

sensuctl namespace create demo

Use sensuctl to confirm that the namespace was created successfully and set the demo namespace as the default for your sensuctl session:

sensuctl namespace list

sensuctl config set-namespace demo

3. Create a dev user role with full access to the demo namespace.

sensuctl role create dev \
--verb get,list,create,update,delete \
--resource \* --namespace demo

4. Create a dev role binding for the dev group.

sensuctl role-binding create dev --role dev --group dev

5. Create a demo user that is a member of the dev group.

sensuctl user create demo --interactive
? Username: demo
? Password: DEMO_PASSWORD
? Groups: dev

6. Reconfigure sensuctl to use the demo user and demo namespace.

sensuctl configure
? Sensu Backend URL: http://sensu.local
? Username: demo
? Password: password
? Namespace: demo
? Preferred output format: tabular

Deploy Sensu agents and InfluxDB

1. Deploy InfluxDB with a Sensu agent sidecar

Create a Kubernetes ConfigMap for InfluxDB configuration.

kubectl create configmap influxdb-config --from-file go/configmaps/influxdb.conf

Deploy InfluxDB with a Sensu agent sidecar.

kubectl create -f go/deploy/influxdb.sensu.yaml

2. Create a Sensu pipeline to store metrics with InfluxDB.

Use the files provided with the sample app to create a Sensu asset for the Sensu InfluxDB handler and create an influxdb event handler.

sensuctl create --file go/config/assets/influxdb-handler.yaml

sensuctl create --file go/config/handlers/influxdb.yaml

3. Deploy Sensu agent sidecars for the dummy app instances.

kubectl apply -f go/deploy/dummy.sensu.yaml

Monitor the app

Let’s take a look at what you’re monitoring. You can see the Sensu agents installed on your two dummy app instances with their last-seen timestamps, as well as the Sensu agent monitoring your InfluxDB instance.

sensuctl entity list

            ID               Class    OS                   Subscriptions                           Last Seen            
─────────────────────────── ─────── ─────── ─────────────────────────────────────────── ─────────────────────────────── 
dummy-76d8fb7bdf-967q7      agent   linux   dummy,entity:dummy-76d8fb7bdf-967q7         2019-01-18 10:56:56 -0800 PST  
dummy-76d8fb7bdf-knh7r      agent   linux   dummy,entity:dummy-76d8fb7bdf-knh7r         2019-01-18 10:56:56 -0800 PST  
influxdb-64b7d5f884-f9ptg   agent   linux   influxdb,entity:influxdb-64b7d5f884-f9ptg   2019-01-18 10:56:59 -0800 PST  

Create a Sensu pipeline to Slack

Suppose you want to receive a Slack alert if the dummy app returns an unhealthy response. You can create a Sensu pipeline to send events to Slack using the Sensu Slack handler (one of many open-source collections of Sensu building blocks shared by the Sensu community).

1. Create an asset to help agents find and install the Sensu Slack handler.

sensuctl create --file go/config/assets/slack-handler.yaml

2. Get your Slack webhook URL and add it to go/config/handlers/slack.yaml.

If you’re already an admin of a Slack, visit https://YOUR WORKSPACE NAME HERE.slack.com/services/new/incoming-webhook and follow the steps to add the Incoming WebHooks integration and save the settings. If you’re not yet a Slack admin, start here to create a new workspace. After saving, you’ll see your webhook URL under Integration Settings.

Open go/config/handlers/slack.yaml. In the following line, replace SECRET with your Slack workspace webhook URL and #demo with the Slack channel of your choice:

"command": "slack-handler --channel '#demo' --timeout 20 --username 'sensu' --webhook-url 'SECRET'",

So it looks something like:

"command": "slack-handler --channel '#my-channel' --timeout 20 --username 'sensu' --webhook-url 'https://hooks.slack.com/services/XXXXXXXXXXXXXXXX'",

3. Create a handler to send events to Slack using the slack-handler asset.

sensuctl create --file go/config/handlers/slack.yaml

Create a Sensu service check to monitor the status of the dummy app

To automatically monitor the status of the dummy app, create an asset that lets the Sensu agent use a Sensu HTTP plugin.

1. Create the check-plugins asset.

sensuctl create --file go/config/assets/check-plugins.yaml

2. Create a check to monitor the status of the dummy app that uses the check-plugins asset and the Slack pipeline.

sensuctl create --file go/config/checks/dummy-app-healthz.yaml

With the automated alert workflow in place, you can see the resulting events in the Sensu dashboard.

3. Sign in to the Sensu dashboard.

Sign in to the Sensu dashboard with your sensuctl username (demo) and password (password). Since you’re working within the demo namespace, select the demo namespace in the Sensu dashboard menu.

4. Toggle the health of the dummy app to simulate a failure.

# Linux/macOS
curl -iXPOST http://dummy.local/healthz
# Windows
Invoke-WebRequest -Uri http://dummy.local/healthz -Method POST

You should now be able to see a critical alert in the Sensu dashboard as well as by using sensuctl:

sensuctl event list

You should also see an alert in Slack.

Continue to post to /healthz until all Sensu entities return to a healthy state.

# Linux/macOS
curl -iXPOST http://dummy.local/healthz
# Windows
Invoke-WebRequest -Uri http://dummy.local/healthz -Method POST

Collect app metrics

Create a Sensu metric check to collect Prometheus metrics

To automatically collect Prometheus metrics from the dummy app, create an asset that lets the Sensu agents use the Sensu Prometheus Collector.

1. Create the prometheus-collector asset.

sensuctl create --file go/config/assets/prometheus-collector.yaml

2. Create a check to collect Prometheus metrics that uses the prometheus-collector asset.

sensuctl create --file go/config/checks/dummy-app-prometheus.yaml

Visualize metrics with Grafana

1. Deploy Grafana with a Sensu agent sidecar.

Create Kubernetes ConfigMaps for Grafana configuration:

kubectl create configmap grafana-provisioning-datasources --from-file=./go/configmaps/grafana-provisioning-datasources.yaml

kubectl create configmap grafana-provisioning-dashboards --from-file=./go/configmaps/grafana-provisioning-dashboards.yaml

Deploy Grafana with a Sensu agent sidecar:

kubectl apply -f go/deploy/grafana.sensu.yaml

After a few minutes, you can see the Sensu agents you installed on the dummy app, InfluxDB, and Grafana pods.

sensuctl entity list
            ID               Class    OS                   Subscriptions                           Last Seen            
─────────────────────────── ─────── ─────── ─────────────────────────────────────────── ─────────────────────────────── 
dummy-6c57b8f868-ft5dz      agent   linux   dummy,entity:dummy-6c57b8f868-ft5dz         2018-11-20 18:43:15 -0800 PST  
dummy-6c57b8f868-m24hw      agent   linux   dummy,entity:dummy-6c57b8f868-m24hw         2018-11-20 18:43:15 -0800 PST  
grafana-5b88f8df8d-vgjtm    agent   linux   grafana,entity:grafana-5b88f8df8d-vgjtm     2018-11-20 18:43:14 -0800 PST  
influxdb-78d64bcfd9-8km56   agent   linux   influxdb,entity:influxdb-78d64bcfd9-8km56   2018-11-20 18:43:12 -0800 PST  

2. Log in to Grafana.

To see the metrics you’re collecting from the dummy app, log into Grafana with the username admin and password password.

3. Create a dashboard.

Create a new dashboard using the InfluxDB datasource to see live metrics from the dummy app.

Collect Kubernetes metrics

Now that you have a pipeline set up to send metrics, you can create a check that collects Prometheus metrics from Kubernetes and connect it to the pipeline.

Deploy a Sensu agent as a DameonSet on your Kubernetes node:

kubectl apply -f go/deploy/sensu-agent-daemonset.yaml

Create a check to collect Prometheus metrics from Kubernetes using the prometheus-collector asset and influxdb handler:

sensuctl create --file go/config/checks/kube-state-prometheus.yaml

You should now be able to access Kubernetes metrics data in Grafana and see metric events in the Sensu dashboard.

Next steps

To stop or delete the sample app, use minikube stop or minikube delete, respectively.

For more information about monitoring with Sensu, check out these resources: