How to monitor external resources with proxy requests and entities

Proxy entities allow Sensu to monitor external resources on systems or devices where a Sensu agent cannot be installed, like a network switch or a website. You can create proxy entities using sensuctl, the Sensu API, or the proxy_entity_name check attribute. When executing checks that include a proxy_entity_name or proxy_requests attributes, Sensu agents report the resulting event under the proxy entity instead of the agent entity.

This guide requires a running Sensu backend, a running Sensu agent, and a sensuctl instance configured to connect to the backend as a user with get, list, and create permissions for entities, checks, and events.

Using a proxy entity to monitor a website

In this section, we’ll monitor the status of sensu.io by configuring a check with a proxy entity name so that Sensu creates an entity representing the site and reports the status of the site under this entity.

Installing an HTTP check script

First, we’ll install a bash script, named http_check.sh, to perform an HTTP check using curl.

sudo curl https://raw.githubusercontent.com/sensu/sensu-go/5.1.0/examples/checks/http_check.sh \
-o /usr/local/bin/http_check.sh && \
sudo chmod +x /usr/local/bin/http_check.sh

PRO TIP: While this command may be appropriate when running a few agents, you should consider using Sensu assets or a configuration management tool to provide runtime dependencies.

Creating the check

Now that the HTTP check script is installed, we’ll create a check named check-sensu-site, which runs the command http_check.sh https://sensu.io, at an interval of 60 seconds, for all agents subscribed to the proxy subscription, using the sensu-site proxy entity name. To avoid duplicate events, we’ll add the round_robin attribute to distribute the check execution across all agents subscribed to the proxy subscription.

Create a file called check.json and add the following check definition.

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check-sensu-site",
    "namespace": "default"
  },
  "spec": {
    "command": "http_check.sh https://sensu.io",
    "interval": 60,
    "proxy_entity_name": "sensu-site",
    "publish": true,
    "round_robin": true,
    "subscriptions": [
      "proxy"
    ]
  }
}

Now we can use sensuctl to add this check to Sensu.

sensuctl create --file check.json

sensuctl check list
       Name                    Command               Interval   Cron   Timeout   TTL   Subscriptions   Handlers   Assets   Hooks   Publish?   Stdin?   Metric Format   Metric Handlers  
────────────────── ──────────────────────────────── ────────── ────── ───────── ───── ─────────────── ────────── ──────── ─────── ────────── ──────── ─────────────── ───────────────── 
 check-sensu-site   http_check.sh https://sensu.io         60                0     0   proxy                                       true       false                                     

Adding the subscription

To run the check, we’ll need a Sensu agent with the subscription proxy. After installing an agent, open /etc/sensu/agent.yml and add the proxy subscription so the subscription configuration looks like:

subscriptions:
  - proxy

Then restart the agent.

sudo service sensu-agent restart

Validating the check

Now we can use sensuctl to see that Sensu has created the proxy entity sensu-site.

sensuctl entity list
      ID        Class    OS           Subscriptions                   Last Seen            
────────────── ─────── ─────── ─────────────────────────── ─────────────────────────────── 
sensu-centos   agent   linux   proxy,entity:sensu-centos   2019-01-16 21:50:03 +0000 UTC  
sensu-site     proxy           entity:sensu-site           N/A  

NOTE: It might take a few moments for Sensu to execute the check and create the proxy entity.

And that Sensu is now monitoring sensu-site using the check-sensu-site check.

sensuctl event info sensu-site check-sensu-site
=== sensu-site - check-sensu-site
Entity:    sensu-site
Check:     check-sensu-site
Output:    
Status:    0
History:   0,0
Silenced:  false
Timestamp: 2019-01-16 21:51:53 +0000 UTC

We can also see our new proxy entity in the Sensu dashboard.

Using proxy requests to monitor a group of websites

Now let’s say that, instead of monitoring just sensu.io, we want to monitor multiple sites, for example: docs.sensu.io, packagecloud.io, and github.com. In this section of the guide, we’ll use the proxy_requests check attribute, along with entity labels and token substitution, to monitor three sites using the same check. Before we get started, go ahead and install the HTTP check script if you haven’t already.

Creating proxy entities

Instead of creating a proxy entity using the proxy_entity_name check attribute, we’ll be using sensuctl to create proxy entities to represent the three sites we want to monitor. Our proxy entities need the entity_class attribute set to proxy to mark them as proxy entities as well as a few custom labels that we’ll use to identify them as a group and pass in individual URLs.

Create a file called entities.json and add the following entity definitions.

{
  "type": "Entity",
  "api_version": "core/v2",
  "metadata": {
    "name": "sensu-docs",
    "namespace": "default",
    "labels": {
      "proxy_type": "website",
      "url": "https://docs.sensu.io"
    }
  },
  "spec": {
    "entity_class": "proxy"
  }
}
{
  "type": "Entity",
  "api_version": "core/v2",
  "metadata": {
    "name": "packagecloud-site",
    "namespace": "default",
    "labels": {
      "proxy_type": "website",
      "url": "https://packagecloud.io"
    }
  },
  "spec": {
    "entity_class": "proxy"
  }
}
{
  "type": "Entity",
  "api_version": "core/v2",
  "metadata": {
    "name": "github-site",
    "namespace": "default",
    "labels": {
      "proxy_type": "website",
      "url": "https://github.com"
    }
  },
  "spec": {
    "entity_class": "proxy"
  }
}

PRO TIP: When creating proxy entities, you can add whatever custom labels make sense for your environment. For example, when monitoring a group of routers, you may want to add ip_address labels.

Now we can use sensuctl to add these proxy entities to Sensu.

sensuctl create --file entities.json

sensuctl entity list
        ID           Class    OS           Subscriptions                   Last Seen            
─────────────────── ─────── ─────── ─────────────────────────── ─────────────────────────────── 
 github-site         proxy                                       N/A                            
 packagecloud-site   proxy                                       N/A                            
 sensu-centos        agent   linux   proxy,entity:sensu-centos   2019-01-16 23:05:03 +0000 UTC  
 sensu-docs          proxy                                       N/A                            

Creating a reusable HTTP check

Now that we have our three proxy entities set up, each with a proxy_type and url label, we can use proxy requests and token substitution to create a single check that monitors all three sites.

Create a file called check-proxy-requests.json and add the following check definition.

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check-http",
    "namespace": "default"
  },
  "spec": {
    "command": "http_check.sh {{ .labels.url }}",
    "interval": 60,
    "subscriptions": [
      "proxy"
    ],
    "publish": true,
    "round_robin": true,
    "proxy_requests": {
      "entity_attributes": [
        "entity.entity_class == 'proxy'",
        "entity.labels.proxy_type == 'website'"
      ]
    }
  }
}

Our check-http check uses the proxy_requests attribute to specify the applicable entities. In our case, we want to run the check-http check on all entities of entity class proxy and proxy type website. Since we’re using this check to monitor multiple sites, we can use token substitution to apply the correct url in the check command, and we can use the round_robin attribute to distribute the executions evenly across agents with the proxy subscription.

Now we can use sensuctl to add this check to Sensu.

sensuctl create --file check-proxy-requests.json

sensuctl check list
       Name                    Command               Interval   Cron   Timeout   TTL   Subscriptions   Handlers   Assets   Hooks   Publish?   Stdin?   Metric Format   Metric Handlers  
───────────────── ───────────────────────────────── ────────── ────── ───────── ───── ─────────────── ────────── ──────── ─────── ────────── ──────── ─────────────── ───────────────── 
  check-http        http_check.sh {{ .labels.url }}         60                0     0   proxy                                       true       false                                     

Validating the check

Before validating the check, make sure that you’ve installed the HTTP check script and added the proxy subscription to a Sensu agent if you haven’t already.

Now we can use sensuctl to see that Sensu is monitoring docs.sensu.io, packagecloud.io, and github.com using the check-http, returning a status of 0 (OK).

sensuctl event list
      Entity                Check          Output   Status   Silenced             Timestamp            
─────────────────── ───────────────────── ──────── ──────── ────────── ─────────────────────────────── 
github-site         check-http                           0   false      2019-01-17 17:10:31 +0000 UTC  
packagecloud-site   check-http                           0   false      2019-01-17 17:10:34 +0000 UTC  
sensu-centos        keepalive               ...          0   false      2019-01-17 17:10:34 +0000 UTC  
sensu-docs          check-http                           0   false      2019-01-17 17:06:59 +0000 UTC  

Next steps

You now know how to run a proxy check to verify the status of a website, as well as using proxy requests to run a check on two different proxy entities based on label evaluation. From this point, here are some recommended resources: