Metrics reference

Sensu Go offers built-in support for collecting and processing service and time-series metrics for your entire infrastructure.

In Sensu, metrics are an optional component of observation data in events. Sensu events may contain check execution results, metrics, or both. Certain inputs like the Sensu StatsD listener or patterns like the Prometheus collector pattern will create metrics-only events. Events can also include metrics from check output metric extraction.

Use Sensu handlers to process extracted metrics and route them to databases like Elasticsearch, InfluxDB, Grafana, and Graphite. You can also use Sensu’s time-series and long-term event storage integrations to process service and time-series metrics.

NOTE: This reference describes the metrics component of observation data included in Sensu events, which is distinct from the Sensu metrics API. For information about HTTP GET access to internal Sensu metrics, read our metrics API documentation.

Metric check example

This check definition collects metrics in InfluxDB Line Protocol format and sends the collected metrics to a metrics handler configured with the Sensu InfluxDB Handler dynamic runtime asset:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  name: collect-metrics
  namespace: default
spec:
  check_hooks: null
  command: metrics-disk-usage.rb
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 30
  low_flap_threshold: 0
  output_metric_format: influxdb_line
  output_metric_handlers:
  - sensu-influxdb-handler
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - sensu-plugins/sensu-plugins-disk-checks
  - sensu/sensu-ruby-runtime
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - linux
  timeout: 0
  ttl: 0
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "created_by": "admin",
    "name": "collect-metrics",
    "namespace": "default"
  },
  "spec": {
    "check_hooks": null,
    "command": "metrics-disk-usage.rb",
    "env_vars": null,
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 30,
    "low_flap_threshold": 0,
    "output_metric_format": "influxdb_line",
    "output_metric_handlers": [
      "sensu-influxdb-handler"
    ],
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "sensu-plugins/sensu-plugins-disk-checks",
      "sensu/sensu-ruby-runtime"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "linux"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

Metric event example

The example metric check will produce events similar to this metric event:

---
type: Event
api_version: core/v2
metadata:
  namespace: default
spec:
  check:
    check_hooks: null
    command: metrics-disk-usage.rb
    duration: 0.059301963
    env_vars: null
    executed: 1614379313
    handlers: []
    high_flap_threshold: 0
    history:
    - executed: 1614379193
      status: 0
    - executed: 1614379223
      status: 0
    - executed: 1614379253
      status: 0
    - executed: 1614379283
      status: 0
    - executed: 1614379313
      status: 0
    interval: 30
    is_silenced: false
    issued: 1614379313
    last_ok: 1614379313
    low_flap_threshold: 0
    metadata:
      name: collect-metrics
      namespace: default
    occurrences: 5
    occurrences_watermark: 5
    output: |
      sensu-centos.disk_usage.root.used 1476 1614379313
      sensu-centos.disk_usage.root.avail 40472 1614379313
      sensu-centos.disk_usage.root.used_percentage 4 1614379313
      sensu-centos.disk_usage.root.dev.used 0 1614379313
      sensu-centos.disk_usage.root.dev.avail 485 1614379313
      sensu-centos.disk_usage.root.dev.used_percentage 0 1614379313
      sensu-centos.disk_usage.root.run.used 20 1614379313
      sensu-centos.disk_usage.root.run.avail 477 1614379313
      sensu-centos.disk_usage.root.run.used_percentage 4 1614379313
      sensu-centos.disk_usage.root.home.used 33 1614379313
      sensu-centos.disk_usage.root.home.avail 20446 1614379313
      sensu-centos.disk_usage.root.home.used_percentage 1 1614379313
      sensu-centos.disk_usage.root.boot.used 130 1614379313
      sensu-centos.disk_usage.root.boot.avail 885 1614379313
      sensu-centos.disk_usage.root.boot.used_percentage 13 1614379313
      sensu-centos.disk_usage.root.vagrant.used 82069 1614379313
      sensu-centos.disk_usage.root.vagrant.avail 871836 1614379313
      sensu-centos.disk_usage.root.vagrant.used_percentage 9 1614379313
    output_metric_format: influxdb_line
    output_metric_handlers: null
    proxy_entity_name: ""
    publish: true
    round_robin: false
    runtime_assets:
    - sensu-plugins/sensu-plugins-disk-checks
    - sensu/sensu-ruby-runtime
    scheduler: memory
    secrets: null
    state: passing
    status: 0
    stdin: false
    subdue: null
    subscriptions:
    - linux
    timeout: 0
    total_state_change: 0
    ttl: 0
  entity:
    deregister: false
    deregistration: {}
    entity_class: agent
    last_seen: 1614379313
    metadata:
      created_by: admin
      name: sensu-centos
      namespace: default
    redact:
    - password
    - passwd
    - pass
    - api_key
    - api_token
    - access_key
    - secret_key
    - private_key
    - secret
    sensu_agent_version: 6.2.5
    subscriptions:
    - entity:sensu-centos
    - linux
    system:
      arch: amd64
      cloud_provider: ""
      hostname: sensu-centos
      libc_type: glibc
      network:
        interfaces:
        - addresses:
          - 127.0.0.1/8
          - ::1/128
          name: lo
        - addresses:
          - 10.0.2.15/24
          - fe80::146d:22df:fb9a:1c7c/64
          mac: 08:00:27:8b:c9:3f
          name: eth0
        - addresses:
          - 172.28.128.33/24
          - fe80::a00:27ff:fee1:857a/64
          mac: 08:00:27:e1:85:7a
          name: eth1
      os: linux
      platform: centos
      platform_family: rhel
      platform_version: 7.5.1804
      processes: null
      vm_role: guest
      vm_system: vbox
    user: agent
  id: 38de280d-444b-4017-a438-b35cd4a6f2f8
  sequence: 5
  timestamp: 1614379313
{
  "type": "Event",
  "api_version": "core/v2",
  "metadata": {
    "namespace": "default"
  },
  "spec": {
    "check": {
      "check_hooks": null,
      "command": "metrics-disk-usage.rb",
      "duration": 0.059301963,
      "env_vars": null,
      "executed": 1614379313,
      "handlers": [],
      "high_flap_threshold": 0,
      "history": [
        {
          "executed": 1614379193,
          "status": 0
        },
        {
          "executed": 1614379223,
          "status": 0
        },
        {
          "executed": 1614379253,
          "status": 0
        },
        {
          "executed": 1614379283,
          "status": 0
        },
        {
          "executed": 1614379313,
          "status": 0
        }
      ],
      "interval": 30,
      "is_silenced": false,
      "issued": 1614379313,
      "last_ok": 1614379313,
      "low_flap_threshold": 0,
      "metadata": {
        "name": "collect-metrics",
        "namespace": "default"
      },
      "occurrences": 5,
      "occurrences_watermark": 5,
      "output": "sensu-centos.disk_usage.root.used 1476 1614379313\nsensu-centos.disk_usage.root.avail 40472 1614379313\nsensu-centos.disk_usage.root.used_percentage 4 1614379313\nsensu-centos.disk_usage.root.dev.used 0 1614379313\nsensu-centos.disk_usage.root.dev.avail 485 1614379313\nsensu-centos.disk_usage.root.dev.used_percentage 0 1614379313\nsensu-centos.disk_usage.root.run.used 20 1614379313\nsensu-centos.disk_usage.root.run.avail 477 1614379313\nsensu-centos.disk_usage.root.run.used_percentage 4 1614379313\nsensu-centos.disk_usage.root.home.used 33 1614379313\nsensu-centos.disk_usage.root.home.avail 20446 1614379313\nsensu-centos.disk_usage.root.home.used_percentage 1 1614379313\nsensu-centos.disk_usage.root.boot.used 130 1614379313\nsensu-centos.disk_usage.root.boot.avail 885 1614379313\nsensu-centos.disk_usage.root.boot.used_percentage 13 1614379313\nsensu-centos.disk_usage.root.vagrant.used 82069 1614379313\nsensu-centos.disk_usage.root.vagrant.avail 871836 1614379313\nsensu-centos.disk_usage.root.vagrant.used_percentage 9 1614379313\n",
      "output_metric_format": "influxdb_line",
      "output_metric_handlers": null,
      "proxy_entity_name": "",
      "publish": true,
      "round_robin": false,
      "runtime_assets": [
        "sensu-plugins/sensu-plugins-disk-checks",
        "sensu/sensu-ruby-runtime"
      ],
      "scheduler": "memory",
      "secrets": null,
      "state": "passing",
      "status": 0,
      "stdin": false,
      "subdue": null,
      "subscriptions": [
        "linux"
      ],
      "timeout": 0,
      "total_state_change": 0,
      "ttl": 0
    },
    "entity": {
      "deregister": false,
      "deregistration": {
      },
      "entity_class": "agent",
      "last_seen": 1614379313,
      "metadata": {
        "created_by": "admin",
        "name": "sensu-centos",
        "namespace": "default"
      },
      "redact": [
        "password",
        "passwd",
        "pass",
        "api_key",
        "api_token",
        "access_key",
        "secret_key",
        "private_key",
        "secret"
      ],
      "sensu_agent_version": "6.2.5",
      "subscriptions": [
        "entity:sensu-centos",
        "linux"
      ],
      "system": {
        "arch": "amd64",
        "cloud_provider": "",
        "hostname": "sensu-centos",
        "libc_type": "glibc",
        "network": {
          "interfaces": [
            {
              "addresses": [
                "127.0.0.1/8",
                ":1/128"
              ],
              "name": "lo"
            },
            {
              "addresses": [
                "10.0.2.15/24",
                "fe80::146d:22df:fb9a:1c7c/64"
              ],
              "mac": "08:00:27:8b:c9:3f",
              "name": "eth0"
            },
            {
              "addresses": [
                "172.28.128.33/24",
                "fe80::a00:27ff:fee1:857a/64"
              ],
              "mac": "08:00:27:e1:85:7a",
              "name": "eth1"
            }
          ]
        },
        "os": "linux",
        "platform": "centos",
        "platform_family": "rhel",
        "platform_version": "7.5.1804",
        "processes": null,
        "vm_role": "guest",
        "vm_system": "vbox"
      },
      "user": "agent"
    },
    "id": "38de280d-444b-4017-a438-b35cd4a6f2f8",
    "sequence": 5,
    "timestamp": 1614379313
  }
}

Extract metrics from check output

The Sensu agent can extract metrics data from check command output and populate an event’s metrics attribute before sending the event to the Sensu backend for processing.

To extract metrics from check output:

When a check includes correctly configured command and output_metric_format attributes, Sensu will extract the specified metrics from the check output and add them to the event data in the metrics attribute.

You can also configure the check output_metric_handlers attribute to use a Sensu handler that is equipped to handle Sensu metrics. See the check reference or InfluxDB handler guide to learn more.

Supported output metric formats

Sensu supports the following formats for check output metric extraction.

Graphite
output_metric_format graphite_plaintext
documentation Graphite Plaintext Protocol
example 
local.random.diceroll 4 123456789
InfluxDB
output_metric_format influxdb_line
documentation InfluxDB Line Protocol
example 
weather,location=us-midwest temperature=82 1465839830100400200
Nagios
output_metric_format nagios_perfdata
documentation Nagios Performance Data
example 
PING ok - Packet loss = 0%, RTA = 0.80 ms | percent_packet_loss=0, rta=0.80
OpenTSDB
output_metric_format opentsdb_line
documentation OpenTSDB Data Specification
example 
sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0
Prometheus
output_metric_format prometheus_text
documentation Prometheus Exposition Text
example 
http_requests_total{method="post",code="200"} 1027 1395066363000

Enrich metrics with tags

Output metric tags are custom tags you can apply to enrich the metric points produced by check output metric extraction.

Use output metric tags for the output metric formats that do not natively support tags: Graphite Plaintext Protocol and Nagios Performance Data.

Values for output metric tags are passed through to the metric points produced by check output metric extraction for formats that natively support tags (InfluxDB Line Protocol, OpenTSDB Data Specification, and Prometheus Exposition Text).

You can use check token substitution for the value attribute to include any event attribute in an output metric tag. For example, this tag will list the event.time attribute:

"output_metric_tags": [
  {
    "name": "instance",
    "value": "{{ .entity.system.hostname }}"
  }
]

Process extracted and tagged metrics

Specify the event handlers you want to process your Sensu metrics in the check output_metric_handlers attribute. With these event handlers, you can route metrics to one or more databases for storing and visualizing metrics, like Elasticsearch, InfluxDB, Grafana, and Graphite.

Many of our most popular metrics integrations for time-series and long-term event storage include curated, configurable quick-start templates to integrate Sensu with your existing workflows. You can also use Bonsai, the Sensu asset hub, to discover, download, and share dynamic runtime assets for processing metrics.

In check definitions, the output_metric_handlers list for metrics event handlers is distinct and separate from the handlers list for status event handlers. Having separate handlers attributes allows you to use different workflows for metrics and status without applying conditional filter logic. The events reference includes an example event with check and metric data.

You do not need to add a mutator to your check definition to process metrics with an event handler. The metrics attribute format automatically reduces metrics data complexity so event handlers can process metrics effectively.

Validate metrics

If the check output is formatted correctly according to its output_metric_format, the metrics will be extracted in Sensu metric format and passed to the observability pipeline. To confirm that your check is extracting metrics, inspect the events the check yields:

NOTE: Replace entity_name and check_name with the names of the entity and check whose events you want to review.

sensuctl event info entity_name check_name --format json

You should expect to see errors logged by sensu-agent if it is unable to parse the check output. See Troubleshoot Sensu for an example debug handler that writes events to a file for inspection.

Metrics specification

The check specification describes metrics attributes in checks.

The event specification describes metrics attributes in events.