Pipelines reference

IMPORTANT: The pipeline resources described on this page are different from the handler resources you can create with the pipeline API.

The pipeline API does not create pipeline resources. Instead, it allows you to create handlers that can only be used in pipeline resources. Read the Sumo Logic metrics handlers reference and TCP stream handlers reference for more information.

Sensu executes pipelines during the process stage of the observability pipeline.

Pipelines are Sensu resources composed of observation event processing workflows made up of filters, mutators, and handlers. Instead of specifying filters and mutators in handler definitions, you can specify all three in a single pipeline workflow.

To use a pipeline, list it in a check definition’s pipelines array. All the observability events that the check produces will be processed according to the pipeline’s workflows.

Pipelines can replace handler sets and handler stacks. We recommend migrating your existing handler sets and stacks to pipeline workflows.

NOTE: To use pipelines, upgrade your agents to Sensu Go 6.5.0.

Pipeline example

This example shows a pipeline resource definition that includes event filters, a mutator, and a handler:

---
type: Pipeline
api_version: core/v2
metadata:
  name: incident_alerts
  namespace: default
  created_by: admin
spec:
  workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "incident_alerts",
    "namespace": "default",
    "created_by": "admin"
  },
  "spec": {
    "workflows": [
      {
        "name": "labeled_email_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "mutator": {
          "name": "add_labels",
          "type": "Mutator",
          "api_version": "core/v2"
        },
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}

To use this pipeline in a check, list it in the check’s pipelines array. For example:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: incident_pipelines
  namespace: default
spec:
  command: collect.sh
  interval: 10
  publish: true
  subscriptions:
  - system
  pipelines:
  - type: Pipeline
    api_version: core/v2
    name: incident_alerts
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "incident_pipelines",
    "namespace": "default"
  },
  "spec": {
    "command": "collect.sh",
    "interval": 10,
    "publish": true,
    "subscriptions": [
      "system"
    ],
    "pipelines": [
      {
        "type": "Pipeline",
        "api_version": "core/v2",
        "name": "incident_alerts"
      }
    ]
  }
}

Workflows

The workflow attribute is an array of event processing workflows that Sensu will apply for events produced by any check that references the pipeline.

Workflows do not have to include an event filter or mutator, but they must specify at least one handler.

Workflows can include more than one event filter. If a workflow has more than one filter, Sensu applies the filters in a series, starting with the filter that is listed first.

Pipelines with multiple workflows

Pipelines can include more than one workflow.

In this example, the pipeline includes labeled_email_alerts and slack_alerts workflows:

---
type: Pipeline
api_version: core/v2
metadata:
  name: incident_alerts
  namespace: default
  created_by: admin
spec:
  workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
  - name: slack_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    handler:
      name: slack
      type: Handler
      api_version: core/v2
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "incident_alerts",
    "namespace": "default",
    "created_by": "admin"
  },
  "spec": {
    "workflows": [
      {
        "name": "labeled_email_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "mutator": {
          "name": "add_labels",
          "type": "Mutator",
          "api_version": "core/v2"
        },
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      },
      {
        "name": "slack_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "slack",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}

All events from checks that specify this pipeline will be processed with both workflows, in series, starting with the workflow that is listed first in the resource definition.

Pipeline specification

Top-level attributes

type
description Top-level attribute that specifies the sensuctl create resource type. Pipelines should always be type Pipeline.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example
type: Pipeline
{
  "type": "Pipeline"
}
api_version
description Top-level attribute that specifies the Sensu API group and version. For pipelines in this version of Sensu, the api_version should always be core/v2.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example
api_version: core/v2
{
  "api_version": "core/v2"
}
metadata
description Top-level collection of metadata about the pipeline that includes name, namespace, and created_by as well as custom labels and annotations. The metadata map is always at the top level of the pipeline definition. This means that in wrapped-json and yaml formats, the metadata scope occurs outside the spec scope. Read metadata attributes for details.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example
metadata:
  name: incident_alerts
  namespace: default
  created_by: admin
  labels:
    region: us-west-1
  annotations:
    slack-channel: "#incidents"
{
  "metadata": {
    "name": "incident_alerts",
    "namespace": "default",
    "created_by": "admin",
    "labels": {
      "region": "us-west-1"
    },
    "annotations": {
      "slack-channel": "#incidents"
    }
  }
}
spec
description Top-level map that includes the pipeline spec attributes.
required Required for pipeline definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example
spec:
  workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
{
  "spec": {
    "workflows": [
      {
        "name": "labeled_email_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          },
          {
            "name": "state_change_only",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "mutator": {
          "name": "add_labels",
          "type": "Mutator",
          "api_version": "core/v2"
        },
        "handler": {
          "name": "email",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}

Metadata attributes

name
description Unique string used to identify the pipeline. Pipeline names cannot contain special characters or spaces (validated with Go regex \A[\w\.\-]+\z). Each pipeline must have a unique name within its namespace.
required true
type String
example
name: incident_alerts
{
  "name": "incident_alerts"
}
namespace
description Sensu RBAC namespace that the pipeline belongs to.
required false
type String
default default
example
namespace: default
{
  "namespace": "default"
}
created_by
description Username of the Sensu user who created the pipeline or last updated the handler. Sensu automatically populates the created_by field when the pipeline is created or updated.
required false
type String
example
created_by: admin
{
  "created_by": "admin"
}
labels
description Custom attributes to include with observation data in events that you can use for response and web UI view filtering.

If you include labels in your event data, you can filter API responses, sensuctl responses, and web UI views based on them. In other words, labels allow you to create meaningful groupings for your data.

Limit labels to metadata you need to use for response filtering. For complex, non-identifying metadata that you will not need to use in response filtering, use annotations rather than labels.
required false
type Map of key-value pairs. Keys can contain only letters, numbers, and underscores and must start with a letter. Values can be any valid UTF-8 string.
default null
example
labels:
  environment: production
  region: us-west-1
{
  "labels": {
    "environment": "production",
    "region": "us-west-1"
  }
}
annotations
description Non-identifying metadata to include with observation data in events that you can access with event filters. You can use annotations to add data that’s meaningful to people or external tools that interact with Sensu.

In contrast to labels, you cannot use annotations in API response filtering, sensuctl response filtering, or web UI views.
required false
type Map of key-value pairs. Keys and values can be any valid UTF-8 string.
default null
example
annotations:
  managed-by: ops
  slack-channel: "#incidents"
{
  "annotations": {
    "managed-by": "ops",
    "slack-channel": "#incidents"
  }
}

Spec attributes

workflows
description Array of workflows (by names) to use when filtering, mutating, and handling observability events with a pipeline. Each array item must be a string. Read workflows attributes for details.
required false
type Array
example
workflows:
  - name: labeled_email_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    - name: state_change_only
      type: EventFilter
      api_version: core/v2
    mutator:
      name: add_labels
      type: Mutator
      api_version: core/v2
    handler:
      name: email
      type: Handler
      api_version: core/v2
{
  "workflows": [
    {
      "name": "labeled_email_alerts",
      "filters": [
        {
          "name": "is_incident",
          "type": "EventFilter",
          "api_version": "core/v2"
        },
        {
          "name": "state_change_only",
          "type": "EventFilter",
          "api_version": "core/v2"
        }
      ],
      "mutator": {
        "name": "add_labels",
        "type": "Mutator",
        "api_version": "core/v2"
      },
      "handler": {
        "name": "email",
        "type": "Handler",
        "api_version": "core/v2"
      }
    }
  ]
}

Workflows attributes

filters
description Reference for the Sensu event filters to use when filtering events for the pipeline. Each pipeline workflow can reference more than one event filter. If a workflow has more than one filter, Sensu applies the filters in a series, starting with the filter that is listed first. Read filters attributes for details.
required false
type Map of key-value pairs
default null
example
filters:
- name: is_incident
  type: EventFilter
  api_version: core/v2
- name: state_change_only
  type: EventFilter
  api_version: core/v2
{
  "filters": [
    {
      "name": "is_incident",
      "type": "EventFilter",
      "api_version": "core/v2"
    },
    {
      "name": "state_change_only",
      "type": "EventFilter",
      "api_version": "core/v2"
    }
  ]
}
mutator
description Reference for the Sensu mutator to use to mutate event data for the workflow. Each pipeline workflow can reference only one mutator. Read mutator attributes for details.
required false
type Map of key-value pairs
default null
example
mutator:
  name: add_labels
  type: Mutator
  api_version: core/v2
{
  "mutator": {
    "name": "add_labels",
    "type": "Mutator",
    "api_version": "core/v2"
  }
}

handler
description Reference for the Sensu handler to use for event processing in the workflow. Each pipeline workflow must reference one handler. Read handler attributes for details.
required true
type Map of key-value pairs
example
handler:
  name: email
  type: Handler
  api_version: core/v2
{
  "handler": {
    "name": "email",
    "type": "Handler",
    "api_version": "core/v2"
  }
}

Filters attributes

name
description Name of the Sensu event filter to use for the workflow.
required true
type String
default null
example
name: is_incident
{
  "name": "is_incident"
}
type
description The sensuctl create resource type for the event filter. Event filters should always be type EventFilter.
required true
type String
default null
example
type: EventFilter
{
 "type": "EventFilter"
}
api_version
description The Sensu API group and version for the event filter. For event filters in this version of Sensu, the api_version should always be core/v2.
required true
type String
default null
example
api_version: core/v2
{
  "api_version": "core/v2"
}

Mutator attributes

name
description Name of the Sensu mutator to use for the workflow.
required true
type String
default null
example
name: add_labels
{
  "name": "add_labels"
}
type
description The sensuctl create resource type for the mutator. Mutators should always be type Mutator.
required true
type String
default null
example
type: Mutator
{
 "type": "Mutator"
}
api_version
description The Sensu API group and version for the mutator. For mutators in this version of Sensu, the api_version should always be core/v2.
required true
type String
default null
example
api_version: core/v2
{
  "api_version": "core/v2"
}

Handler attributes

name
description Name of the Sensu handler to use for the workflow.
required true
type String
default null
example
name: email
{
  "name": "email"
}
type
description The sensuctl create resource type for the handler.
required true
type String
allowed values Handler for a pipe handler, TCP or UDP handler, or handler set

TCPStreamHandler for a TCP stream handler

SumoLogicMetricsHandler for a Sumo Logic metrics handler
default null
example
type: Handler
{
 "type": "Handler"
}
api_version
description The Sensu API group and version for the handler.
required true
type String
allowed values core/v2 for a pipe handler, TCP or UDP handler, or handler set

pipeline/v1 for a TCP stream handler or Sumo Logic metrics handler
default null
example
api_version: core/v2
{
  "api_version": "core/v2"
}