Use secrets management in Sensu
- Retrieve your PagerDuty Integration Key
- Use Env for secrets management
- Use HashiCorp Vault for secrets management
- Add a handler
- Next steps
COMMERCIAL FEATURE: Access the Env and VaultProvider secrets provider datatypes in the packaged Sensu Go distribution. For more information, see Get started with commercial features.
Sensu’s secrets management allows you to avoid exposing secrets like usernames, passwords, and access keys in your Sensu configuration.
In this guide, you’ll learn how to use Sensu’s built-in secrets provider,
Env, or HashiCorp Vault as your external secrets provider and authenticate without exposing your secrets.
You’ll set up your PagerDuty Integration Key as a secret and create a PagerDuty handler definition that requires the secret.
Your Sensu backend can then execute the handler with any check.
Secrets are configured via secrets resources.
A secret resource definition refers to the secrets provider (
VaultProvider) and an ID (the named secret to fetch from the secrets provider).
This guide only covers the handler use case, but you can use secrets management in handler, mutator, and check execution. For secrets management in checks, the Sensu backend will transmit requests over its secure transport (TLS-encrypted websockets) to your Sensu agent to execute your check, so mTLS must be enabled.
The secret included in your Sensu handler will be exposed to Sensu services at runtime as an environment variable. Sensu only exposes secrets to Sensu services like environment variables and automatically redacts secrets from all logs, the API, and the dashboard.
Retrieve your PagerDuty Integration Key
The example in this guide uses the PagerDuty Integration Key as a secret and a PagerDuty handler definition that requires the secret.
Here’s how to find your Integration Key in PagerDuty so you can set it up as your secret:
- Log in to your PagerDuty account.
- In the Configuration drop-down menu, select Services.
- Click your Sensu service.
- Click the Integrations tab. The Integration Key is listed in the second column.
PagerDuty Integration Key location
Use Env for secrets management
The Sensu Go commercial distribution includes a built-in secrets provider,
Env, that exposes secrets from environment variables on your Sensu backend nodes.
Env secrets provider is automatically created with an empty
spec when you start your Sensu backend.
Create your backend environment variable
To use the built-in
Env secrets provider, you will add your secret as a backend environment variable.
First, make sure you have created the files you need to store backend environment variables.
Then, run the following code, replacing
INTEGRATION_KEY with your PagerDuty Integration Key:
$ echo 'SENSU_PAGERDUTY_KEY=INTEGRATION_KEY' | sudo tee -a /etc/default/sensu-backend $ sudo systemctl restart sensu-backend
$ echo 'SENSU_PAGERDUTY_KEY=INTEGRATION_KEY' | sudo tee -a /etc/sysconfig/sensu-backend $ sudo systemctl restart sensu-backend
This configures the
SENSU_PAGERDUTY_KEY environment variable to your PagerDuty Integration Key in the context of the sensu-backend process.
Create your Env secret
Now you’ll use
sensuctl create to create your secret.
This code creates a secret named
pagerduty_key that refers to the environment variable ID
cat << EOF | sensuctl create --- type: Secret api_version: secrets/v1 metadata: name: pagerduty_key namespace: default spec: id: SENSU_PAGERDUTY_KEY provider: env EOF
You can securely pass your PagerDuty Integration Key in Sensu checks, handlers, and mutators by referring to the
Skip to the add a handler section, where you’ll use your
pagerduty_key secret in your handler definition.
Use HashiCorp Vault for secrets management
NOTE: You will need to set up HashiCorp Vault to use
VaultProvider secrets management in production. The examples in this guide use the Vault dev server, which is useful for learning and experimenting. The Vault dev server gives you access to a preconfigured, running Vault server with in-memory storage that you can use right away. Follow the HashiCorp Learn curriculum when you are ready to set up a production server in Vault.
Configure your Vault authentication method (token or TLS)
Vault token authentication
Retrieve your Vault root token
NOTE: The examples in this guide use the
Root Token for the the Vault dev server, which gives you access to a preconfigured, running Vault server with in-memory storage that you can use right away.
Follow the HashiCorp Learn curriculum when you are ready to set up a production server in Vault.
To retrieve your Vault root token:
- Download and install the Vault edition for your operating system.
- Open a terminal window and run
vault server -dev.
The command output includes a
Root Token line.
Find this line in your command output and copy the
Root Token value.
You will use it next to create your Vault secrets provider.
HashiCorp Vault Root Token location
Leave the Vault dev server running.
Because you aren’t using TLS, you will need to set
VAULT_ADDR=http://127.0.0.1:8200 in your shell environment.
Create your Vault secrets provider
NOTE: In Vault’s dev server, TLS is not enabled, so you won’t be able to use certificate-based authentication.
sensuctl create to create your secrets provider,
In the code below, replace
ROOT_TOKEN with the
Root Token value for your Vault dev server.
cat << EOF | sensuctl create --- type: VaultProvider api_version: secrets/v1 metadata: name: vault spec: client: address: http://localhost:8200 token: ROOT_TOKEN version: v2 tls: null max_retries: 2 timeout: 20s rate_limiter: limit: 10 burst: 100 EOF
To continue, skip ahead to create your Vault secret.
Vault TLS certificate authentication
NOTE: You will need to set up HashiCorp Vault in production to use TLS certificate-based authentication. In Vault’s dev server, TLS is not enabled. Follow the HashiCorp Learn curriculum when you are ready to set up a production server in Vault.
First, in your Vault, enable and configure certificate authentication. For example, your Vault might be configured for certificate authentication like this:
vault write auth/cert/certs/sensu-backend \ display_name=sensu-backend \ policies=sensu-backend-policy \ email@example.com \ ttl=3600
Second, configure your
VaultProvider in Sensu:
--- type: VaultProvider api_version: secrets/v1 metadata: name: vault spec: client: address: https://vault.example.com:8200 version: v2 tls: ca_cert: /path/to/your/ca.pem client_cert: /etc/sensu/ssl/sensu-backend-vault.pem client_key: /etc/sensu/ssl/sensu-backend-vault-key.pem cname: sensu-backend.example.com max_retries: 2 timeout: 20s rate_limiter: limit: 10 burst: 100
The certificate you specify for
tls.client_cert should be the same certificate you configured in your Vault for certificate authentication.
Next, create your Vault secret.
Create your Vault secret
First, retrieve your PagerDuty Integration Key (the secret you will set up in Vault).
Next, open a new terminal and run
vault kv put secret/pagerduty key=INTEGRATION_KEY.
INTEGRATION_KEY with your PagerDuty Integration Key.
This writes your secret into Vault.
In this example, the name of the secret is
pagerduty secret contains a key, and you specified that the
key value is your PagerDuty Integration Key.
id value for your secret will be
id value for secrets that target a HashiCorp Vault must start with the name of the secret’s path in Vault.
The Vault dev server is preconfigured with the
secret keyspace already set up, so we recommend using the
secret/ path for the
id value while you are learning and getting started with Vault secrets management.
vault kv get secret/pagerduty to see the secret you just set up.
sensuctl create to create your
cat << EOF | sensuctl create --- type: Secret api_version: secrets/v1 metadata: name: pagerduty_key namespace: default spec: id: secret/pagerduty#key provider: vault EOF
Now you can securely pass your PagerDuty Integration Key in the handlers, and mutators by referring to the
In the add a handler section, you’ll use your
pagerduty_key secret in your handler definition.
Add a handler
Register the PagerDuty Handler asset
sensuctl asset add sensu/sensu-pagerduty-handler:1.2.0 -r pagerduty-handler
This example uses the
-r (rename) flag to specify a shorter name for the asset:
NOTE: You can adjust the asset definition according to your Sensu configuration if needed.
sensuctl asset list --format yaml to confirm that the asset is ready to use.
With this handler, Sensu can trigger and resolve PagerDuty incidents. However, you still need to add your secret to the handler spec so that it requires your backend to request secrets from your secrets provider.
Add your secret to the handler spec
To create a handler definition that uses your
pagerduty_key secret, run:
cat << EOF | sensuctl create --- api_version: core/v2 type: Handler metadata: namespace: default name: pagerduty spec: type: pipe command: pagerduty-handler --token $PD_TOKEN secrets: - name: PD_TOKEN secret: pagerduty_key runtime_assets: - pagerduty-handler timeout: 10 filters: - is_incident EOF
Now that your handler is set up and Sensu can create incidents in PagerDuty, you can automate this workflow by adding your
pagerduty handler to your Sensu service check definitions.
See Monitor server resources to learn more.