Service logging

Logs produced by Sensu services – i.e. sensu-backend and sensu-agent – are often the best place to start when troubleshooting a variety of issues.

Log levels

Each log message is associated with a log level, indicative of the relative severity of the event being logged:

Log Level Description
panic 😰 An error caused the service to exit uncleanly
fatal πŸ‘‹ An error caused the service to exit cleanly (status 0)
error ❗❗ You should definitely look into this
warn ❗❓ Maybe you should look into this?
info πŸ‘Œ Just so you know
debug πŸ“ˆ Verbose output describing lower-level service operations

These log levels can be configured by specifying the desired log level as the value of log-level in the service configuration file (e.g. agent.yml or backend.yml configuration files), or as an argument to the --log-level command line flag:

sensu-agent start --log-level debug

Changes to log level via configuration file or command line arguments require restarting the service. For guidance on restarting a service, please consult the Operating section of the agent or backend reference, respectively.

Log file locations

Sensu services print structured log messages to standard output. In order to capture these log messages to disk or another logging facility, Sensu services make use of capabilities provided by the underlying operating system’s service management. For example, logs are sent to the journald when systemd is the service manager, whereas log messages are redirected to /var/log/sensu when running under sysv init schemes. If you are running systemd as your service manager and would rather have logs written to /var/log/sensu/, see the guide to forwarding logs from journald to syslog.

In the table below, the common targets for logging and example commands for following those logs are described. The name of the desired service, e.g. backend or agent may be substituted for ${service} variable.

Platform Version Target Command to follow log
RHEL/Centos >= 7 journald
journalctl --unit sensu-${service} --follow
RHEL/Centos <= 6 log file
tail --follow /var/log/sensu/sensu-${service}
Ubuntu >= 15.04 journald
journalctl --unit sensu-${service} --follow
Ubuntu <= 14.10 log file
tail --follow /var/log/sensu/sensu-${service}
Debian >= 8 journald
journalctl --unit sensu-${service} --follow
Debian <= 7 log file
tail --follow /var/log/sensu/sensu-${service}
Windows Any log file
Get-Content -  Path "C:\scripts\test.txt" -Wait

NOTE: Platform versions described above are for reference only and do not supercede the documented supported platforms.

Log messages

Permission issues

Files and folders within /var/cache/sensu/ and /var/lib/sensu/ need to be owned by the sensu user and group. You will see a logged error similar to the following if there is a permission issue with either the sensu-backend or the sensu-agent:

{"component":"agent","error":"open /var/cache/sensu/sensu-agent/assets.db: permission denied","level":"fatal","msg":"error executing sensu-agent","time":"2019-02-21T22:01:04Z"}
{"component":"backend","level":"fatal","msg":"error starting etcd: mkdir /var/lib/sensu: permission denied","time":"2019-03-05T20:24:01Z"}

You can use a recursive chown to resolve permission issues with the sensu-backend:

sudo chown -R sensu:sensu /var/cache/sensu/sensu-backend

or the sensu-agent:

sudo chown -R sensu:sensu /var/cache/sensu/sensu-agent