Logging

If you need to monitor your cluster, tune performance, or troubleshoot issues, you can check the CockroachDB logs, which include details about notable cluster, node, and range-level events.

Note:

This logging system has been introduced in v21.1 and is backward-compatible. For a summary of what has changed, see Changes to logging system.

Details

When a node processes a cockroach command, it produces a stream of messages about the command's activities. Each message is composed of:

  • A payload that contains events either structured in JSON or conveyed in an arbitrary string. For details on structured event types and their fields, see Notable Event Types.
  • An envelope that contains event metadata (e.g., severity, date, timestamp, channel). Depending on the log format you specify when configuring logs, the envelope can be formatted either in JSON or as a flat prefix to the message.

Messages are organized into appropriate logging channels and then routed through log sinks. Each sink further processes and filters the messages before emitting them to destinations outside CockroachDB. The mapping of channels to sinks, as well as the processing and filtering done by each sink, is configurable and is intended to support multiple use cases. For more information on configuring sinks and severity levels, see Configure Logs.

Tip:

All cockroach commands support logging. However, note that most messages related to cluster operation are generated by cockroach start or cockroach start-single-node. Other commands generate messages related to their own execution, which are mainly useful when troubleshooting the behaviors of those commands.

Logging channels

Log messages in CockroachDB are directed into logging channels, which can in turn be assigned to output to one or more log sinks.

This allows you to group channels that log related information (e.g., operational, security, or SQL events) into their own sinks. Each sink can output to a predetermined destination where the logs can be collected and parsed. For usage examples, see Logging Use Cases.

Channel Description
DEV Uncategorized and debug messages.
OPS Process starts, stops, shutdowns, and crashes (if they can be logged); changes to cluster topology, such as node additions, removals, and decommissions.
HEALTH Resource usage; node-node connection events, including connection errors; up- and down-replication and range unavailability.
STORAGE Low-level storage logs from Pebble or RocksDB.
SESSIONS Client connections and disconnections (when enabled via the server.auth_log.sql_connections.enabled cluster setting); SQL authentication logins/attempts and session/query terminations (when enabled via the server.auth_log.sql_sessions.enabled cluster setting).
SQL_SCHEMA Database, schema, table, sequence, view, and type creation; changes to table columns and sequence parameters.
USER_ADMIN Changes to users, roles, and authentication credentials.
PRIVILEGES Changes to privileges and object ownership.
SENSITIVE_ACCESS SQL audit events (when enabled via ALTER TABLE ... EXPERIMENTAL_AUDIT).
SQL_EXEC SQL statement executions (when enabled via the sql.trace.log_statement_execute) cluster setting and uncaught Go panic errors during SQL statement execution.
SQL_PERF SQL executions that impact performance, such as slow queries (when enabled via the sql.log.slow_query.latency_threshold and/or sql.log.slow_query.experimental_full_table_scans.enabled cluster settings).

Logging channels are analogous to logging facilities in Syslog or logging services in Datadog. For more details on the contents of each logging channel, see the Logging reference.

Changes to logging system

Prior to v21.1, logs were separated into a general CockroachDB log and secondary SQL and storage logs. These were output to correspondingly named log files.

The events collected by those logs are now directed into the following logging channels:

Filename (legacy) Description Channel
cockroach.log General CockroachDB log DEV
cockroach-pebble.log Pebble/RocksDB log STORAGE
cockroach-sql-audit.log SQL audit log SENSITIVE_ACCESS
cockroach-sql-exec.log SQL execution log SQL_EXEC
cockroach-auth.log SQL authentication log SESSIONS
cockroach-sql-slow.log Slow query log SQL_PERF
Note:

In v21.1, the default logging configuration keeps the legacy filenames for these channels. Starting in v21.2, you will need to explicitly map the channels to the legacy filenames in order to preserve this configuration. For details on this, see Configure Logs.

Notable events that were previously collected in the general CockroachDB log are now directed into several new logging channels:

  • OPS
  • HEALTH
  • SQL_SCHEMA
  • USER_ADMIN
  • PRIVILEGES

New in v21.1: Logging is now configurable via YAML. The YAML configuration allows you to customize which kinds of events are output to different logging destinations, along with many other parameters. As a result, the logging flags previously used with cockroach commands are now deprecated in favor of the YAML configuration. For details, see Configure Logs.

See also

YesYes NoNo