If you need to troubleshoot issues with your cluster, you can check a node's logs, which include details about certain node-level and range-level events, such as errors. For example, if CockroachDB crashes, it normally logs a stack trace to what caused the problem.
CockroachDB also generates secondary logs for queries being executed against your system. See SQL logging for details on obtaining these logs.
When a node processes a
cockroach command, it produces a stream of messages about the command's activities. Each message's body describes the activity, and its envelope contains metadata such as the message's severity level.
As a command generates messages, CockroachDB uses the command's logging flags and the message's severity level to determine the appropriate location for it.
Each node's logs detail only the internal activity of that node without visibility into the behavior of other nodes in the cluster. When troubleshooting, this means that you must identify the node where the problem occurred or collect the logs from all active nodes in your cluster.
cockroach commands support logging. However, it's important to note:
cockroach startgenerates most messages related to the operation of your cluster.
- Other commands do generate messages, but they're typically only interesting in troubleshooting scenarios.
CockroachDB identifies each message with a severity level, letting operators know if they need to intercede:
INFO(lowest severity; no action necessary)
FATAL(highest severity; requires operator attention)
Default behavior by severity level
||Write to file||Write to file|
|All other commands||Discard||Print to
Based on the command's flags and the message's severity level, CockroachDB does one of the following:
Write to file
CockroachDB can write messages to log files. The files are named using the following format:
cockroach.[host].[user].[start timestamp in UTC].[process ID].log
To make it easier to watch a log without knowing the full filename, a symlink with the short filename
cockroach.log is also created. This symlink points to the most recent log.
All log file timestamps are in UTC because CockroachDB is designed to be deployed in a distributed cluster. Nodes may be located in different time zones, and using UTC makes it easy to correlate log messages from those nodes no matter where they are located.
||All other commands|
|Default File Destination||
|Change File Destination||
|Default Severity Level Threshold||
|Change Severity Threshold||
1 If the
cockroach process does not have access to on-disk storage,
cockroach start does not write messages to log files; instead it prints all messages to
Log files can be accessed using the DB Console, which displays them in JSON format.
Access the DB Console and then click Advanced Debug in the left-hand navigation.
Under Raw Status Endpoints (JSON), click Log Files to view the JSON of all collected logs.
Copy one of the log filenames. Then click Specific Log File and replace the
cockroach.logplaceholder in the URL with the filename.
CockroachDB can print messages to
stderr, which normally prints them to the machine's terminal but does not store them.
||All other commands|
|Default Severity Level Threshold||N/A||
|Change Severity Threshold||
cockroach startdoes not print any messages to
cockroachprocess does not have access to on-disk storage, in which case it defaults to
--logtostderr=INFOand prints all messages to
Messages with severity levels below the
--log-file-verbosity flag's values are neither written to files nor printed to
stderr, so they are discarded.
By default, commands besides
cockroach start discard messages with the
INFO severity level.
These logging flags are used with
||Enable logging to files and write logs to the specified directory.
||After the logging group (i.e.,
||After logs reach the specified size, begin writing logs to a new file. The flag's argument takes standard file sizes, such as
||Only writes messages to log files if they are at or above the specified severity level, such as
||Enable logging to
If you use this flag without specifying the severity level (e.g.,
||Do not colorize
When set to
||If non-empty, create a SQL audit log in this directory. By default, SQL audit logs are written in the same directory as the other logs generated by CockroachDB.
Note that enabling SQL audit logs can negatively impact performance. As a result, we recommend using SQL audit logs for security purposes only. For more information, see the
If you contact CockroachDB Support for troubleshooting help, you might be asked to run
cockroach debug zip and share the resulting file with the CockroachDB team. The log files created by
cockroach debug zip may contain highly sensitive, identifiable information, such as usernames, hashed passwords, and possibly your table's data.
New in v20.2 You can run
cockroach debug zip with the
redact-logs flag to redact the sensitive data out of log files and crash reports before sharing them with Cockroach Labs. Redactable sensitive data includes but is not limited to:
- Stored values
- Text of SQL statements, especially the values embedded therein
- Result rows
- The dynamic part of error messages that includes application-provided parameters.
- IP addresses or hostnames
- Database, schema, table, or column names
- Cluster IDs
- File names of stored data or log files
- User/role names
- Hashed passwords