This page helps you understand and resolve error messages written to
stderr or your logs.
This message indicates a client is trying to connect to a node that is either not running or is not listening on the specified interfaces (i.e., hostname or port).
To resolve this issue, do one of the following:
- If the node hasn't yet been started, start the node.
- If you specified a
--advertise-addrflag when starting the node, you must include the specified IP address/hostname and port with all other
cockroachcommands or change the
If you're not sure what the IP address/hostname and port values might have been, you can look in the node's logs. If necessary, you can also terminate the
cockroach process, and then restart the node:
- If the node was started with a process manager, gracefully stop the node by sending
SIGTERMwith the process manager. If the node is not shutting down after 1 minute, send
SIGKILLto terminate the process. When using
systemd, for example, set
TimeoutStopSecs=60in your configuration template and run
systemctl stop <systemd config filename>to stop the node without
- If the node was started using
cockroach startand is running in the foreground, press
ctrl-cin the terminal.
- If the node was started using
cockroach startand the
kill <pid>, where
<pid>is the process ID of the node.
The amount of time you should wait before sending
SIGKILL can vary depending on your cluster configuration and workload, which affects how long it takes your nodes to complete a graceful shutdown. In certain edge cases, forcefully terminating the process before the node has completed shutdown can result in temporary data unavailability, latency spikes, uncertainty errors, ambiguous commit errors, or query timeouts. If you need maximum cluster availability, you can run
cockroach node drain prior to node shutdown and actively monitor the draining process instead of automating it.
Then restart the node:
$ cockroach start [flags]
node is running secure mode, SSL connection required
This message indicates that the cluster is using TLS encryption to protect network communication, and the client is trying to open a connection without using the required TLS certificates.
To resolve this issue, use the
cockroach cert create-client command to generate a client certificate and key for the user trying to connect. For a secure deployment walkthrough, including generating security certificates and connecting clients, see Manual Deployment.
Messages with the error code
40001 and the string
restart transaction indicate that a transaction failed because it conflicted with another concurrent or recent transaction accessing the same data. The transaction needs to be retried by the client. For more information about how to implement client-side retries, see client-side retry handling.
For more information about the different types of transaction retry errors such as "retry write too old", "read within uncertainty interval", etc., see the Transaction Retry Error Reference.
node belongs to cluster <cluster ID> but is attempting to connect to a gossip network for cluster <another cluster ID>
This message usually indicates that a node tried to connect to a cluster, but the node is already a member of a different cluster. This is determined by metadata in the node's data directory. To resolve this issue, do one of the following:
Choose a different directory to store the CockroachDB data:
$ cockroach start [flags] --store=[new directory] --join=[cluster host]:26257
Remove the existing directory and start a node joining the cluster again:
$ rm -r cockroach-data/
$ cockroach start [flags] --join=[cluster host]:26257
clock synchronization error: this node is more than 500ms away from at least half of the known nodes
This error indicates that a node has spontaneously shut down because it detected that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default). CockroachDB requires moderate levels of clock synchronization to preserve data consistency, so the node shutting down in this way avoids the risk of consistency anomalies.
To prevent this from happening, you should run clock synchronization software on each node. For guidance on synchronizing clocks, see the tutorial for your deployment environment:
|Manual||Use NTP with Google's external NTP service.|
|AWS||Use the Amazon Time Sync Service.|
|Azure||Disable Hyper-V time synchronization and use NTP with Google's external NTP service.|
|Digital Ocean||Use NTP with Google's external NTP service.|
|GCE||Use NTP with Google's internal NTP service.|
open file descriptor limit of <number> is under the minimum required <number>
CockroachDB can use a large number of open file descriptors, often more than is available by default. This message indicates that the machine on which a CockroachDB node is running is under CockroachDB's recommended limits.
For more details on CockroachDB's file descriptor limits and instructions on increasing the limit on various platforms, see File Descriptors Limit.
replicas failing with "0 of 1 store with an attribute matching ; likely not enough nodes in cluster
When running a single-node cluster
When running a single-node CockroachDB cluster, an error about replicas failing will eventually show up in the node's log files, for example:
E160407 09:53:50.337328 storage/queue.go:511 [replicate] 7 replicas failing with "0 of 1 store with an attribute matching ; likely not enough nodes in cluster"
This happens because CockroachDB expects three nodes by default. If you do not intend to add additional nodes, you can stop this error by using
ALTER RANGE ... CONFIGURE ZONE to update your default zone configuration to expect only one node:
# Insecure cluster: $ cockroach sql --execute="ALTER RANGE default CONFIGURE ZONE USING num_replicas=1;" --insecure
# Secure cluster: $ cockroach sql --execute="ALTER RANGE default CONFIGURE ZONE USING num_replicas=1;" --certs-dir=[path to certs directory]
When running a multi-node cluster
When running a multi-node CockroachDB cluster, if you see an error like the one above about replicas failing, some nodes might not be able to talk to each other. For recommended actions, see Cluster Setup Troubleshooting.
split failed while applying backpressure; are rows updated in a tight loop?
In CockroachDB, a table row is stored on disk as a key-value pair. Whenever the row is updated, CockroachDB also stores a distinct version of the key-value pair to enable concurrent request processing while guaranteeing consistency (see multi-version concurrency control (MVCC)). All versions of a key-value pair belong to a larger "range" of the total key space, and the historical versions remain until the garbage collection period defined by the
gc.ttlseconds variable in the applicable zone configuration has passed (25 hours by default). Once a range reaches a size threshold (512 MiB by default), CockroachDB splits the range into two ranges. However, this message indicates that a range cannot be split as intended.
One possible cause is that the range consists only of MVCC version data due to a row being repeatedly updated, and the range cannot be split because doing so would spread MVCC versions for a single row across multiple ranges.
To resolve this issue, make sure you are not repeatedly updating a single row. If frequent updates of a row are necessary, consider one of the following:
- Reduce the
gc.ttlsecondsvariable in the applicable zone configuration to reduce the garbage collection period and prevent such a large build-up of historical values.
- If a row contains large columns that are not being updated with other columns, put the large columns in separate column families.
context deadline exceeded
This message occurs when a component of CockroachDB gives up because it was relying on another component that has not behaved as expected, for example, another node dropped a network connection. To investigate further, look in the node's logs for the primary failure that is the root cause.
pq: failed to verify protection id...
Messages that begin with
pq: failed to verify protection id… indicate that your incremental backup failed because the data you are trying to backup was garbage collected. This happens when incremental backups are taken less frequently than the garbage collection periods for any of the objects in the base backup. For example, if your incremental backups recur daily, but the garbage collection period of one table in your backup is less than one day, all of your incremental backups will fail.
The error message will specify which part of your backup is causing the failure. For example,
r13485:/Table/771 indicates that table
771 is part of the problem. You can then inspect this table by running
SELECT * FROM crdb_internal.tables WHERE id=771. You can also run
SHOW ZONE CONFIGURATIONS and look for any
gc.ttlseconds values that are set lower than your incremental backup frequency.
To resolve this issue, take a new full backup after doing either of the following:
- Increase the garbage collection period by configuring the
gc.ttlsecondsreplication zone variable, or
- Increase the frequency of incremental backups.
result is ambiguous
In a distributed system, some errors can have ambiguous results. For
example, if you receive a
connection closed error while processing a
COMMIT statement, you cannot tell whether the transaction
successfully committed or not. These errors are possible in any
database, but CockroachDB is somewhat more likely to produce them than
other databases because ambiguous results can be caused by failures
between the nodes of a cluster. These errors are reported with the
PostgreSQL error code
statement_completion_unknown) and the
result is ambiguous.
Ambiguous errors can be caused by nodes crashing, network failures, or
timeouts. If you experience a lot of these errors when things are
otherwise stable, look for performance issues. Note that ambiguity is
only possible for the last statement of a transaction (
RELEASE SAVEPOINT) or for statements outside a transaction. If a connection drops during a transaction that has not yet tried to commit, the transaction will definitely be aborted.
In general, you should handle ambiguous errors the same way as
connection closed errors. If your transaction is
it is safe to retry it on ambiguous errors.
UPSERT operations are
typically idempotent, and other transactions can be written to be
idempotent by verifying the expected state before performing any
writes. Increment operations such as
UPDATE my_table SET x=x+1 WHERE
id=$1 are typical examples of operations that cannot easily be made
idempotent. If your transaction is not idempotent, then you should
decide whether to retry or not based on whether it would be better for
your application to apply the transaction twice or return an error to
Try searching the rest of our docs for answers or using our other support resources, including: