cockroach start

​

Synopsis

$ cockroach start <flags, including --join

$ cockroach init <flags>

$ cockroach start <flags, including --join

$ cockroach start --help

​

Flags

​

General

​

Networking

​

Security

​

Locality

• Specifying a region with a region tier is required in order to enable CockroachDB’s .
• CockroachDB spreads the replicas of each piece of data across as diverse a set of localities as possible, with the order determining the priority. Locality can also be used to influence the location of data replicas in various ways using high-level or low-level .
• When there is high latency between nodes (e.g., cross-availability zone deployments), CockroachDB uses locality to move range leases closer to the current workload, reducing network round trips and improving read performance, also known as . In a deployment across more than 3 availability zones, however, to ensure that all data benefits from “follow-the-workload”, you must increase your replication factor to match the total number of availability zones.
• Locality is also a prerequisite for using the , , and Enterprise features.

​

Example

cockroach start --locality=region=us-east-1,zone=us-east-1a # ... other required flags go here

cockroach start --locality=region=us-east-1,zone=us-east-1b # ... other required flags go here

cockroach start --locality=region=us-east-1,zone=us-east-1c # ... other required flags go here

cockroach start --locality=region=us-west-1,zone=us-west-1a # ... other required flags go here

cockroach start --locality=region=us-west-1,zone=us-west-1b # ... other required flags go here

cockroach start --locality=region=us-west-1,zone=us-west-1c # ... other required flags go here

cockroach start --locality=region=europe-west-1,zone=europe-west-1a # ... other required flags go here

cockroach start --locality=region=europe-west-1,zone=europe-west-1b # ... other required flags go here

cockroach start --locality=region=europe-west-1,zone=europe-west-1c # ... other required flags go here

​

Load-based lease rebalancing in uneven latency deployments

• Your cluster runs in datacenters which are very different distances away from each other.
• Each node was started with a single tier of --locality, e.g., --locality=datacenter=a.
• Most client requests get sent to a single datacenter because that’s where all your application traffic is.

​

Storage

• Storage engine
• Store
• Write Ahead Log (WAL) Failover

​

Storage engine

​

Store

​

Write Ahead Log (WAL) failover

• Pairs each primary store with a secondary failover store at node startup.
• Monitors latency of all write operations against the primary WAL. If any operation exceeds the duration of , the node redirects new WAL writes to the secondary store.
• Checks the primary store while failed over by performing a set of filesystem operations against a small internal “probe file” on its volume. This file contains no user data and exists only when WAL failover is enabled.
• Switches back to the primary store once the set of filesystem operations against the probe file on its volume starts consuming less than a latency threshold (order of tens of milliseconds). If a probe fsync blocks longer than , CockroachDB emits a log like: disk stall detected: sync on file probe-file has been ongoing for 40.0s and, if the stall persists, the node exits (fatals) to and allow recovery elsewhere.
• Exposes status at so you can monitor each store’s health and failover state.

Enable WAL failover

• Pass --wal-failover=among-stores to cockroach start, or
• Set the environment variable COCKROACH_WAL_FAILOVER=among-stores before starting the node.

1. Disable . File-based audit logging cannot coexist with the WAL failover feature. File-based audit logging provides guarantees that every log message makes it to disk, or CockroachDB must be shut down. For this reason, resuming operations in the face of disk unavailability is not compatible with audit logging.
2. Enable asynchronous buffering of using the buffering configuration option. The buffering configuration can be applied to or individual file-groups as needed. Note that enabling asynchronous buffering of file-groups log sinks is in .
3. Set max-staleness: 1s and flush-trigger-size: 256KiB.
4. When buffering is enabled, buffered-writes must be explicitly disabled as shown in the following example. This is necessary because buffered-writes does not provide true asynchronous disk access, but rather a small buffer. If the small buffer fills up, it can cause internal routines performing logging operations to hang. This will in turn cause internal routines doing other important work to hang, potentially affecting cluster stability.
5. The recommended logging configuration for using file-based logging with WAL failover is as follows:

file-defaults:
 buffered-writes: false
 auditable: false
 buffering:
   max-staleness: 1s
   flush-trigger-size: 256KiB
   max-buffer-size: 50MiB

Disable WAL failover

• Pass the --wal-failover=disabled flag to cockroach start, or
• Set the environment variable COCKROACH_WAL_FAILOVER=disabled before restarting the node.

Monitor WAL failover

• storage.wal.failover.secondary.duration: Cumulative time spent (in nanoseconds) writing to the secondary WAL directory. Only populated when WAL failover is configured.
• storage.wal.failover.primary.duration: Cumulative time spent (in nanoseconds) writing to the primary WAL directory. Only populated when WAL failover is configured.
• storage.wal.failover.switch.count: Count of the number of times WAL writing has switched from primary to secondary store, and vice versa.
• storage.wal.fsync.latency monitors the latencies of WAL files. If you have WAL failover enabled and are failing over, storage.wal.fsync.latency will include the latency of the stalled primary.
• storage.wal.failover.write_and_sync.latency: When WAL failover is configured in a cluster, the operator should monitor this metric which shows the effective latency observed by the higher layer writing to the WAL. This metric is expected to stay low in a healthy system, regardless of whether WAL files are being written to the primary or secondary.

• The in .
• By .

​

Logging

​

Defaults

​

Standard output

CockroachDB node starting at
build:               CCL v26.1.1 @ 2026-03-09 00:00:00 (go1.12.6)
webui:               http://localhost:8080
sql:                 postgresql://root@localhost:26257?sslmode=disable
sql (JDBC):          jdbc:postgresql://localhost:26257/defaultdb?sslmode=disable&user=root
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --insecure
logs:                /Users/<username/node1/logs
temp dir:            /Users/<username/node1/cockroach-temp242232154
external I/O path:   /Users/<username/node1/extern
store[0]:            path=/Users/<username/node1
status:              initialized new cluster
clusterID:           8a681a16-9623-4fc1-a537-77e9255daafd
nodeID:              1

​

Examples

​

Start a multi-node cluster

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node1 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node2 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node3 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

$ cockroach init \
--certs-dir=certs \
--host=<address of any node

$ cockroach start \
--insecure \
--advertise-addr=<node1 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--insecure \
--advertise-addr=<node2 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--insecure \
--advertise-addr=<node3 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

$ cockroach init \
--insecure \
--host=<address of any node

​

Start a multi-region cluster

1. Start a node in the us-east1 region:
   cockroach start --locality=region=us-east1,zone=us-east-1a \
                     --insecure --store=/tmp/node0 \
                     --listen-addr=localhost:26257 \
                     --http-port=8888 \
                     --join=localhost:26257,localhost:26258,localhost:26259
   
2. Start a node in the us-west1 region:
   cockroach start --locality=region=us-west1,zone=us-west-1a \
                     --insecure \
                     --store=/tmp/node2 \
                     --listen-addr=localhost:26259 \
                     --http-port=8890 \
                     --join=localhost:26257,localhost:26258,localhost:26259
   
3. Start a node in the europe-west1 region:
   cockroach start --locality=region=europe-west1,zone=europe-west-1a \
                     --insecure \
                     --store=/tmp/node1 \
                     --listen-addr=localhost:26258 \
                     --http-port=8889 \
                     --join=localhost:26257,localhost:26258,localhost:26259
   
4. Initialize the cluster:
   cockroach init --insecure --host=localhost --port=26257
   
5. Connect to the cluster using :
   cockroach sql --host=localhost --port=26257 --insecure
   
6. Issue the statement to verify that the list of regions is expected:
   SHOW REGIONS;
   
       region    | zones | database_names | primary_region_of
   ---------------+-------+----------------+--------------------
     europe-west1 | {}    | {}             | {}
     us-east1     | {}    | {}             | {}
     us-west1     | {}    | {}             | {}
   (3 rows)
   

​

Start a multi-node cluster across private networks

• You have a cluster that spans GCE and AWS.
• The nodes on each cloud can reach each other on public addresses, but the private addresses aren’t reachable from the other cloud.

1. Start each node on GCE with --locality set to describe its location, --locality-advertise-addr set to advertise its private address to other nodes in on GCE, --advertise-addr set to advertise its public address to nodes on AWS, and --join set to the public addresses of 3-5 of the initial nodes:
   $ cockroach start \
   --certs-dir=certs \
   --locality=cloud=gce \
   --locality-advertise-addr=cloud=gce@<private address of node \
   --advertise-addr=<public address of node \
   --join=<public address of node1,<public address of node2,<public address of node3 \
   --cache=.25 \
   --max-sql-memory=.25
   
2. Start each node on AWS with --locality set to describe its location, --locality-advertise-addr set to advertise its private address to other nodes on AWS, --advertise-addr set to advertise its public address to nodes on GCE, and --join set to the public addresses of 3-5 of the initial nodes:
   $ cockroach start \
   --certs-dir=certs \
   --locality=cloud=aws \
   --locality-advertise-addr=cloud=aws@<private address of node \
   --advertise-addr=<public address of node \
   --join=<public address of node1,<public address of node2,<public address of node3 \
   --cache=.25 \
   --max-sql-memory=.25
   
3. Run the command against any node to perform a one-time cluster initialization:
   $ cockroach init \
   --certs-dir=certs \
   --host=<address of any node
   

​

Add a node to a cluster

$ cockroach start \
--certs-dir=certs \
--advertise-addr=<node4 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

$ cockroach start \
--insecure \
--advertise-addr=<node4 address \
--join=<node1 address,<node2 address,<node3 address \
--cache=.25 \
--max-sql-memory=.25

​

Create a table with node locality information

$ cockroach start --insecure --port=26257 --http-port=26258 --store=cockroach-data/1 --cache=256MiB --locality=region=eu-west-1,cloud=aws,zone=eu-west-1a

$ cockroach start --insecure --port=26259 --http-port=26260 --store=cockroach-data/2 --cache=256MiB --join=localhost:26257 --locality=region=eu-west-1,cloud=aws,zone=eu-west-1b

$ cockroach start --insecure --port=26261 --http-port=26262 --store=cockroach-data/3 --cache=256MiB --join=localhost:26257 --locality=region=eu-west-1,cloud=aws,zone=eu-west-1c

$ cockroach sql --insecure

> CREATE TABLE charges (
  zone STRING NOT NULL DEFAULT crdb_internal.locality_value('zone'),
  id INT PRIMARY KEY NOT NULL
);

> INSERT INTO charges (id) VALUES (1);

> SELECT * FROM charges WHERE id = 1;

     zone    | id
+------------+----+
  eu-west-1a |  1
(1 row)

$ cockroach sql --insecure --port 26259

> INSERT INTO charges (id) VALUES (2);

> SELECT * FROM charges WHERE id = 2;

     zone    | id
+------------+----+
  eu-west-1b |  2
(1 row)

$ cockroach sql --insecure --port 26261

> INSERT INTO charges (id) VALUES (3);

> SELECT * FROM charges WHERE id = 3;

     zone    | id
+------------+----+
  eu-west-1c |  3
(1 row)

​

Start a cluster with separate RPC and SQL networks

$ cockroach start --sql-addr=:26257 --listen-addr=:26258 --join=node1:26258,node2:26258,node3:26258 --certs-dir=~/cockroach-certs

​

See also