Performance Benchmarking with TPC-C

​

Before you begin

• Review TPC-C concepts
• Request a trial license

​

Review TPC-C concepts

​

Request a trial license

​

Step 1. Set up the environment

• Provision VMs
• Configure your network

​

Provision VMs

1. Create 16 VM instances, 15 for CockroachDB nodes and 1 for the TPC-C workload.
   • Create all instances in the same region and the same security group.
   • Use the c5d.4xlarge machine type.
   • Use local SSD instance store volumes. Local SSDs are low latency disks attached to each VM, which maximizes performance. This configuration best resembles what a bare metal deployment would look like, with machines directly connected to one physical disk each. We do not recommend using network-attached block storage.
2. Note the internal IP address of each instance. You’ll need these addresses when starting the CockroachDB nodes.

​

Configure your network

• 26257 for inter-node communication (i.e., working as a cluster) and for the TPC-C workload to connect to nodes
• 8080 for exposing your DB Console

​

Inter-node and TPCC-to-node communication

​

DB Console

​

Step 2. Start CockroachDB

1. SSH to the first VM where you want to run a CockroachDB node.
2. .
3. Run the command:
   cockroach start \
   --insecure \
   --advertise-addr=<node1 internal address \
   --join=<node1 internal address,<node2 internal address,<node3 internal address \
   --cache=.25 \
   --locality=rack=0
   
   Each node will start with a that includes an artificial “rack number” (e.g., --locality=rack=0). Use 5 racks for 15 nodes so that 3 nodes will be assigned to each rack.
4. Repeat steps 1 - 3 for the other 14 VMs for CockroachDB nodes. Each time, be sure to:
   • Adjust the --advertise-addr flag.
   • Set the flag to the appropriate “rack number”.
5. On any of the VMs with the cockroach binary, run the one-time command to join the first nodes into a cluster:
   cockroach init --insecure --host=<address of any node on --join list
   

​

Step 3. Configure the cluster

1. SSH to any VM with the cockroach binary.
2. Launch the :
   $ cockroach sql --insecure --host=<address of any node
   
3. Adjust some :
   > SET CLUSTER SETTING rocksdb.ingest_backpressure.l0_file_count_threshold = 100;
   SET CLUSTER SETTING schemachanger.backfiller.max_buffer_size = '5 GiB';
   SET CLUSTER SETTING kv.snapshot_rebalance.max_rate = '128 MiB';
   SET CLUSTER SETTING rocksdb.min_wal_sync_interval = '500us';
   SET CLUSTER SETTING kv.range_merge.queue_enabled = false;
   
4. Enable the trial license you requested earlier:
   > SET CLUSTER SETTING cluster.organization = '<your organization';
   
   > SET CLUSTER SETTING enterprise.license = '<your license key';
   
5. Exit the SQL shell:
   > \q
   

​

Step 4. Import the TPC-C dataset

1. SSH to the VM where you want to run TPC-C.
2. .
3. Import the TPC-C dataset:
   $ cockroach workload fixtures import tpcc \
   --partitions=5 \
   --warehouses=13000 \
   'postgres://root@<address of any CockroachDB node:26257?sslmode=disable'
   
   This will load 1.04 TB of data for 13,000 “warehouses”. This can take around 1 hour to complete. You can monitor progress on the Jobs screen of the DB Console. Open the by pointing a browser to the address in the admin field in the standard output of any node on startup.

​

Step 5. Allow the cluster to rebalance

1. Still on the same VM, briefly run TPC-C to let the cluster balance and the leases settle. Bump the file descriptor limits with ulimit to the high value shown in the following snippet, since the workload generators create a lot of database connections.
   $ ulimit -n 100000 && cockroach workload run tpcc \
   --partitions=5 \
   --warehouses=13000 \
   --duration=1m \
   --ramp=1ms \
   'postgres://root@<address of any CockroachDB node:26257?sslmode=disable'
   
2. Wait for range rebalancing to finish. This will likely take 10s of minutes. To watch the progress, go to the Metrics > Queues > Replication Queue graph in the DB Console. Once the Replication Queue gets to 0 for all actions and stays there, you can move on to the next step.

​

Step 6. Run the benchmark

1. Back on the VM with the workload binary, create an addrs file containing connection strings to all 15 CockroachDB nodes:
   postgres://root@<node 1 internal address:26257?sslmode=disable postgres://root@<node 2 internal address:26257?sslmode=disable postgres://root@<node 3 internal address:26257?sslmode=disable postgres://root@<node 4 internal address:26257?sslmode=disable ...
   
2. Run TPC-C for 30 minutes:
   $ ulimit -n 100000 && cockroach workload run tpcc \
   --partitions=5 \
   --warehouses=13000 \
   --ramp=1m \
   --duration=30m \
   $(cat addrs)
   

​

Step 7. Interpret the results

_elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms)
 1800.0s   160420.0  96.0%    303.1    285.2    570.4    671.1    939.5   4160.7

​

See also

• Hardware CockroachDB works well on commodity hardware in public cloud, private cloud, on-prem, and hybrid environments. For hardware recommendations, see our .
• Performance tuning For guidance on tuning a real workload’s performance, see , and for guidance on techniques to minimize network latency in multi-region or global clusters, see .