> ## Documentation Index
> Fetch the complete documentation index at: https://www.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Performance Benchmarking with TPC-C

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

This page shows you how to reproduce <InternalLink path="performance#scale">CockroachDB TPC-C performance benchmarking results</InternalLink>. Across all scales, CockroachDB can process tpmC (new order transactions per minute) at near maximum efficiency. Start by choosing the scale you're interested in:

| Workload             | Cluster size                                            | Warehouses | Data size |
| -------------------- | ------------------------------------------------------- | ---------- | --------- |
| Local                | 3 nodes on your laptop                                  | 10         | 2 GB      |
| Local (multi-region) | 9 in-memory nodes on your laptop using `cockroach demo` | 10         | 2 GB      |
| Small                | 3 nodes on `c5d.4xlarge` machines                       | 2500       | 200 GB    |
| Medium               | 15 nodes on `c5d.4xlarge` machines                      | 13,000     | 1.04 TB   |
| Large                | 81 nodes on `c5d.9xlarge` machines                      | 140,000    | 11.2 TB   |

## Before you begin

* [Review TPC-C concepts](#review-tpc-c-concepts)
* [Request a trial license](#request-a-trial-license)

### Review TPC-C concepts

TPC-C provides the most realistic and objective measure for OLTP performance at various scale factors. Before you get started, consider reviewing <InternalLink path="performance#tpc-c">what TPC-C is and how it is measured</InternalLink>.

### Request a trial license

Reproducing these TPC-C results involves using CockroachDB's <InternalLink path="partitioning">partitioning</InternalLink> feature to ensure replicas for any given section of data are located on the same nodes that will be queried by the load generator for that section of data. Partitioning helps distribute the workload evenly across the cluster.

The partitioning feature requires an Enterprise license, so [request a 30-day trial license](https://www.cockroachlabs.com/get-cockroachdb/enterprise) before you get started.

You should receive your trial license via email within a few minutes. You'll enable your license once your cluster is up-and-running.

## Step 1. Set up the environment

* [Provision VMs](#provision-vms)
* [Configure your network](#configure-your-network)

### Provision VMs

1. [Create 86 VM instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances), 81 for CockroachDB nodes and 5 for the TPC-C workload.
   * Create all instances in the same region and the same security group.
   * Use the `c5d.9xlarge` machine type.
   * Use [local SSD instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage#instance-store-volumes). Local SSDs are low latency disks attached to each VM, which maximizes performance. This configuration best resembles what a bare metal deployment would look like, with machines directly connected to one physical disk each. We do not recommend using network-attached block storage.
2. Note the internal IP address of each instance. You'll need these addresses when starting the CockroachDB nodes.

<Danger>
  This configuration is intended for performance benchmarking only. For production deployments, there are other important considerations, such as security, load balancing, and data location techniques to minimize network latency. For more details, see the <InternalLink path="recommended-production-settings">Production Checklist</InternalLink>.
</Danger>

### Configure your network

CockroachDB requires TCP communication on two ports:

* `26257` for inter-node communication (i.e., working as a cluster) and for the TPC-C workload to connect to nodes
* `8080` for exposing your DB Console

[Create inbound rules](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security#adding-security-group-rule) for your security group:

#### Inter-node and TPCC-to-node communication

| Field      | Recommended Value                                     |
| ---------- | ----------------------------------------------------- |
| Type       | Custom TCP Rule                                       |
| Protocol   | TCP                                                   |
| Port Range | **26257**                                             |
| Source     | The name of your security group (e.g., *sg-07ab277a*) |

#### DB Console

| Field      | Recommended Value        |
| ---------- | ------------------------ |
| Type       | Custom TCP Rule          |
| Protocol   | TCP                      |
| Port Range | **8080**                 |
| Source     | Your network's IP ranges |

## Step 2. Start CockroachDB

<Danger>
  The `--insecure` flag used in this tutorial is intended for non-production testing only. To run CockroachDB in production, use a secure cluster instead.
</Danger>

1. SSH to the first VM where you want to run a CockroachDB node.
2. <InternalLink path="install-cockroachdb">Install CockroachDB for Linux</InternalLink>.
3. Start CockroachDB using the <InternalLink path="cockroach-start">`cockroach start`</InternalLink> command:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroach start \
   --insecure \
   --advertise-addr=<node1 internal address \
   --join=<node1 internal address,<node2 internal address,<node3 internal address \
   --cache=.25 \
   --locality=rack=0
   ```

   Each node will start with a <InternalLink path="cockroach-start#locality">locality</InternalLink> that includes an artificial "rack number" (e.g., `--locality=rack=0`). Use 81 racks for 81 nodes so that 1 node will be assigned to each rack.
4. Repeat these steps for the other 80 VMs for CockroachDB nodes. Each time, be sure to:
   * Adjust the `--advertise-addr` flag.
   * Set the <InternalLink path="cockroach-start#locality">`--locality`</InternalLink> flag to the appropriate "rack number".
5. On any of the VMs with the `cockroach` binary, run the one-time <InternalLink path="cockroach-init">`cockroach init`</InternalLink> command to join the first nodes into a cluster:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroach init --insecure --host=<address of any node on --join list
   ```

## Step 3. Configure the cluster

You'll be importing a large TPC-C data set. To speed that up, you can temporarily disable replication and tweak some cluster settings. You'll also need to enable the Enterprise license you requested earlier.

1. SSH to any VM with the `cockroach` binary.
2. Launch the <InternalLink path="cockroach-sql">built-in SQL shell</InternalLink>:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach sql --insecure --host=<address of any node
   ```
3. Adjust some <InternalLink path="cluster-settings">cluster settings</InternalLink>:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SET CLUSTER SETTING kv.dist_sender.concurrency_limit = 2016;
   SET CLUSTER SETTING kv.snapshot_rebalance.max_rate = '256 MiB';
   SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false;
   SET CLUSTER SETTING schemachanger.backfiller.max_buffer_size = '5 GiB';
   SET CLUSTER SETTING rocksdb.min_wal_sync_interval = '500us';
   SET CLUSTER SETTING kv.range_merge.queue_enabled = false
   ```
4. Change the default <InternalLink path="configure-replication-zones#gc-ttlseconds">GC TTL</InternalLink> to the following value:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 600;
   ```
5. Enable the trial license you requested earlier:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SET CLUSTER SETTING cluster.organization = '<your organization';
   ```

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SET CLUSTER SETTING enterprise.license = '<your license key';
   ```
6. Exit the SQL shell:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > \q
   ```

## Step 4. Import the TPC-C dataset

CockroachDB comes with a number of <InternalLink path="cockroach-workload">built-in workloads</InternalLink> for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc) workload.

1. SSH to the VM where you want to run TPC-C.
2. <InternalLink path="install-cockroachdb">Install CockroachDB for Linux</InternalLink>.
3. Import the TPC-C dataset:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ cockroach workload fixtures import tpcc \
   --partitions=81 \
   --warehouses=140000 \
   --replicate-static-columns \
   --partition-strategy=leases \
   'postgres://root@<address of any CockroachDB node:26257?sslmode=disable'
   ```

   This will load 11.2 TB of data for 140,000 "warehouses". This can take up to 8 hours to complete.

   You can monitor progress on the **Jobs** screen of the DB Console. Open the <InternalLink path="ui-overview">DB Console</InternalLink> by pointing a browser to the address in the `admin` field in the standard output of any node on startup.

## Step 5. Partition the database

1. <InternalLink path="partitioning">Partition your database</InternalLink> to divide all of the TPC-C tables and indexes into 81 partitions, one per rack, and then use <InternalLink path="configure-replication-zones#gc-ttlseconds">zone configurations</InternalLink> to pin those partitions to a particular rack.
2. Wait for up-replication and partitioning to finish. You will know when they have finished because both the number of *lease transfers* and *snapshots* will go down to `0` and stay there. This will likely take 10s of minutes.
   * To monitor the number of lease transfers, open the <InternalLink path="ui-overview">DB Console</InternalLink>, select the **Replication** dashboard, hover over the **Range Operations** graph, and check the **Lease Transfers** data point.
   * To check the number of snapshots, open the <InternalLink path="ui-overview">DB Console</InternalLink>, select the **Replication** dashboard, and hover over the **Snapshots** graph.

     <img src="https://mintcdn.com/cockroachlabs/mWJHSqov-YrcB8t4/images/v25.4/tpcc-large-replication-dashboard.png?fit=max&auto=format&n=mWJHSqov-YrcB8t4&q=85&s=2e842017eca287dc2d009784782b0566" alt="TPC-C 140k replication and partitioning dashboards" width="2072" height="1048" data-path="images/v25.4/tpcc-large-replication-dashboard.png" />

## Step 7. Allocate partitions

Before running the benchmark, it's important to allocate partitions to workload binaries properly to ensure that the cluster is balanced.

1. Create an `addrs` file containing connection strings to all 81 CockroachDB nodes:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   postgres://root@<node 1 internal address:26257?sslmode=disable postgres://root@<node 2 internal address:26257?sslmode=disable postgres://root@<node 3 internal address:26257?sslmode=disable postgres://root@<node 4 internal address:26257?sslmode=disable ...
   ```
2. Upload the `addrs` file to the 5 VMs with the `workload` binary:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp addrs <username>@<workload instance 1 address:.
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp addrs <username>@<workload instance 2 address:.
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp addrs <username>@<workload instance 3 address:.
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp addrs <username>@<workload instance 4 address:.
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp addrs <username>@<workload instance 5 address:.
   ```
3. SSH to each VM with `workload` and allocate partitions:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ulimit -n 500000 && cockroach workload run tpcc --partitions=81 \
   --warehouses=140000 \
   --partition-affinity=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 \
   --ramp=30m \
   --duration=1ms \
   --histograms=workload1.histogram.ndjson \
   $(cat addrs)
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ulimit -n 500000 && cockroach workload run tpcc \
   --partitions 81 \
   --warehouses 140000 \
   --partition-affinity=17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32 \
   --ramp=30m \
   --duration=1ms \
   --histograms=workload2.histogram.ndjson \
   $(cat addrs)
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ulimit -n 500000 && cockroach workload run tpcc \
   --partitions=81 \
   --warehouses=140000 \
   --partition-affinity=33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 \
   --ramp=30m \
   --duration=1ms \
   --histograms=workload3.histogram.ndjson \
   $(cat addrs)
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ulimit -n 500000 && cockroach workload run tpcc \
   --partitions=81 \
   --warehouses=140000 \
   --partition-affinity=49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64 \
   --ramp=30m \
   --duration=1ms \
   --histograms=workload4.histogram.ndjson \
   $(cat addrs)
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ulimit -n 500000 && cockroach workload run tpcc \
   --partitions=81 \
   --warehouses=140000 \
   --partition-affinity=65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80 \
   --ramp=30m \
   --duration=1ms \
   --histograms=workload5.histogram.ndjson \
   $(cat addrs)
   ```

## Step 8. Run the benchmark

Once the allocations finish, run TPC-C for 30 minutes on each VM with `workload`:

<Note>
  It is critical to run the benchmark from the workload nodes in parallel, so start them as simultaneously as possible.
</Note>

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ulimit -n 500000 && cockroach workload run tpcc \
--partitions=81 \
--warehouses=140000 \
--partition-affinity=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16 \
--ramp=4m \
--duration=30m \
--histograms=workload1.histogram.ndjson \
$(cat addrs)
```

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ulimit -n 500000 && cockroach workload run tpcc \
--partitions=81 \
--warehouses=140000 \
--partition-affinity=17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32 \
--ramp=4m \
--duration=30m \
--histograms=workload2.histogram.ndjson \
$(cat addrs)
```

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ulimit -n 500000 && cockroach workload run tpcc \
--partitions=81 \
--warehouses=140000 \
--partition-affinity=33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 \
--ramp=4m \
--duration=30m \
--histograms=workload3.histogram.ndjson \
$(cat addrs)
```

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ulimit -n 500000 && cockroach workload run tpcc \
--partitions=81 \
--warehouses=140000 \
--partition-affinity=49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64 \
--ramp=4m \
--duration=30m \
--histograms=workload4.histogram.ndjson \
$(cat addrs)
```

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ulimit -n 500000 && cockroach workload run tpcc \
--partition=81 \
--warehouses=140000 \
--partition-affinity=65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80 \
--ramp=4m \
--duration=30m \
--histograms=workload5.histogram.ndjson \
$(cat addrs)
```

## Step 9. Interpret the results

1. Collect the result files from each VM with `workload`:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp <username>@<workload instance 1 address:workload1.histogram.ndjson .
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp <username>@<workload instance 2 address:workload2.histogram.ndjson .
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp <username>@<workload instance 3 address:workload3.histogram.ndjson .
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp <username>@<workload instance 4 address:workload4.histogram.ndjson .
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ scp <username>@<workload instance 5 address:workload5.histogram.ndjson .
   ```
2. Upload the result files to one of the VMs with the `workload` binary:

<Note>
  The following commands assume you're uploading to the VM with the `workload1.histogram.ndjson` file.
</Note>

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp workload2.histogram.ndjson <username>@<workload instance 2 address:.
```

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp workload3.histogram.ndjson <username>@<workload instance 3 address:.
```

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp workload4.histogram.ndjson <username>@<workload instance 4 address:.
```

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp workload5.histogram.ndjson <username>@<workload instance 5 address:.
```

3. SSH to the VM where you uploaded the results files.
4. Run the `workload debug tpcc-merge-results` command to synthesize the results:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroach workload debug tpcc-merge-results \
   --warehouses=140000 \
   workload*.histogram.ndjson
   ```

   You'll should see results similar to the following, with **1.68M tpmC with 140,000 warehouses, resulting in an efficiency score of 95%**:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   Duration: 30m1., Warehouses: 140000, Efficiency: 95.45, tpmC: 1684437.21
   _elapsed___ops/sec(cum)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms)
   1801.1s      2824.0     302.0   1140.9   2415.9   9126.8  55834.6 delivery
   1801.1s     28074.0     402.7   1409.3   2684.4   9126.8  45097.2 newOrder
   1801.1s      2826.0       6.8     62.9    125.8   4160.7  33286.0 orderStatus
   1801.1s     28237.4     251.7   1006.6   2415.9  15032.4 103079.2 payment
   1801.1s      2823.5      39.8    469.8    906.0   5905.6  38654.7 stockLevel
   ```

## See also

* <InternalLink path="performance">Performance Overview</InternalLink>
* Hardware

  CockroachDB works well on commodity hardware in public cloud, private cloud, on-prem, and hybrid environments. For hardware recommendations, see our <InternalLink path="recommended-production-settings#hardware">Production Checklist</InternalLink>.
* Performance tuning

  For guidance on tuning a real workload's performance, see <InternalLink path="performance-best-practices-overview">SQL Best Practices</InternalLink>, and for guidance on techniques to minimize network latency in multi-region or global clusters, see <InternalLink path="multiregion-overview">Multi-Region Capabilities Overview</InternalLink>.