[For CockroachDB's most up-to-date performance benchmarks, please read our Performance Overview page]
Back in April of 2018, we ran extensive tests comparing CockroachDB to Amazon Aurora in terms of TPC-C throughput—the industry standard for testing real-world transactional throughput (you can check out those findings here).
No matter how crucial a component’s performance is, though, teams must also consider the price of those numbers––an app that consumes more financial resources than it generates is difficult to maintain.
To better understand the financial impact when choosing between software, we decided to also determine the price of running TPC-C against both CockroachDB and Aurora. As the headline says, we found it costs between 5-10 times more to choose Aurora over CockroachDB.
A Quick Refresher on TPC-C
We have much more detail about TPC-C in our performance guide if you’re interested, but here’s a rundown of what you need to know to understand the rest of this post.
TPC-C is the industry standard for establishing a consistent point of comparison among different databases’ performance in an online transaction processing (OLTP) workload. The benchmark heavily relies on a robust set of SQL features such as foreign keys, multiple indexes, transaction rollbacks, and joins.
Because of its volume, TPC-C also stresses the underlying storage capabilities of the database, which is crucial to demonstrating how it performs under real load.
Data Layout
TPC-C models nine tables that one might create to manage inventory for an online store, tracking elements such as SKUs, inventory, customers, and orders. TPC-C also relies heavily on the notion of a “warehouse,” which is meant to simulate a distinct physical warehouse where the retailer would store goods. Warehouses, though, are not explicitly defined in the TPC-C schema. They are instead represented through a column in the stock table.
Scaling
To scale the benchmark, TPC-C lets you create more warehouses, each with 80MB of data (e.g. 100 warehouses would be 8GB).
The benchmark then also linearly scales all other variables with the number of warehouses. An example is 12.8 transactions per minute per warehouse. In the example of 100 warehouses, you cannot exceed 1280 transactions/min.
Comparing Price
While benchmarks primarily focus on reporting throughput, it’s also crucial to understand the financial assets that are required to achieve those numbers. If it’s only possible to reproduce someone’s benchmark results on a fleet of 32-core machines, that’s important to know because it impacts a project’s feasibility.
Further, because TPC-C places an upper bound on the number of transactions any service can complete on a given number of warehouses, a cost-based metric lets you meaningfully differentiate between software with similar TPC-C throughput.
To actually compare the price, then, simply divide the total cost of the test by the benchmark’s tpmC number, giving you a “Dollars per tpmC” metric.
CockroachDB vs. Amazon Aurora RDS
In our blog post detailing the results of CockroachDB vs. Aurora’s TPC-C throughput on 1,000 TPC-C warehouses, the two databases achieved similar results. Given that, we thought it would be interesting to compare the price of the two services.
To make the evaluation mirror reality as closely as possible, we chose to price both services using three-year contracts with their respective partners. With this length of commitment, infrastructure providers offer aggressive discounts which, in our experience, are attractive enough that many users are eager take them.
| Database | CockroachDB 2.0 | Amazon Aurora RDS MySQL |
|---|---|---|
| Throughput | 12,475 tpmC | 12,582 tpmC |
| 3-Year Cost | $49,670 | $260,670 – $540,6051* |
| $/tpmC | $3.98 | $20.72 – $42.97* |
| Hardware | 3x Google Compute Engine n1- highcpu-16 with attached Local SSDs | 2x r3.8XL (One read replica with automatic failover) |
* Aurora pricing is shown as a range because Amazon does not reveal the number of I/O per second (IOPS) Aurora requires to achieve their TPC-C results, so we made two estimates. We set an upper-bound at 30k IOPS, which is the amount pre-provisioned for MySQL on a similar workload. Knowing that Aurora can be more efficient than MySQL, we then estimated a lower-bound of 10k IOPS.
Comparison
CockroachDB matches over 99% of Aurora’s throughput, and has only 10 to 20% of the price. We think it’s safe to say that on the TPC-C 1k warehouse benchmark, CockroachDB is a much more attractive option.
CockroachDB vs. Aurora tpmC
CockroachDB vs. Aurora Dollars per tpmC
Details & Differentiators
- Because Aurora is a managed service and CockroachDB is open source software, our total costs account for a DBA with a $200k/yr salary to spend 7 hours per month managing a CockroachDB cluster.
- CockroachDB was tested on Google Cloud because we found their hardware to be more price efficient.
- As mentioned above, we had to estimate that Aurora achieved their tpmC number using somewhere between 10k-30k IOPS.
More Detail?
If you want more detail about these numbers, as well as how to reproduce CockroachDB’s TPC-C throughput, download our guide, CockroachDB vs. Amazon Aurora Cost Comparison.
If you’re interested in how much CockroachDB would cost for your team’s specific scenario, get in touch with us.
Illustration by Zoë van Dijk
