Benchmarks
& beyond

Which cloud provider prevails across the key benchmarks today’s cloud applications care about most?


In the 2022 Cloud Report, we sought to put performance in real-world context—to provide practical insights, not just pure numbers.


5 key insights from the 2022 Cloud Report

Insight 1

AMD Milan chipsets set the pace for CPU benchmarks

Machines with AMD Milan (EPYC Gen 3) processors claimed the top spots in both CPU benchmarking and OLTP testing for both large and small instance types.

In past years, we saw Intel lead the pack in overall performance, with AMD competing on price-for-performance metrics. This year, both the overall performance leader and the price-for-performance leader were AMD-based instances.

Download the report for more CPU benchmarks
Tweet Tweet

Insight 2

All three clouds have price-competitive offerings

$0.04-$0.05

There is at least one instance type and storage combination in the 4-5 cent reserved $/TPM range for all three clouds.

Download the report for more cost comparisons
Tweet

Insight 3

You get what you pay for, but not always what you don’t

When we paid explicitly for a resource we got the promised performance out of that resource.

However, in cases where the cloud provider advertised a performance range (e.g. “up to 10 Gbps”) and did not explicitly charge for a specific level of performance, the results varied between cloud providers and between runs.

We saw a wide range in performance for non-network optimized instance types, including some apparent throttling.

Download the report for more speed tests
Tweet

Insight 4

Storage & transfer costs have an outsized impact on total cost to operate

High-perfomance
storage

69.68%
69.02%
68.97%

General purpose
storage

32.87%
35.41%
29.42%

For even relatively small amounts of persistent block storage, the cost of running a particular workload is much more influenced by the cost of the storage than it is the cost of the instance.

For persistent workloads, it is extremely important to optimize cost calculations based on this consideration.

This chart shows average storage costs as a percentage of the total instance costs (including storage).

Download the report for more cost analysis
Tweet

Insight 5

vCPU to RAM ratio directly impacts performance consistency

TPM per vCPU

Warehouses per vCPU (workload complexity)

TPM per vCPU

Warehouses per vCPU (workload complexity)

Our tests suggest that while you may save a bit of money by choosing instance types with a lower vCPU to RAM ratio, you will likely see more consistent performance from instance types with more available memory. The impact of this decision is more apparent with larger, more complicated workloads.

In our tests, we found the sweet spot to be a vCPU:RAM ratio of 1:4.

Download the report for more best practices
Tweet

What’s new in 2022?

  • Improved test design and configuration for all tests
  • Testing for different-sized instance types
  • Testing cross-region latency
  • Storage tests with fsync
  • Variable OLTP workload complexity testing
  • 12X longer throughput test duration

Dig deeper

Our annual cloud report is an open-source project, and you’re welcome to run the benchmarks yourself. Hop over to GitHub to get started. If you have questions, come say “hello” in our community Slack channel or watch tutorials via our YouTube channel.

70+ pages. 100% free.

The 2022 Cloud Report was a massive undertaking we're proud to share with the community free of charge. Simply fill out the form below to get your copy.