In 2018, a team of CockroachDB engineers was testing features for the 2.1 release when they discovered something surprising. In repeated tests, AWS delivered 40% greater throughput than GCP (Google Cloud Platform). The team’s curiosity was piqued. Why did they see such a stark difference between the two cloud providers? They set out to test the performance of GCP and AWS in more detail, ultimately comparing the two platforms across a host of benchmarks including CPU, network, storage I/O, and TPC-C performance (e.g., throughput and latency).
This unexpected discovery in throughput inspired what would become the annual Cloud Report from Cockroach Labs, an evaluation of machine performance across AWS, Azure, and GCP.
The Cloud Report continues to offer surprises, and the 2021 edition was no exception. When we ran the CPU benchmarks, we made something of an accidental discovery about performance of the processors: Intel, AMD, and the new-to-market Arm-based Graviton2 from AWS.
A little background: For the 2021 Cloud Report, we evaluated each cloud’s CPU performance using the CoreMark version 1.0 benchmark. Designed for CPU benchmarking, CoreMark produces a single-number score — the higher the score, the better the performance.
We arrived at CoreMark because it is open source, cloud-agnostic, and more representative of general CPU performance than stress-ng, which we had used in previous reports. For example, CoreMark tests against various real-world workloads, like list, sort, and search.
CoreMark can be limited to run on a single vCPU, or it can execute workloads on multiple vCPUs in parallel. We ran CoreMark in both modes ten times and reported the average number of iterations per second for both the single-core and 16-core results.
The CPU benchmark looked at how each cloud provider (AWS, Azure, and GCP) performs on single-core and 16-core CPU benchmarks. When we tested dozens of machines across the three clouds, it became apparent that what’s inside each machine matters. Intel swept the board in single-core processing, but when more processing units were added, it faltered.
For the single-core CoreMark runs, all the winning machines ran Intel processors.
When we looked at the 16-core CoreMark performance, none of the winning machines ran Intel processors. Instead, the AWS custom-built Graviton2 Processor, which uses a 64-bit ARM architecture, came in just ahead of GCP and Azure’s winning machines — both of which ran AMD processors.
This is bad news for Intel, as the industry gravitates toward the idea that more is better. While single-core performance is important for some types of applications, complex applications — particularly cloud based ones — benefit from multi-core CPUs. This is especially valuable as applications become more feature-rich, which creates added demand. Combined with the fact that AMD (and ARM) CPUs use less power, it’s not surprising that major cloud vendors invest significantly into their AMD offerings.
We found that CoreMark performance on a single core served as an effective indicator for raw throughput performance. Here GCP led the way, achieving the highest CoreMark score by 10% over the best performing compute-optimized machines in AWS and Azure.
As a less indicative metric of raw throughput than single performance, 16-core CoreMark highlighted each machine’s ability to scale to multiple cores. Amazon’s Graviton2 led the way here.
The AWS Graviton2 Processor, which runs on ARM, enabled the top-performing m6g.4xlarge machine to achieve better multi-CPU scaling than the other machines. AWS’s m6g.4xlarge hit 308,852 iterations per second, while GCP and Azure’s machines produced 5% and 7% lower scores, respectively.
Note: Where AWS machines demonstrated minimal variability, results have been grouped by type and processor for improved readability.
While machines with Intel processors outperformed the others on single-core CPU performance, with the notable exception of the c5a series running AMD EPYC 7002 processors, AWS machines with Graviton2 processors consistently outperformed all other AWS machines on 16-core (and, indeed, all other machines in the field). These machines are also AWS' cheapest.
Like the single-core benchmark on AWS, Azure machines with AMD processors performed on par with Azure machines with Intel processors. Consistent with both AWS and GCP, Azure machines with AMD processors outperformed Azure machines with Intel processors on the 16-core benchmark.
Notably, the memory-optimized Standard_M16ms machine and the storage-optimized Standard_L16s_v2 machine were the bottom performers among Azure’s machines on both metrics.
GCP’s compute-optimized machine (c2-standard) demonstrated the best raw CPU performance among GCP machines. Beyond the c2-standard machine, there was minimal variation in the single-core benchmark among machines tested, because the N2 machines have the same processors and only differ in their vCPU-to-memory ratios.
The same applies to the N2D machines. On the 16-core benchmark, the N2D machines with AMD EPYC 7002 Processors and the c2-standard-16 machines led all other GCP machines.
If you are interested in reading the full results of our benchmarking — including how each cloud performed in CPU, Network, Storage I/O and OLTP benchmarks — download the 2021 Cloud Report.