Form follows function

Even though Form3 wanted to run completely in the cloud, they knew that they had to have a combination of their own data centers and public clouds in order to connect with certain payment schemes. After choosing to migrate their FPS access solution to CockroachDB, they began to think about a different deployment methodology for their new platform that would satisfy regulatory requirements for operational resilience. They needed a second cloud.

They thought they could have replicated the entire platform onto GCP, which has a matching set of technologies to AWS (i.e., instead of SQS they use PubSub). The problem with this approach is that they would be managing two versions of their platform that have different behaviors. This was fairly tough to do and would create a lot of maintenance overtime.

Instead, they decided to run the whole database across all three cloud providers: AWS, GCP, and Azure. They use each vendor’s managed Kubernetes control plane offering, so that the vendor runs Kubernetes. This leaves Form3’s engineers free to concentrate on the value-add for the business:running their software and configuring CockroachDB on top of it.

For their setup, their data has a replication factor of three. Each range is replicated across each cloud and they are able to maintain strong consistency on each range. Here’s what it looks like:

With this deployment model in place, they are able to survive a full cloud provider outage. If a cloud goes down, they still are able to process payments with the same SLAs. They have pretty strict performance requirements when it comes to real-time payment processing with the SLAs at 100 and 200 ms at a P99 latency.

The SLAs also require that they can process different types of requests in a very short period of time. They have an incredibly high number of concurrent requests – up to 700 TPS – coming into the system at the same time. The requests are transient and involve cross cloud communication.

Using CockroachDB gives them the ability to configure different topologies based on where customers are located. For example, if you have a customer in London, you set it up so CockroachDB fetches data from the closest node to their London location and you get great performance. If that region goes down, though, there’s still a replicated and up-to-date copy of that data in a different location and on a different cloud provider.

Form3’s multi-cloud deployment is advanced. They had to privately network all the clouds together and maintain network connectivity through multiple datacenters. Networking across clusters is not simple, but CockroachDB’s operational simplicity can make it easier. By leveraging VPC peering and assigning Kubernetes pods their own IP address, Form3 can point CockroachDB nodes at one another and the pods will end up talking to each other from different clusters.

When you are federating Kubernetes clusters, if a pod goes down it typically takes other pods down with it. However, CockroachDB allows you to federate pods at the data layer. This means the clusters are running independently and so don’t really “know” about each other. Because they are isolated in this way, they won’t take each other down. This helps mitigate a great deal of risk.

Perfect your form

When it came to migrating from PostgreSQL to CockroachDB, it was a “fairly easy process” for the Form3 team. Since CockroachDB is PostgreSQL wire-compatible, Form3’s data structures and queries just worked — only a few needed tweaking. Another factor that helped streamline the migration: Most of the Form3 platform is written in Go. A majority of existing PostgreSQL tools also work with CockroachDB, so the team could leverage the Go drivers they were already familiar with. Finally, CockroachDB also has a Kubernetes Helm chart available, packaged up nicely for an easy install.

The Form3 team recommends setting up a good test suite and running end-to-end tests that include the database — they even tried dropping the database connection to ensure that the software handleds the failure correctly. Finally, before completing the migration, they tested latency to make sure that their payment journeys were rock solid and met all SLAs.

The Form3 team also highly recommends leveraging managed Kubernetes; otherwise, you might end up needing an army of experts to run and maintain the platform. There’s no competitive advantage to managing it yourself. If Kubernetes goes down, Form3 has enterprise support with the cloud vendor so they can get someone to help. And, since they are using CockroachDB, they are not locked into a single provider, which does give them a competitive edge.

Now that the migration is complete, Form3 continues to improve its platform and engages in a practice called chaos engineering. They use a tool called Chaos Monkey to simulate the failures. They have scenarios that create cluster failures, cloud outages, and disaster recovery failures to see how the platform responds.