How to build a multi-region betting application (with reference architecture)

How to build a multi-region betting application (with reference architecture)

The popularity of sports betting, also called real-money gaming, is exploding.

But with demanding users putting real money on the line and a complex regulatory environment, actually building a performant betting application can present a challenge — especially if the architecture isn’t carefully thought out.

Technical requirements for a modern sports betting app

Architecting a betting application presents many of the same requirements that are present with any sort of transactional application. Among them:

Elastic scalability

Betting applications have to be capable of scaling up to handle huge bursts during major sporting events, and then scale back down during the lulls for maximum efficiency. Scale up too slowly and the user experience will be impacted. Scale down too slowly and you’ll likely be paying for more resources than you actually need.

Generally, this means embracing a distributed, event-driven approach, often based in Kubernetes. This will allow for fast, smooth horizontal scaling up and down, although fine tuning that scaling to maximize for both user experience and resource efficiency is likely to take some work.

Consistency

In any transactional application involving money or where timing matters, consistency is a table-stakes requirement. And for fast-paced use cases such as sports betting, eventual consistency generally isn’t going to cut it. Every element of your system has to be certain of exactly what bets were placed and exactly when they were placed; there can be no disagreement.

This means the ideal database for a sports betting application’s system of record is a relational database that can offer ACID guarantees, and ideally serializable isolation. However, choosing a relational database presents its own set of challenges. We’ll dig more into this problem later, but the short-answer solution is to use a distributed SQL database such as CockroachDB, which was built for use in cloud-native, distributed transactional applications.

Resilience

There is no type of user base or application that likes downtime. But going down at the wrong time can be particularly devastating for a sports betting application — imagine having your app go offline during the World Cup final! The financial implications of having an outage at the wrong time will be familiar to anyone who’s worked in the retail space and had to imagine — or experience — the implications of an outage on Black Friday.

This requirement, too, suggests the need for distributed application architecture so that there is no single fail point, and one node going down cannot take the application offline. At sufficient scale, it may even make sense to architect the application to survive AZ or even zone failures.

Latency

Almost any kind of application typically aims for low latency because it provides a better user experience, but latency can be particularly important in the realm of sports betting. Some types of sports betting, such as micro betting or live betting, can play out in the span of just a few seconds, so any sort of delay is likely to leave users furious.

This means architecting the application carefully to minimize bottlenecks, and potentially taking advantage of features such as UDFs to minimize the distance data has to travel.

Minimizing latency typically also requires localizing your data, which is a topic worthy of particular focus in the context of sports betting.

Locality

At a certain scale, almost any application will benefit from implementing a multi-region architecture, as it can put the user’s data closer to them, minimizing the impact of latency on their overall experience. But in the context of sports betting in the US, multi-region architecture is even more important because of The Wire Act, which prohibits businesses from using “a wire communication facility for the transmission in interstate or foreign commerce of bets or wagers or information assisting in the placing of bets or wagers on any sporting event or contest.”

Precisely how this 1961 law should be applied to online applications in 2022 is a question of interpretation for a company’s legal department. But for architects, it virtually always means building a multi-region application where specific services and data can be placed within (and restricted to) specific state boundaries.

Another challenge: Geolocation and the database layer

Achieving these requirements at any kind of scale generally means embracing cloud-based microservices architecture that allows you to spin up local instances of the relevant application services, both to improve latency and comply with regulatory requirements.

When it comes to the persistence layer, though, this kind of geolocation is not always straightforward.

Consider, for example, the challenge of using PostgreSQL as your system-of-record database for a US-based betting application. Because data cannot be geolocated within a table in PostgreSQL, you might create separate tables for bets in each state, which would result in the duplication of a lot of data. To ensure these remained available, you’d have to set up some kind of replication, likely an active-passive system. You would also have to write application logic to ensure that bets and other relevant data was routed to the correct table based on the user’s location.

This approach already contains a lot of complexity, as well as the potential for data loss if you’re using active-passive replication. And ultimately, particularly in larger states such as California, scaling it up would require sharding, which then requires additional customized application logic to route data correctly, and adds additional complexity to the challenges of replication. Perhaps needless to say, as you add additional states and shards, the complexity of this approach can quickly make it untenable.

three-state sports betting architecture with database sharding and an active-passive system

Above, we can see a relatively simple three-state implementation of this. Aside from the potential for data loss with active-passive replication, we can see that there’s already quite a bit of custom logic required to correctly route transactions from the application, and this code will have to be maintained and updated anytime the application is scaled up or down and anytime the company begins operations in a new state or country.

There are other potential approaches, of course. You might also choose to create separate PostgreSQL databases for each state. However, this would require duplicating all the global data (such as tables with the events available to bet on, odds, etc.) in each of these separate databases, which in turn introduces a plethora of potential consistency issues. This kind of architecture would also make user management and customer service more difficult. Users’ info would have to be stored in their state’s database, which could cause problems if they travel or move and attempt to use the application.

Perhaps it’s needless to say, but there’s a better approach!

Reference architecture

A lot of the architectural complexity required for the PostgreSQL approach detailed above can be simplified with better tooling. In this case, we’ll look at a version of the real-world architecture used by Hard Rock Digital. Hard Rock Digital uses CockroachDB, a PostgreSQL-compatible distributed SQL database, as its system of record because it makes geolocating data far simpler.

Here’s a look at Hard Rock’s setup from a geographic perspective (simplified for readability; in actuality it operates in different states and has different node counts than what is pictured here):

a two-state example of a multi-region betting application and database deployment

In the setup above, there are separate application instances and database nodes for two different states, Texas and Colorado. Data from both states is also backed up to database nodes in AWS’s us-east-1 region.

The advantages of this setup over the previous one are numerous:

  • CockroachDB allows you to keep a single copy of global tables (market data like events, odds, etc.), rather than having to duplicate this data repeatedly across separate databases for each state.
  • CockroachDB is a multi-active system, allowing for very high availability with no risk of data loss or consistency issues.
  • CockroachDB allows for regional-by-row geolocation, and geolocation rules for data can be defined using simple DDL statements. This means that instead of needing complex application logic to route data by region, you can have a single bets table with a location column and the database will locate each row’s data according to the value in that column, automatically.
  • CockroachDB was built from the ground up for distributed cloud applications, and it automatically handles all of the routing associated with sharding internally. This means no application logic to add or maintain when you need to scale up or down. You can simply add or remove database nodes as needed, and your application can treat the database as though it were a single instance of PostgreSQL. The database will distribute your data efficiently between the nodes automatically, in accordance with the geolocation rules you’ve given it.

Below, we can see an example of how this approach looks in terms of application architecture. Note that no routing logic is needed: the application treats CockroachDB as a single logical database, and all data is geolocated and replicated according to rules you’ve outlined using simple DDL statements.

sports betting gambling reference architecture

Data stored in CockroachDB can be easily synchronized to other services using the built-in change data capture (CDC) feature, and even transformed if needed. Hard Rock, for example, uses CDC to push data into Confluent Kafka, and from there into Snowflake.

Learn more about how to build highly resilient and performant real-money gaming applications with our free guide to Data Architecture for Gambling Applications.

About the authors

Charlie Custer

Charlie is a former teacher, tech journalist, and filmmaker who’s now combined those three professions into writing and making videos about databases and application development (and occasionally messing with NLP and Python to create weird things in his spare time).

github link linkedin link

Keep Reading

How Hard Rock Digital built a highly available and compliant sports betting app

In 2020, Hard Rock International (HRI) and Seminole Gaming (SGA) launched Hard Rock Digital and tasked that organization …

Read More
How to build modern gaming services — with reference architecture

Let game developers develop games. It doesn’t exactly sound revolutionary.

But back in the day, that’s not always how …

Read More
x
Developer Resources