CASE STUDY

Gaming

The ultimate battle for consistency: concurrent betting at massive scale

How Ultimate Tournament built a turnkey eSports betting platform from scratch on CockroachDB

  • 4 engineers

  • 10,000+ community members

  • 23 games and 5K matches per month

  • Expansive affiliate network

About Ultimate Tournament

Over the past several years, there’s been an explosion in popularity for a new type of real money betting: Electronic Sports, better known as eSports. eSports betting is the concept of betting on digital athletes who play competitive video games. And it’s becoming a very profitable industry; the global eSports market generated $14B in revenue in 2020 and is projected to grow substantially and exceed $205B by 2027.

As with any type of gambling, eSports companies face several challenges associated with navigating a complex set of regulations when trying to distribute their platform to users wherever they live. And, if they want to attract and retain a loyal customer base, they need to introduce a frictionless betting experience for their users that guarantees accurate payouts.

Dan Goodman (CTO and Co-Founder) and his team behind Ultimate Tournament are placing their bet on the eSports industry and building a platform that overcomes challenges. They are on a mission to demystify eSports betting and help more operators, casinos, bettors and players get involved in the action. Dan and team have built eSportsbook™️ which is a white-labeled platform that allows commercial gambling operators the option to rebrand Ultimate Tournament’s turnkey solution as their own and offer eSports betting to their customers. 

As they set out to architect and create their eSports platform, Dan and team knew that they needed to build on the right technologies from the start. They also knew that they needed to deliver a product to market quickly without compromising the integrity of the solution that would be handling real money. Ultimately they choose CockroachDB as their system of record and their single source of truth for all data.

“When building an application from scratch, it’s important to use the right tool for the job. Take a good look at your data and really understand it – how it's constructed, what the queries look like, where it's located. We’ve been really impressed with how CockroachDB’s consistency model handles our data. It delivers single digit millisecond operations [for both reads and writes] on pretty much every synchronous interaction that a user has with our system.” - Dan Goodman, CTO and Co-Founder

Requirements for building a eSports platform

Building from the ground up is a large undertaking and the architecture of the system is critical to get right as it will help you avoid long term technical debt that may cause significant headaches down the road. The Ultimate Tournament team wanted an affordable, managed solution for their backend that would allow them to easily scale in the future. This also wanted to alleviate some of the operational burdens of the backend, so that their relatively small team could focus on building the right app and frontend UX that would excite customers. 

When it came to requirements for a backend database solution, consistency was non-negotiable; meaning guaranteed correct, ACID transactions were a must in order to achieve idempotency (a critical attribute of a payment system). They also needed their data to be consistent at scale even when collecting a lot of high-cardinality data. A traditional relational database system would not be able to handle their projected volume of data. 

They plan to work with large casinos that have millions of users and they need the ability to facilitate enormous amounts of concurrent bets in real time. For this heavy read and write data, they wanted to develop a centralized data store that could be their single source of truth. This data repository had to be able to scale horizontally and ensure high availability via a fault tolerant system. The risk of having any downtime or outages at all was too high since it would completely shut off their direct revenue stream.  

Finally, they needed the option to scale their solution across multiple regions and multiple clouds. This would enable them to control where data is located which is crucial when it comes to meeting data privacy regulations for a global user base. Distributing your application across multiple regions also helps with achieving high availability. 

To summarize, their requirements for a database solution included:

  • Managed, affordable solution for a startup * Guaranteed consistent ACID transactions at scale * Built in high availability/fault tolerance  * Ability to control where data resides * Option to deploy across multiple regions & multiple clouds

    Given these requirements, Cassandra and DynamoDB were eliminated from the running since there’s a lot of complexity involved with guaranteeing consistent transactions. Many of these essential elements align with what a distributed SQL database can offer. When searching for a distributed solution, the team considered Amazon Aurora, but the single write node introduced too much risk into the architecture. They also considered Google Spanner, however, they didn’t want to be locked into a single cloud provider which would limit their ability to service every customer and expand into certain regions in the future. 

    Next they came across CockroachDB which met all of their requirements. After testing on the free open-source version, CockroachDB Core, they found that it added a lot of value from a development perspective.

“Consistency isn’t even a question; it's an inferred requirement. We can’t afford to make any compromises meaning we can’t duplicate or lose money. The fact that CockroachDB has ACID transactions built into the system allows us to achieve idempotency which is a huge advantage when building a distributed system that handles real money.” - Dan Goodman, CTO and Co-Founder

The odds are in your favor with CockroachDB

Once they were ready to take advantage of a managed service offering which includes additional features like change data capture (CDC) and the ability to partition data by location, the team upgraded to CockroachDB Dedicated. With this plan, they can start small and scale really fast as they onboard new customers. Dan and his team remarked how impressed they have been with the guidance and support from the Cockroach Labs team.

“The team at Cockroach Labs treats us as if we were at Amazon and giving them $1 million bucks a month. We talk to the people that help us solve things, move forward, have suggestions about what we should do. It’s proven to be extremely valuable and allows us to focus on building new features.” - Dan Goodman, CTO and Co-Founder

CockroachDB is being used as a system of record for all of their data which includes payments, (bets, account values, etc), pools, and gamification features. They also use TimescaleDB for managing their time-series data and Redis for processing real-time odds and values. Ultimately all of their data makes its way into CockroachDB which serves as their single source of truth.

They are building on distributed microservices-based systems which gives them a lot of flexibility when it comes to deployment. At the moment, they are on a single cluster with 2 core nodes in one AWS region. However, they are designing their infrastructure with the notion that they will expand into multiple regions and into Europe as the business grows. This is when setting locality to domicile data close to users will become extremely important in order to comply with GDPR and additional government regulatory requirements specifically for the gambling industry. Each country within the EU has its own rules for how they organize their gambling services which makes data locality essential. 

Their developers are also using the free tier of CockroachDB Serverless for testing and development. They set up a local dev environment that’s connected to a serverless cluster. This allows their developers to go through quick dev cycles, push out new features, collaborate on the same data, and reduce iteration time – at zero cost and zero liability of breaking anything. 

Here’s how it works: a user enters a bet into the system which asks CockroachDB if the pool is open (yes/no). If the pool is open, the bet is inserted and asks CockroachDB if there’s enough money in the user’s account to place the bet (yes/no). If there is enough money, the value is detracted from the user’s account and the bet is successfully placed.

The bet becomes an event which travels downstream in the form of an event source pipeline. (Also known as CDC which is included with CockroachDB – many other systems require you to manually build an event source pipeline). Once confirmed, the betting API talks to Redis to update the real time value of the entire pool and new odds. It also talks to TimescaleDB and fetches the time-series data which is stored as a record in CockroachDB. 

For this system to work, CockroachDB’s CDC capabilities are extremely important since it creates an audit log of every transaction (bet slip, pool, user account value) and constantly updates it to Amazon S3. This is crucial for two reasons 1) they can replay every transaction state in the event that they need to debug the system or trace money 2) if regulators ever need to see a report for fraud investigation, they can go back and show every transaction.

“We keep everything downstream from CockroachDB which is our single source of truth. It’s the most resilient, distributed, and highly available system. Everything else comes as a sort of `quality of life` feature that we get downstream. Our time-series doesn't have to work for our system to behave properly. Our Redis can vanish and operationally we would still be OK because we would just fall back to CockroachDB.” - Dan Goodman, CTO and Co-Founder

Bet on a distributed environment and know your data

Dan’s best piece of advice for people who are new to distributed systems: understand your data and how it interacts with the system your data lives on. For Ultimate Tournament, they knew they would be primarily dealing with two types of data: real-time transactional data and time-series data. 

Both of these data types can create high cardinality which refers to columns with values that are very uncommon or unique. For example, any time a bet is placed for a particular pool it has an associated USER_ID and SLIP_ID which is unique to that specific bet slip. Now imagine 100 different people with unique USER_ID and SLIP_ID are placing bets on that same pool at the same time. This data set starts to grow exponentially and the more unique values it has, the higher the cardinality. If you are not using the right solution and properly organizing your data, high cardinality data can damage insert rates and query performance.

Ultimate Tournament’s transactional data includes payments, bets, pools, user IDs, etc. that needs to be consistently updated (placed, canceled, processed) in real-time. Right now, they provide betting on 5,000 matches a month. This cardinality is low; a relational database can easily handle that volume. However, when they open up their platform to casinos with millions of users, it presents the challenge of accommodating a large influx of concurrent transactions at once, in real-time. They also need to offer the ability to manipulate bets (place, cancel, modify) as many times as the users want, and of course all this data has to be accurate. Most databases simply can’t handle this complexity which is why they use CockroachDB for transactional workloads. 

They also have a lot of time-series data which can become high cardinality data since it tracks a series of data points collected over time intervals (i.e. it creates a large number of unique values in a set). That’s why Ultimate Tournament is using TimescaleDB (a relational database with support for time-series) to help optimize their time-series workloads which is important to keep cardinality low. 

In order to work with their data, Dan created two development rules: 
1) no full table scans
2) no hot keys and hot rows (rows that are frequently added, deleted, modified, or queried)

Dan needed to make sure they were organizing data in CockroachDB the right way to reduce the risk of high cardinality data slowing the system. At the same time, he wanted to optimize performance. They designed their data so they can read it efficiently which meant intelligently designing primary keys upfront so they can get multiple access patterns out of the same primary key.  

This also meant designing indexesfor specific columns, which creates a copy of the columns and then sorts their values (without sorting the values in the table itself). After a column is indexed, they can easily filter its values using the index instead of scanning each row one-by-one. On large tables, this greatly reduces the number of rows scanned (i.e. no full table scans) which enables them to execute queries exponentially faster. 

Now, they can go one step further and create multiple indexes of the same table and pin the indexes to different regions. This feature helps avoid performance bottlenecks. For example, say they are constantly going back to the same pool to make sure it’s still open; this can get really “hot”. Instead, they have multiple indexes of the same table in different regions. So they aren’t jumping to access a certain node, they can access any node because there is consistent copy of that data on every node (thanks to ranges which are part of CockroachDB’s distribution layer). This eliminates the hotkey problem. 

As you can tell, the data that’s generated from gambling applications is complex.

What’s Next

The team describes their app as the “Robinhood of eSports betting” since it allows eSports betting to be intuitive for everyone, and their platform as the “Shopify of eSports betting” since companies can white label the platform and make it their own. They have two main focuses right now which include getting more gamers to join their community and piloting eSportsbook™️ to a select group of testers. 

The Ultimate Tournament Community is already 10,000+ members strong. Gamers can sign up to be part of the community and enter to play in weekly tournaments that have prizes ranging from $100-$2,000. 

The early access to eSportsbook™️ lets users register, customize accounts, and place fake money bets on real matches. These testers are providing Ultimate Tournament with the necessary KPIs, feedback, and stress tests needed to squash bugs and optimize their platform. 

In early 2022, they plan to open eSportsbook™️ up to the public. They hope to have several operators white labeling their platform within the next year. Along the way, they will be relying on CockroachDB to grow with them as their business scales. Dan and his team feel confident that they will be able to build “the best eSports betting platform on the face of the earth”. 

Let the games begin.

“We only have one really good shot at this because the [gambling] space is extremely difficult for startups. Getting it right up front is not just a question of end user experience, but a question of the business surviving. Making sure that we have consistency guarantees as well as high availability up front were non-negotiables. When you start lining up the requirement boxes next to the number of databases that can check them, it starts shrinking fast. For us, all the arrows were pointing to CockroachDB.” - Dan Goodman, CTO and Co-Founder

Ready To Get Started?