[CASE STUDY]

How FanDuel scaled their financial ledger with CockroachDB

independent ledgers

28

scale

4x

RPO

0

Founded in 2009, FanDuel is considered to be the driving force behind a now multi-billion dollar industry. However, the developer of America’s #1 sportsbook is much more than a sports betting company. In fact, they deliver a whole suite of product offerings to keep customers “part of the action from the comfort of their own home” including FanDuel Casino, FanDuel Fantasy, FanDuel TV, and FanDuel Racing.

Today, the company is at the top of the leaderboard in the U.S. for their Sportsbook and Casino offerings. They serve over 13M active players across 28 states where sports betting is legalized, and 5 states where they are allowed to operate their online casino.

To stay ahead of the competition and remain the number one seed, FanDuel must have a rock-solid infrastructure to support their business momentum and future growth. That’s why they modernized their financial ledger system and migrated to CockroachDB.

[ INDUSTRY ]

Online gambling, sports betting, mobile gaming

[ CHALLENGES ]

Keeping pace with millions of customers who expect always-on access and fast performance.

[ SOLUTION ]

A future-proof, resilient, scalable financial ledger built on CockroachDB.

Placing a bet on distributed SQL

All FanDuel applications have a shared wallet which requires a unified ledger system to track all customer financial transactions. The wallet also allows customers to monitor and use their funds across various product lines.

Gambling is a heavily regulated industry in the U.S. and operators must comply with The Wire Act, meaning that all bets must be settled in the state where they were originally placed. This mandate creates a lot of complications for sports betting companies who want to do business in as many states as possible without having an overly complex infrastructure. In order to adhere to this regulation, FanDuel needs to have a ledger plus a database in each state where they do business.

For many years, FanDuel was using Percona and leveraging a MySQL Galera Cluster. As the company grew, this setup started to create some limitations. For example, it was virtually synchronous (i.e. slow) and had some significant disaster recovery constraints (i.e. manual) which created a high recovery time objective (RTO). It also only allowed for vertical scaling and management operations on huge tables required lots of planning.

It was time for FanDuel to migrate to a new solution that would allow them to keep pace with their growing customer base and seamlessly expand into new states as soon as it was legal.

FanDuel’s requirements for a new database included:

Ability to exceed current scale
Capabilities to aid with regulatory requirements
Simplified operations
High availability with better disaster recovery

Although they are an AWS customer, they recognized early on that Amazon Aurora would not be able to meet their requirements. They needed the ability to deploy in their customers’ locality/state and there’s not a managed service option that would meet high availability requirements for those deployments which is why self-hosting had to be an option. A few of the engineers had experience with CockroachDB, a cloud-neutral distributed SQL solution, and they decided to take a closer look.

Drafting a new team

On paper, CockroachDB met all of FanDuel’s requirements but they wanted to run a proof-of-concept (POC) to ensure it would work for their use cases. They also wanted to push the scaling limitations to see how the database would respond under load.

They benchmarked a 9-node CockroachDB cluster with 2x the scale of their baseline with Percona, and had no issues. They made a few adjustments to their cluster such as changing the primary keys for core tables and removing sequential transaction IDs. Then they increased to 4x the scale which was around 5K transactions per second. They started leveraging built-in features such as change data capture and row-level time to live (TTL), and also removed old indexes. After testing, the team had confidence that CockroachDB was the right technology to achieve FanDuel’s future scaling goals.

The team in charge of the ledger system reports that the process of migrating was “actually super simple.” They used AWS Database Migration Service (DMS) for migrating bulk loads of data in, which they said was “basically magic” and did all the heavy lifting for them.

However, they did need to figure out what kind of rollback mechanism they could perform if something went wrong during the migration, and they weren’t able to forward fix after the cutover version. They worked with the Cockroach Labs team on a solution ultimately using the MOLT toolkit which supports migrations from MySQL. By leveraging MOLT, they would have a rollback process in place in case they needed to use it.

During testing we started to adopt this mindset of not just making CockroachDB work for our ledger system, but also the other side of the coin of having our ledger system work for CockroachDB. We identified changes and modifications we needed to make to the system to make sure they work seamlessly together.

Jordan Leach

Software Engineering Director, FanDuel

The winning combination

Following the migration, CockroachDB became the new backend for their financial ledger system, which is responsible for tracking every financial transaction that happens between a customer and FanDuel, as well as customer balances. It’s a very critical system for the business and has to stay up at all times.

The ledger must be able to handle distributed and concurrent reads and writes in order to deliver a great customer experience, however the volume of transactions tends to vary. For example, Casino traffic is what FanDuel classifies as “marathon-style” meaning a single state might have a thousand transactions per second for their write workload that remains steady over a period of time. On the flip side, Sportsbook might have to handle a large-scale event like the Super Bowl or Kentucky Derby where they see more spread happen as they get closer to the actual time for the event as well as huge spikes in transactional volume.

Regardless of the event, they handle every deposit and withdrawal with care knowing that customers are going to frequently check their balances, bonuses, or promotions. Because CockroachDB guarantees data correctness, the system can accurately process and track customer data even during peak periods.

As mentioned above, FanDuel is required to run a ledger and a database(s) in each state they do business. That means their deployment spans across 28 sites and each site has up to three deployment locations. They leverage a combination of AWS Regional Data Centers, AWS Outposts, and AWS Local Zones. Below is an example of what the cluster looks like using three different sites within one locality (i.e. state).

The locations are spread between two different data centers (Outpost, Local Zone) as well as an AWS Region to offload backups (leveraging S3 storage) and operational tasks. The application query is proxied to CockroachDB nodes in that site. Temporary users are managed by HCP Vault; Vault roles are determined by Okta groups, and Vault creates a temporary CockroachDB user based on Vault roles and returns connection details. Finally the audit logs identify users and their actions within the application.

[For a full demonstration of their setup, watch FanDuel’s talk: Scaling to support America’s #1 Sportsbook]

A team sport

When it comes to getting started with CockroachDB, the FanDuel team says that you cannot assume that CockroachDB is going to perform like your existing solution. They report that testing was key to ensuring they were set up for success and would be able to accomplish the migration.

Overall they say it's made several operational tasks much easier especially when it comes to how they deploy and manage connecting databases, as well as restoring backups and making big schema changes. Additionally, they believe that the DB console has been beneficial to their engineers because they can go in and view active stats whenever they like. They can see things like how indexes are being used, how the system is performing, if there’s any hot spots, etc.

Last but not least, FanDuel reports that they’ve had a great experience with the Cockroach Labs team. They say that they will go online or submit a ticket and within minutes Cockroach Labs support is responding and even willing to jump on a quick call to engage with our engineers to troubleshoot and work through problems.

Testimonial

CockroachDB has made operational tasks much easier in terms of how we deploy, manage, and connect databases. Also how we do things like restore backups and big schema changes. It all happens online with minimal intervention. Our engineers have had great experiences interacting with the product and the Cockroach Labs support team. If we ever run into problems, we get an almost instant response which has been really fantastic.

Alex Chadwick

Staff Software Engineer, FanDuel

What’s next

At the time of publishing, FanDuel Sportsbook is available in 28 states (with the most recent launch in Missouri), and they are watching closely to see which state is legalized next.

Ultimately as soon as a new state legalizes, FanDuel will be able to quickly spin up in that region thanks to CockroachDB. From a financial perspective, they can be there from day one so they never miss a moment of the game.