Finding a new destination for reservations 

Back when the company was founded almost 30 years ago, there were not a lot of database options available and they ended up adopting MySQL as their primary data store. As the business began to scale and the data set grew, they started facing a lot of scaling challenges. They also had a couple of massive failures that impacted business continuity. 

In their effort to deliver the “Connected Trip” they decided to build a centralized system for all of the reservations data instead of the siloed systems. The new system, called Order Platform, was designed on Cassandra. This was primarily because of the lessons they learned over the years with managing large scale sharded MySQL that often lead to complicating the application layers as well as downtimes during mater failovers.

As time went on, they started to face several issues with the NoSQL solution. Reservation data is transactional by nature and Cassandra doesn’t have proper support for ACID transactions. This forced the team to use highly denormalized data models to guarantee atomicity. However, this wasn’t enough because Cassandra provides only row level isolation which is not suitable for multi-step read-modify-write across multiple rows/partitions in a single operation. 

The team implemented a locking system for such cases that relied on lightweight transactions to overcome this limitation, but they soon learned that this wasn't the right solution since sometimes they would timeout (just because of the way it works). It was hard to know if the timeouts were actually because of a legitimate failure due to unfulfilled conditions in the query, or because of another underlying issue. 

Also, reservations data was in high demand across the business as it powers tons of use cases. This includes front-end use cases that serve travellers and partners which require fast retrieval, as well as accounting and analytics which require streaming of data. Due to limited support of secondary indexes, they started using OpenSearch to help with this issue. However, that quickly increased operational and maintenance costs quite significantly because of the need to  replicate massive datasets across multiple databases, scale them, and guarantee that they stay in sync in near real-time. 

Overall, this order management system was becoming too complex and creating cognitive overload, so they started looking at the database landscape for a new solution. The team knew there would be no silver bullet, but wanted less operational overhead without having to compromise on the capabilities that they provide to users. 

In summary, their requirements for a new solution included: 

• Strong data consistency with ACID guarantees

• Change Data Capture (CDC)

• Support for secondary indexes

• Managed solution to reduce operations

• Compatibility with existing ecosystems of monitoring, security and access management

• SOX compatible and PCI compliant

The team selected three databases that they thought would fit including CockroachDB. During the evaluation process they tested how well these databases could scale, and how easy they were to operate. After completing a proof of concept (POC), CockroachDB was selected as the new solution for Order Platform.