Inventory management on Black Friday: challenges and solutions

Inventory management on Black Friday: challenges and solutions

For retailers both online and offline, Black Friday and Cyber Monday are often the busiest time of year.

But while the deluge of deals evokes dreams of dollars for CEOs, it can be a bit of a nightmare for CTOs. That’s because the onslaught of shoppers puts huge amounts of stress on their application architecture. And particularly in the realm of inventory management, even a minor problem can have major consequences. (DoorDash recently published a useful blog about how they navigate the challenge of real-time inventory level changes with changefeeds.)

How Black Friday stresses inventory management systems

On every day but particularly on Black Friday, backend inventory systems have to deliver on at least four things:

1. Scalability

Inventory management systems must be able to meet the increased flood of both reads and writes that will come in over the Black Friday period, as shoppers view items, add them to carts, and purchase them.

While companies can attempt to estimate the demand beforehand and provision hardware resources accordingly, it’s far better to be flexible. A particular product could get unexpectedly hot. An outage elsewhere could drive extra customers to your site. The inventory management system needs to be able to function in real-time regardless of load, which generally means that it needs to be architected in such a way that additional resources can be provisioned quickly and easily.

In practice, this generally means embracing a distributed, container-based cloud architecture so that new instances of application services can be quickly spun up whenever the load on the system increases.

2. High Availability

If any part of the inventory management system goes offline during Black Friday, the financial consequences could be massive.

One possibility, of course, is all of the lost sales that can occur if customers hit your product page and see a 404. But potentially even worse is the prospect of having some inventory management services go down while others are still working, potentially allowing customers to purchase products that aren’t in inventory.

The goal of retailers, on Black Friday as on all days, is to sell to zero but not beyond zero. If the wrong part of the system goes down, or simply gets bogged down, at the wrong time, selling beyond zero can become a possibility. Placating angry customers who’ve paid for products you don’t have can end up being even more costly than missing the sales!

3. Low latency

Inventory management services on the backend need to be able to answer queries quickly – shoppers are fickle and if one seller is making them wait, it’s likely they’ll find someone else.

For enterprises, this typically requires a multi-region architecture so that key data and service instances can be located close to the customers accessing them. Your UK-based customers, for example, will have a much snappier shopping experience if their page views and purchases are querying inventory management services located in the UK, rather than having to constantly ping a server in California.

4. Consistency

Every aspect of the application, from the source-of-record database to user-facing application services, has to agree on how much inventory exists for each product, and where that inventory is located. Any kind of inconsistency can lead to the dreaded selling beyond zero, or a variety of other problems that create poor customer experience, such as products still being listed as in stock on the product page when the database knows they’re actually sold out.

Achieving consistency in a small-scale system is easy: you spin up a Postgres database somewhere and have all of your services query that. But at scale, consistency becomes quite complicated. The previous three requirements in this list suggest that inventory management systems should be cloud-native, distributed multi-region systems. How can you keep a single-source-of-truth database when separate regions of your application are processing huge numbers of transactions all at the same time?

Black-Friday-ready inventory management architecture

While there’s no way of making Black Friday easy, architecting your inventory management system in the right way can make achieving scale, high availability, low latency, and consistency much more attainable.

Let’s look at the backend databases that are typically called by application services as a user goes through the steps of making a purchase:

Black-Friday-ready Inventory management architecture1

As we can see, a single purchase could potentially hit the inventory database at least three times:

  1. A read when the user loads the product page, to see if the item they’re looking at is in stock, and potentially also where it’s located.
  2. (Often) a write when the user adds the item to their cart to mark that the item isn’t currently available, although it is still in stock.
  3. A write when the item is purchased or abandoned to permanently update the database on its status.

Probably needless to say, having each user in a Black Friday traffic wave querying your inventory database three times per transaction is a recipe for disaster.

So if we’re building a containerized, distributed, multi-region inventory management system, we now have two significant hurdles to overcome:

  1. How can we keep a source-of-truth distributed, multi-region database consistent?
  2. How can we keep application services up-to-date with correct information from that database without constantly querying it or risking serving users outdated information from a cache?

Not so long ago, there wasn’t really a good answer to these questions, because relational databases – which can enforce consistency with ACID transactional guarantees – weren’t cloud-native, distributed, or easy to scale.

Companies were forced to either grapple with the technical complexity of trying to scale legacy SQL databases manually via sharding (which is time-consuming and operationally complex) or embrace the compromises inherent in NoSQL databases (which are easier to scale but can offer only eventual consistency). Neither of these options is ideal, and both have the potential to lead to Black Friday disasters such as…

  • …not being able to keep up with unexpected demand because sharding a legacy SQL database isn’t quick or easy.
  • …selling past zero because the eventual consistency of NoSQL isn’t consistent fast enough to keep up with the pace of customer purchases.

Today, thankfully, there are better options. Modern, distributed SQL databases such as CockroachDB offer the ironclad transactional consistency of legacy SQL databases together with the cloud-native, multi-region scalability of NoSQL databases. And since every CockroachDB database can be treated as a single logical database by the application, there’s less operational complexity: your application can treat CockroachDB exactly like it would treat a single Postgres instance, and CockroachDB will handle the complexity of maintaining consistency across a multi-region distributed setup all by itself.

So, choosing the right tooling has solved our first question: how can we keep a source-of-truth distributed, multi-region database consistent? But the second one remains unanswered: how can we keep application services up-to-date with correct information without overloading the database?

The most common solution to that problem is changefeeds, and here too, it turns out that choosing the right tool for the job can make things easier.

It’s possible to build a changefeed into your SQL database manually using the transactional outbox pattern, but modern databases often come with this sort of service built-in. CockroachDB’s built-in Change Data Capture feature, for example, allows you to easily send a changefeed to an event streaming platform such as Kafka.

For example, the diagram below is based on the real architecture of a major online retailer, and illustrates how they keep application services up-to-date with inventory status without constantly querying the inventory database itself. The CDC feed from their CockroachDB database is fed to Kafka, which then serves the variety of application services that are triggered by inventory changes.

Black-Friday-ready Inventory management architecture

For example, imagine it’s Black Friday and 100 TVs are available in your online shop. A customer buys one, and the inventory database is updated to 99. This update is sent via a changefeed to Kafka, and an application service is then triggered to update the product page cache so that the next time a user loads the product page, it reads 99 instead of 100 without having to query the inventory DB.

Unfortunately, there’s no way to make Black Friday easy on your application. Even with the best backend tools and impeccable architecture, the loads Black Friday can bring are always likely to be a source of stress. But building an inventory system that can deliver scale, availability, low latency, and consistency can certainly make things easier. Learn more about how CockroachDB is powering retail and ecommerce applications at massive scale.

About the author

Charlie Custer

Charlie is a former teacher, tech journalist, and filmmaker who’s now combined those three professions into writing and making videos about databases and application development (and occasionally messing with NLP and Python to create weird things in his spare time).

github link linkedin link

Keep Reading

How to build an inventory management system that scales (with reference architecture)

“Do you have any more of these? Can you check in the back?”

This customer query has been the bane of retail employees …

Read More
How to build an inventory management system that scales (with reference architecture)

“Do you have any more of these? Can you check in the back?”

This customer query has been the bane of retail employees …

Read More
Tech Trends & Challenges in the Retail Industry

Tomorrow, as you undoubtedly know, is day-one of an annual two-day retail event known as “Amazon Prime Day”. …

Read More
Developer Resources