How to store billions of files with CockroachDB and Kubernetes

How to store billions of files with CockroachDB and Kubernetes

Optimal Systems came to CockroachDB with the best kind of problem: they had become too successful.

Optimal was founded in 1991 to meet the European Space Agency’s need to archive its satellite photos. Decades of successful growth saw the storage product they built for the ESA evolve and expand to serve clients in more than 30 countries.

In the past few years, as new customers continued to sign on, Optimal Systems realized that their success was outpacing their system. They needed to modernize their infrastructure to keep up.

A next-gen storage solution with Kubernetes and CockroachDB

Rather than try a lift-and-shift approach, Optimal Systems CTO Nikola Milanovic wanted to build something entirely new. Their goals for the project were:

They were looking for either a managed cloud solution or a way they could run their own database using Docker containers and scale with Kubernetes.

They looked at Google Spanner, which led them to investigate CockroachDB. Once they did, they saw that CockroachDB offered everything they were looking for.

Our choice was pretty simple. CockroachDB 100% fits our vision for an infinitely scalable database in the cloud.

Nikola Milanovic, Optimal Systems CTO

Optimal Systems’s document storage solution goes far beyond simple storage – it includes services such as AI-driven character recognition that can read text in static documents such as images or PDFs and make them searchable. Customers can then search them easily using powerful Elasticsearch-driven indexing.

CockroachDB’s support for hybrid deployment models is ideal for this kind of system. In this case, it enables Optimal Systems to optimize their system so that (for example) the database can remain in the cloud where it’s cheap, fault-tolerant, and scalable) while the more expensive AI system can run on premises. All of Optimal’s microservices can be distributed to optimize cost and performance without sacrificing their ability to communicate with each other.

Once it was built, Optimal named its new storage product yuuvis®. It has already won an innovation award, and with good reason! It’s incredibly powerful – so powerful, in fact, that they’re still trying to figure out exactly what its limits are!

Storing billions of documents in the cloud

After another company challenged itself to scale a database that stored 11 billion documents, Optimal decided to take things a step further. They’re planning to host 11 billion completely searchable documents, using their CockroachDB- and Elasticsearch-powered system.

They’ll be working towards this challenge over the course of the summer. As of this writing, they’ve already got more than a billion documents online, and the system is working flawlessly.

Other exciting things are in the works, too. Optimal already has clients in 30 different countries, but its customer base has been primarily in Europe. Having joined the Japan-based Kyocera Group in 2020, Optimal now sees an opportunity to expand into Asian and North American markets.

The fact that they now have an infinitely, globally scalable database means that they can focus on what really matters – growing their business – secure in the knowledge that wherever they go and however they grow, their database can handle anything they throw at it.

Learn more about Optimal Systems and the system they built using CockroachDB.

About the author

Charlie Custer github link linkedin link

Charlie is a former teacher, tech journalist, and filmmaker who’s now combined those three professions into writing and making videos about databases and application development (and occasionally messing with NLP and Python to create weird things in his spare time).

Keep Reading

Building better IAM with a scale-out, OLTP database

The identity and access management platforms that I either used or researched in the past always left much to be …

Read more
Fueling the startup economy: Crowdfunding on a transaction-oriented system

This is the story of how Australian-based Birchal created a crowdfunding platform on CockroachDB 

Early stage capital is …

Read more