Running CockroachDB on Kubernetes
[Instructions updated on January 4, 2017.]
At Cockroach Labs, we’re working hard to make it easier to keep your data safe and available even in the face of catastrophic failures. However, if you’ve ever been responsible for deploying and operating services in production, you know that there’s more involved in achieving high reliability than just starting up a few processes and stepping away from the keyboard, even for highly survivable applications like CockroachDB. That’s where Kubernetes comes in.
Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications, where “containerized” typically implies the use of Docker. Kubernetes provides a huge assortment of services that help you keep your applications up and running, from application replication and rolling updates to health checking and log collection.
If you’re familiar with CockroachDB, you know that it handles all the messy details of keeping your data replicated in a consistent way even in the face of all sorts of failures. By running CockroachDB on Kubernetes, we are able to pair its built-in replication and survivability model with Kubernetes’ process management, creating a highly available system that truly makes data easy.
Managing State on Kubernetes
If you have some experience with Kubernetes, you may be skeptical about the plan of running a database on top of a system that historically hasn’t offered great facilities for managing stateful, clustered applications. It’s true that Kubernetes primarily targeted stateless applications (those that don’t manage their own persistent data) in its early stages, but the team behind it has been working hard this year to build out first-class support for stateful applications in the form of its
Normally, when a Kubernetes pod (which is a group of one or more containers that are scheduled and run together) dies, it gets replaced by a new pod that has a new identity, including a new IP address and hostname. However, the StatefulSet feature ensures that each replica has its own stable identity (resolvable via DNS) no matter how many times it gets restarted or changes underlying hosts. This is useful for CockroachDB because it means that every time a pod gets replaced, we don’t have to treat it as a new node in the cluster, which would require a lot of data replication every time it happened. This is important for efficiently supporting our consensus protocol and distributed transactions.
The other, more obvious problem involved in running a database like CockroachDB on an orchestration system like Kubernetes is figuring out where the data for each replica will live. The options for this are both more and less mature, depending on what kind of storage you need. Kubernetes has long had good support for `PersistentVolumes`, which are typically remote disks that can be mounted to any node in the Kubernetes cluster. These are nice because they allow a replica to move around without losing its data. However, because the data is remote (and often replicated under the covers, as is the case for EBS volumes on EC2 and Persistent Disks on Google Compute Engine), there’s typically a noticeable latency cost involved in using it rather than local storage.
A cloud-native, clustered database like CockroachDB is actually perfectly fine with losing an individual machine’s data from time to time because it’s capable of detecting that the data from that machine doesn’t have as many replicas as desired and adding new replicas for it. Because of that, the ideal would be to use local disk for each replica for the sake of lower latency. However, Kubernetes doesn’t yet support using local disk with StatefulSets due to a number of concerns (it’s under serious discussion for the 1.6 release). Remote persistent volumes work well enough for our purposes in the meantime.
The Details: Running CockroachDB on Kubernetes
Starting a Kubernetes cluster
There are a lot of different ways to set up a Kubernetes cluster. For the sake of simplicity, we’ll use Container Engine, but this is also doable in other environments. For example, check out our documentation for how to run locally on Minikube or the Kubernetes documentation for setting up a cluster on AWS. If you have gcloud, the Google Cloud command line tool installed, you can create a cluster by running:
gcloud container clusters create cockroachdb-cluster
Spinning up CockroachDB
Kubernetes configurations are typically managed using YAML configuration files, and our CockroachDB config is no exception. We can create a cluster using the configuration below, with comments inlined to explain some of what we’re doing in it. First, copy the configuration file from our Github repository into a file named `cockroachdb-statefulset.yaml`. This file defines the Kubernetes resources to be created, including the StatefulSet object that will spin up CockroachDB containers and attach them to persistent volumes.
Then create the resources as shown below (if you’re using Minikube, you may first need to manually provision persistent volumes). Shortly afterward, you should soon see 3 replicas running in your cluster along with a couple of services. At first, only a subset of the replicas may be showing because they haven’t all started up yet. That’s normal, as StatefulSets create the replicas one-by-one, starting with the first:
$ kubectl create -f cockroachdb-statefulset.yaml service "cockroachdb-public" created service "cockroachdb" created poddisruptionbudget "cockroachdb-budget" created statefulset "cockroachdb" created $ kubectl get services cockroachdb None <none> 26257/TCP,8080/TCP 4s cockroachdb-public 10.0.0.85 <none> 26257/TCP,8080/TCP 4s kubernetes 10.0.0.1 <none> 443/TCP 1h $ kubectl get pods NAME READY STATUS RESTARTS AGE cockroachdb-0 1/1 Running 0 29s cockroachdb-1 0/1 Running 0 9s $ kubectl get pods NAME READY STATUS RESTARTS AGE cockroachdb-0 1/1 Running 0 1m cockroachdb-1 1/1 Running 0 41s cockroachdb-2 1/1 Running 0 21s
If you’re curious to see what’s happening in the cluster behind the scenes, you can check out the logs for one of the pods by running `kubectl logs cockroachdb-0`.
Using the CockroachDB cluster
Once your cluster is up and running, you might want to try it out! To open a SQL shell within the Kubernetes cluster, you can run a one-off interactive pod like this, using the `cockroachdb-public` hostname to access the CockroachDB cluster. Kubernetes will automatically load-balance connections to that hostname across the healthy CockroachDB instances.
$ kubectl run cockroachdb -it --image=cockroachdb/cockroach --rm --restart=Never -- sql --host=cockroachdb-public Waiting for pod default/cockroachdb to be running, status is Pending, pod ready: false Hit enter for command prompt [email protected]:26257> CREATE DATABASE bank; CREATE DATABASE [email protected]:26257> CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL); CREATE TABLE [email protected]:26257> INSERT INTO bank.accounts VALUES (1234, 10000.50); INSERT 1 [email protected]:26257> SELECT * FROM bank.accounts; +------+---------+ | id | balance | +------+---------+ | 1234 | 10000.5 | +------+---------+ (1 row)
Accessing the Admin UI
If you want to see information about how the cluster is doing, you can try pulling up the CockroachDB admin UI by port-forwarding from your local machine to one of the pods:
kubectl port-forward cockroachdb-0 8080
Once you’ve done that, you should be able to access the admin UI by visiting http://localhost:8080/ in your web browser:
Simulating node failure
If you want to test out the resiliency of the cluster, you can try killing some of the containers by running a command like `kubectl delete pod cockroachdb-3` from a different terminal while you’re still accessing the cluster from your SQL shell. You may occasionally get a “bad connection” error if you happened to delete the instance that your shell was talking too, but retrying the query will work. The container will be recreated for you by the StatefulSet controller, just as it would if a machine went down in a production environment.
If you want to test out the durability of the data in the cluster, you can also try deleting all the pods at once and ensuring that they start up properly from their persistent volumes when they come back up. To do that, you can run `kubectl delete pod –selector app=cockroachdb`, which deletes all pods that have the label `app=cockroachdb`, which includes the pods from our StatefulSet. It might take a bit of time for all the pods to come back up (like it did when creating them), but you should be able to get the same data back in response to your queries from the SQL shell once some of them are up and running again.
Scaling the CockroachDB cluster
Kubernetes makes it easy to scale the cluster up or down as desired. If you want to add a new replica to the cluster, all you have to do is resize the StatefulSet as shown below:
kubectl scale statefulset cockroachdb --replicas=4
Shutting down the CockroachDB cluster
If you want to clean up all the resources we’ve created, we can do it in a single command thanks to the labels that we added to all the resources:
kubectl delete statefulsets,pods,persistentvolumes,persistentvolumeclaims,services,poddisruptionbudget -l app=cockroachdb
Alternatively, you can shut down your entire Kubernetes cluster by running:
gcloud container clusters delete cockroachdb-cluster
Where to go from here
Given that CockroachDB isn’t considered production-ready just yet, we wouldn’t recommend trusting this setup with any mission-critical data right now. However, there are many things that you can do with it in the meantime. You could try:
- Writing applications that use the CockroachDB cluster via one of the many supported client libraries.
- Modifying how the cluster is initialized to use certificates for encryption between nodes.
- Setting up a cluster in a cloud or bare metal environment using a different kind of PersistentVolume, rather than on Container Engine.
- Setting up Prometheus to monitor CockroachDB within the cluster, taking advantage of the annotations we put on the CockroachDB StatefulSet.
- Contributing feature requests, issues, or improvements to CockroachDB, either for the Kubernetes documentation or for the core database itself!