This page shows you how to orchestrate the deployment and management of an insecure three-node CockroachDB cluster as a swarm of Docker Engines.
--insecure flag used in this tutorial is intended for non-production testing only. To run CockroachDB in production, use a secure cluster instead.
Before you begin
Before you begin, it's helpful to review some terminology:
|instance||A physical or virtual machine. In this tutorial, you'll use three, one per CockroachDB node.|
|Docker Engine||This is the core Docker application that creates and runs containers. In this tutorial, you'll install and start Docker Engine on each of your three instances.|
|swarm||A swarm is a group of Docker Engines joined into a single, virtual host.|
|swarm node||Each member of a swarm is considered a node. In this tutorial, each instance will be a swarm node, one as the master node and the two others as worker nodes. You'll submit service definitions to the master node, which will dispatch work to the worker nodes.|
|service||A service is the definition of the tasks to execute on swarm nodes. In this tutorial, you'll define three services, each starting a CockrochDB node inside a container and joining it into a single cluster. Each service also ensures a stable network identity on restart via a resolvable DNS name.|
|overlay network||An overlay network enables communication between the nodes of a swarm. In this tutorial, you'll create an overlay network and use it in each of your services.|
Step 1. Create instances
Create three instances, one for each node of your cluster.
- For GCE-specific instructions, read through step 2 of Deploy CockroachDB on GCE.
- For AWS-specific instructions, read through step 2 of Deploy CockroachDB on AWS.
Be sure to configure your network to allow TCP communication on these ports:
26257for inter-node communication (i.e., working as a cluster) and connecting with applications
8080for exposing your DB Console
Step 2. Install Docker Engine
On each instance:
Confirm that the Docker daemon is running in the background:copy
$ sudo docker version
Step 3. Start the swarm
On the instance where you want to run your manager node, initialize the swarm.
Take note of the output for
docker swarm initas it includes the command you'll use in the next step. It should look like this:copy
$ sudo docker swarm init --advertise-addr 10.142.0.2
Swarm initialized: current node (414z67gr5cgfalm4uriu4qdtm) is now a manager. To add a worker to this swarm, run the following command: $ docker swarm join \ --token SWMTKN-1-5vwxyi6zl3cc62lqlhi1jrweyspi8wblh2i3qa7kv277fgy74n-e5eg5c7ioxypjxlt3rpqorh15 \ 10.142.0.2:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
On the other two instances, create a worker node joined to the swarm by running the
docker swarm joincommand in the output from step 1, for example:copy
$ sudo docker swarm join \ --token SWMTKN-1-5vwxyi6zl3cc62lqlhi1jrweyspi8wblh2i3qa7kv277fgy74n-e5eg5c7ioxypjxlt3rpqorh15 \ 10.142.0.2:2377
This node joined a swarm as a worker.
On the instance running your manager node, verify that your swarm is running:copy
$ sudo docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 414z67gr5cgfalm4uriu4qdtm * instance-1 Ready Active Leader ae144s35dx1p1lcegh6bblyed instance-2 Ready Active aivjg2joxyvzvbksjsix27khy instance-3 Ready Active
Step 4. Create an overlay network
On the instance running your manager node, create an overlay network so that the containers in your swarm can talk to each other:
$ sudo docker network create --driver overlay --attachable cockroachdb
--attachable option enables non-swarm containers running on Docker to access services on the network, which makes the service easier to use interactively.
Step 5. Start the CockroachDB cluster
On the instance running your manager node, create one swarm service for each CockroachDB node:copy
# Start the first service: $ sudo docker service create \ --replicas 1 \ --name cockroachdb-1 \ --hostname cockroachdb-1 \ --network cockroachdb \ --mount type=volume,source=cockroachdb-1,target=/cockroach/cockroach-data,volume-driver=local \ --stop-grace-period 60s \ --publish 8080:8080 \ cockroachdb/cockroach:v20.2.2 start \ --join=cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257 \ --cache=.25 \ --max-sql-memory=.25 \ --logtostderr \ --insecurecopy
# Start the second service: $ sudo docker service create \ --replicas 1 \ --name cockroachdb-2 \ --hostname cockroachdb-2 \ --network cockroachdb \ --mount type=volume,source=cockroachdb-2,target=/cockroach/cockroach-data,volume-driver=local \ --stop-grace-period 60s \ cockroachdb/cockroach:v20.2.2 start \ --join=cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257 \ --cache=.25 \ --max-sql-memory=.25 \ --logtostderr \ --insecurecopy
# Start the third service: $ sudo docker service create \ --replicas 1 \ --name cockroachdb-3 \ --hostname cockroachdb-3 \ --network cockroachdb \ --mount type=volume,source=cockroachdb-3,target=/cockroach/cockroach-data,volume-driver=local \ --stop-grace-period 60s \ cockroachdb/cockroach:v20.2.2 start \ --join=cockroachdb-1:26257,cockroachdb-2:26257,cockroachdb-3:26257 \ --cache=.25 \ --max-sql-memory=.25 \ --logtostderr \ --insecure
These commands each create a service that starts a container, joins it to the overlay network, and starts a CockroachDB node inside the container mounted to a local volume for persistent storage. Let's look at each part:
sudo docker service create: The Docker command to create a new service.
--replicas: The number of containers controlled by the service. Since each service will control one container running one CockroachDB node, this will always be
--name: The name for the service.
--hostname: The hostname of the container. It will listen for connections on this address.
--network: The overlay network for the container to join. See Step 4. Create an overlay network for more details.
--mount: This flag mounts a local volume with the same name as the service. This means that data and logs for the node running in this container will be stored in
/cockroach/cockroach-dataon the instance and will be reused on restart as long as restart happens on the same instance, which is not guaranteed.Note:If you plan on replacing or adding instances, it's recommended to use remote storage instead of local disk. To do so, create a remote volume for each CockroachDB instance using the volume driver of your choice, and then specify that volume driver instead of the
volume-driver=localpart of the command above, e.g.,
volume-driver=gceif using the GCE volume driver.
--stop-grace-period: This flag sets a grace period to give CockroachDB enough time to shut down gracefully, when possible.
--publish: This flag makes the DB Console accessible at the IP of any instance running a swarm node on port
8080. Note that, even though this flag is defined only in the first node's service, the swarm exposes this port on every swarm node using a routing mesh. See Publishing ports for more details.
cockroachdb/cockroach:v20.2.2 start ...: The CockroachDB command to start a node in the container in insecure mode and instruct other cluster members to talk to each other using their persistent network addresses, which match the services' names.
Verify that all three services were created successfully:copy
$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE a6g0ur6857j6 cockroachdb-1 replicated 1/1 cockroachdb/cockroach:v20.2.2 dr81a756gaa6 cockroachdb-2 replicated 1/1 cockroachdb/cockroach:v20.2.2 il4m7op1afg9 cockroachdb-3 replicated 1/1 cockroachdb/cockroach:v20.2.2Tip:The service definitions tell the CockroachDB nodes to log to
stderr, so if you ever need access to a node's logs for troubleshooting, use
sudo docker logs <container id>from the instance on which the container is running.
Now all the CockroachDB nodes are running, but we still have to explicitly tell them to initialize a new cluster together. To do so, use the
sudo docker runcommand to run the
cockroach initcommand against one of the nodes. The
cockroach initcommand will initialize the cluster, bringing it into a usable state.copy
$ sudo docker run -it --rm --network=cockroachdb cockroachdb/cockroach:v20.2.2 init --host=cockroachdb-1 --insecure
Step 6. Use the built-in SQL client
sudo docker runcommand to start a new container attached to the CockroachDB network, run the built-in SQL shell, and connect it to the cluster:copy
$ sudo docker run -it --rm --network=cockroachdb cockroachdb/cockroach:v20.2.2 sql --host=cockroachdb-1 --insecure
> CREATE DATABASE insecurenodetest;
Use CTRL-D, CTRL-C, or
\qto exit the SQL shell.
Step 7. Monitor the cluster
To view your cluster's DB Console, open a browser and go to
http://<any node's external IP address>:8080.Note:It's possible to access the DB Console from outside of the swarm because you published port
8080externally in the first node's service definition.
On this page, verify that the cluster is running as expected:
View Node list to ensure that all of your nodes successfully joined the cluster.
Click the Databases tab on the left to verify that
Step 8. Simulate node failure
Since we have three service definitions, one for each node, Docker swarm will ensure that there are three nodes running at all times. If a node fails, Docker swarm will automatically create another node with the same network identity and storage.
To see this in action:
On any instance, use the
sudo docker pscommand to get the ID of the container running the CockroachDB node:copy
$ sudo docker ps | grep cockroachdb
9539871cc769 cockroachdb/cockroach:v20.2.2 "/cockroach/cockroach" 10 minutes ago Up 10 minutes 8080/tcp, 26257/tcp cockroachdb-0.1.0wigdh8lx0ylhuzm4on9bbldq
sudo docker killto remove the container, which implicitly terminates the node:copy
$ sudo docker kill 9539871cc769
Verify that the node was restarted in a new container:copy
$ sudo docker ps | grep cockroachdb
4a58f86e3ced cockroachdb/cockroach:v20.2.2 "/cockroach/cockroach" 7 seconds ago Up 1 seconds 8080/tcp, 26257/tcp cockroachdb-0.1.cph86kmhhcp8xzq6a1nxtk9ng
Back in the DB Console, view the Node list and verify that all 3 nodes are live.
Step 9. Scale the cluster
To increase the number of nodes in your CockroachDB cluster:
- Create an additional instance (see Step 1).
- Install Docker Engine on the instance (see Step 2).
- Join the instance to the swarm as a worker node (see Step 3.2).
- Create a new service to start another node and join it to the CockroachDB cluster (see Step 5.1).
Step 10. Stop the cluster
To stop the CockroachDB cluster, on the instance running your manager node, remove the services:copy
$ sudo docker service rm cockroachdb-0 cockroachdb-1 cockroachdb-2
cockroachdb-0 cockroachdb-1 cockroachdb-2
You may want to remove the persistent volumes used by the services as well. To do this, on each instance:copy
# Identify the name of the local volume: $ sudo docker volume ls
# Remove the local volume: $ sudo docker volume rm cockroachdb-0
- Production Checklist
- Manual Deployment
- Orchestrated Deployment
- Monitoring and Alerting
- Performance Benchmarking
- Performance Tuning
- Local Deployment