Deploy a Local Cluster in Kubernetes

​

Limitations

​

Kubernetes version

​

Public operator

• The Public operator deploys clusters in a single region. For multi-region deployments using manual configs, Cockroach Labs recommends using the which is designed to support multi-region deployments. For guidance on how to force multi-region support with the Public operator, see .
• Using the Public operator, you can give a new cluster an arbitrary number of labels. However, a cluster’s labels cannot be modified after it is deployed. To track the status of this limitation, refer to #993 in the Public operator project’s issue tracker.

​

Helm version

Error: UPGRADE FAILED: template: cockroachdb/templates/tests/client.yaml:6:14: executing "cockroachdb/templates/tests/client.yaml" at <.Values.networkPolicy.enabled>: nil pointer evaluating interface {}.enabled

​

Network

​

Resources

​

Storage

​

Before you begin

​

Step 1. Start Kubernetes

1. Follow the Minikube documentation to install the latest version of minikube, a hypervisor, and the kubectl command-line tool.
2. Start a local Kubernetes cluster:
   minikube start
   

​

Step 2. Start CockroachDB

• Public operator
• Helm package manager
• Manually apply our StatefulSet configuration and related files

​

Install the Operator

1. Apply the custom resource definition (CRD) for the Public operator:
   $ kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.18.3/install/crds.yaml
   
   customresourcedefinition.apiextensions.k8s.io/crdbclusters.crdb.cockroachlabs.com created
   
2. By default, the Public operator is configured to install in the cockroach-operator-system namespace and to manage CockroachDB instances for all namespaces on the cluster.

• To use these defaults, apply the Public operator manifest without modifying it:
  kubectl apply -f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.18.3/install/operator.yaml
  
  clusterrole.rbac.authorization.k8s.io/cockroach-database-role created
  serviceaccount/cockroach-database-sa created
  clusterrolebinding.rbac.authorization.k8s.io/cockroach-database-rolebinding created
  role.rbac.authorization.k8s.io/cockroach-operator-role created
  clusterrolebinding.rbac.authorization.k8s.io/cockroach-operator-rolebinding created
  clusterrole.rbac.authorization.k8s.io/cockroach-operator-role created
  serviceaccount/cockroach-operator-sa created
  rolebinding.rbac.authorization.k8s.io/cockroach-operator-default created
  deployment.apps/cockroach-operator created
  
• To change these defaults:
  1. Download the Public operator manifest:
     curl -O https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.18.3/install/operator.yaml
     
  2. To use a custom namespace, edit all instances of namespace: cockroach-operator-system with your desired namespace.
  3. To limit the namespaces that will be monitored, set the WATCH\_NAMESPACE environment variable in the Deployment pod spec. This can be set to a single namespace or a comma-delimited set of namespaces. When set, only those CrdbCluster resources in the supplied namespace(s) will be reconciled.
  4. Apply your local version of the Public operator manifest to the cluster:
     kubectl apply -f operator.yaml
     

1. Set your current namespace to the one used by the Public operator. For example, to use the Public operator’s default namespace:
   $ kubectl config set-context --current --namespace=cockroach-operator-system
   
2. Validate that the operator is running:
   $ kubectl get pods
   
   NAME                                  READY   STATUS    RESTARTS   AGE
   cockroach-operator-6f7b86ffc4-9ppkv   1/1     Running   0          54s
   

​

Initialize the cluster

1. Download example.yaml, a custom resource that tells the operator how to configure the Kubernetes cluster.
   $ curl -O https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.18.3/examples/example.yaml
   
   By default, this custom resource specifies CPU and memory resources that are appropriate for the virtual machines used in this deployment example. On a production cluster, you should substitute values that are appropriate for your machines and workload. For details on configuring your deployment, see .

1. Apply example.yaml:
   $ kubectl apply -f example.yaml
   
   The operator will create a StatefulSet and initialize the nodes as a cluster.
   crdbcluster.crdb.cockroachlabs.com/cockroachdb created
   
2. Check that the pods were created:
   $ kubectl get pods
   
   NAME                                  READY   STATUS    RESTARTS   AGE
   cockroach-operator-6f7b86ffc4-9t9zb   1/1     Running   0          3m22s
   cockroachdb-0                         1/1     Running   0          2m31s
   cockroachdb-1                         1/1     Running   0          102s
   cockroachdb-2                         1/1     Running   0          46s
   
   Each pod should have READY status soon after being created.

​

Configure the cluster

1. Download and modify our :
   $ curl -O https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/bring-your-own-certs/cockroachdb-statefulset.yaml
   
2. Update secretName with the name of the corresponding node secret. The secret names depend on your method for generating secrets. For example, if you follow the below steps using cockroach cert, use this secret name:
   secret:
     secretName: cockroachdb.node
   
3. The StatefulSet configuration deploys CockroachDB into the default namespace. To use a different namespace, search for kind: RoleBinding and change its subjects.namespace property to the name of the namespace. Otherwise, a failed to read secrets error occurs when you attempt to follow the steps in Initialize the cluster.

​

Create certificates

1. Create two directories:
   $ mkdir certs my-safe-directory
   

   Directory	Description
   certs	You’ll generate your CA certificate and all node and client certificates and keys in this directory.
   my-safe-directory	You’ll generate your CA key in this directory and then reference the key when generating node and client certificates.

2. Create the CA certificate and key pair:
   $ cockroach cert create-ca \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   
3. Create a client certificate and key pair for the root user:
   $ cockroach cert create-client \
   root \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   
4. Upload the client certificate and key to the Kubernetes cluster as a secret:
   $ kubectl create secret \
   generic cockroachdb.client.root \
   --from-file=certs
   
   secret/cockroachdb.client.root created
   
5. Create the certificate and key pair for your CockroachDB nodes:
   $ cockroach cert create-node \
   localhost 127.0.0.1 \
   cockroachdb-public \
   cockroachdb-public.default \
   cockroachdb-public.default.svc.cluster.local \
   *.cockroachdb \
   *.cockroachdb.default \
   *.cockroachdb.default.svc.cluster.local \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   
6. Upload the node certificate and key to the Kubernetes cluster as a secret:
   $ kubectl create secret \
   generic cockroachdb.node \
   --from-file=certs
   
   secret/cockroachdb.node created
   
7. Check that the secrets were created on the cluster:
   $ kubectl get secrets
   
   NAME                      TYPE                                  DATA   AGE
   cockroachdb.client.root   Opaque                                3      41m
   cockroachdb.node          Opaque                                5      14s
   default-token-6qjdb       kubernetes.io/service-account-token   3      4m
   

​

Initialize the cluster

1. Use the config file you downloaded to create the StatefulSet that automatically creates 3 pods, each running a CockroachDB node:
   $ kubectl create -f cockroachdb-statefulset.yaml
   
   serviceaccount/cockroachdb created
   role.rbac.authorization.k8s.io/cockroachdb created
   rolebinding.rbac.authorization.k8s.io/cockroachdb created
   service/cockroachdb-public created
   service/cockroachdb created
   poddisruptionbudget.policy/cockroachdb-budget created
   statefulset.apps/cockroachdb created
   
2. Initialize the CockroachDB cluster:
   1. Confirm that three pods are Running successfully. Note that they will not be considered Ready until after the cluster has been initialized:
      $ kubectl get pods
      
      NAME            READY     STATUS    RESTARTS   AGE
      cockroachdb-0   0/1       Running   0          2m
      cockroachdb-1   0/1       Running   0          2m
      cockroachdb-2   0/1       Running   0          2m
      
   2. Confirm that the persistent volumes and corresponding claims were created successfully for all three pods:
      $ kubectl get pv
      
      NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
      pvc-9e435563-fb2e-11e9-a65c-42010a8e0fca   100Gi      RWO            Delete           Bound    default/datadir-cockroachdb-0   standard                51m
      pvc-9e47d820-fb2e-11e9-a65c-42010a8e0fca   100Gi      RWO            Delete           Bound    default/datadir-cockroachdb-1   standard                51m
      pvc-9e4f57f0-fb2e-11e9-a65c-42010a8e0fca   100Gi      RWO            Delete           Bound    default/datadir-cockroachdb-2   standard                51m
      
   3. Run cockroach init on one of the pods to complete the node startup process and have them join together as a cluster:
      $ kubectl exec -it cockroachdb-0 \
      -- /cockroach/cockroach init \
      --certs-dir=/cockroach/cockroach-certs
      
      Cluster successfully initialized
      
   4. Confirm that cluster initialization has completed successfully. The job should be considered successful and the Kubernetes pods should soon be considered Ready:
      $ kubectl get pods
      
      NAME            READY     STATUS    RESTARTS   AGE
      cockroachdb-0   1/1       Running   0          3m
      cockroachdb-1   1/1       Running   0          3m
      cockroachdb-2   1/1       Running   0          3m
      

1. Install the Helm client (version 3.0 or higher) and add the cockroachdb chart repository:
   $ helm repo add cockroachdb https://charts.cockroachdb.com/
   
   "cockroachdb" has been added to your repositories
   
2. Update your Helm chart repositories to ensure that you’re using the latest CockroachDB chart:
   $ helm repo update
   
3. The cluster configuration is set in the Helm chart’s values file.

1. Create a local YAML file (e.g., my-values.yaml) to specify your custom values. These will be used to override the defaults in values.yaml.
2. To avoid running out of memory when CockroachDB is not the only pod on a Kubernetes node, you must set memory limits explicitly. This is because CockroachDB does not detect the amount of memory allocated to its pod when run in Kubernetes. We recommend setting conf.cache and conf.max-sql-memory each to 1/4 of the memory allocation specified in statefulset.resources.requests and statefulset.resources.limits.

conf:
  cache: "2Gi"
  max-sql-memory: "2Gi"

1. Install the CockroachDB Helm chart, specifying your custom values file. Provide a “release” name to identify and track this particular deployment of the chart, and override the default values with those in my-values.yaml.

$ helm install my-release --values {custom-values}.yaml cockroachdb/cockroachdb

1. Confirm that CockroachDB cluster initialization has completed successfully, with the pods for CockroachDB showing 1/1 under READY and the pod for initialization showing COMPLETED under STATUS:
   $ kubectl get pods
   
   NAME                                READY     STATUS      RESTARTS   AGE
   my-release-cockroachdb-0            1/1       Running     0          8m
   my-release-cockroachdb-1            1/1       Running     0          8m
   my-release-cockroachdb-2            1/1       Running     0          8m
   my-release-cockroachdb-init-hxzsc   0/1       Completed   0          1h
   
2. Confirm that the persistent volumes and corresponding claims were created successfully for all three pods:
   $ kubectl get pv
   
   NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                      STORAGECLASS   REASON    AGE
   pvc-71019b3a-fc67-11e8-a606-080027ba45e5   100Gi      RWO            Delete           Bound     default/datadir-my-release-cockroachdb-0   standard                 11m
   pvc-7108e172-fc67-11e8-a606-080027ba45e5   100Gi      RWO            Delete           Bound     default/datadir-my-release-cockroachdb-1   standard                 11m
   pvc-710dcb66-fc67-11e8-a606-080027ba45e5   100Gi      RWO            Delete           Bound     default/datadir-my-release-cockroachdb-2   standard                 11m
   

​

Step 3. Use the built-in SQL client

$ kubectl create \
-f https://raw.githubusercontent.com/cockroachdb/cockroach-operator/v2.18.3/examples/client-secure-operator.yaml

1. Get a shell into the pod and start the CockroachDB :
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach/cockroach-certs \
   --host=cockroachdb-public
   
   # Welcome to the CockroachDB SQL shell.
   # All statements must be terminated by a semicolon.
   # To exit, type: \q.
   #
   # Server version: CockroachDB CCL v21.1.0 (x86_64-unknown-linux-gnu, built 2021/04/23 13:54:57, go1.13.14) (same version as client)
   # Cluster ID: a96791d9-998c-4683-a3d3-edbf425bbf11
   #
   # Enter \? for a brief introduction.
   #
   root@cockroachdb-public:26257/defaultdb>
   
2. Run some basic :
   > CREATE DATABASE bank;
   
   > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
   
   > INSERT INTO bank.accounts VALUES (1, 1000.50);
   
   > SELECT * FROM bank.accounts;
   
     id | balance
   +----+---------+
      1 | 1000.50
   (1 row)
   
3. :
   > CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
   
   You will need this username and password to access the DB Console later.
4. Exit the SQL shell and pod:
   > \q
   

$ kubectl create \
-f https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/bring-your-own-certs/client.yaml

pod/cockroachdb-client-secure created

1. Get a shell into the pod and start the CockroachDB :
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   
   # Welcome to the cockroach SQL interface.
   # All statements must be terminated by a semicolon.
   # To exit: CTRL + D.
   #
   # Client version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   # Server version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   
   # Cluster ID: 256a8705-e348-4e3a-ab12-e1aba96857e4
   #
   # Enter \? for a brief introduction.
   #
   root@cockroachdb-public:26257/defaultdb>
   

1. Run some basic :
   > CREATE DATABASE bank;
   
   > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
   
   > INSERT INTO bank.accounts VALUES (1, 1000.50);
   
   > SELECT * FROM bank.accounts;
   
     id | balance
   +----+---------+
      1 | 1000.50
   (1 row)
   
2. :
   > CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
   
   You will need this username and password to access the DB Console later.
3. Exit the SQL shell and pod:
   > \q
   

1. Download the file:
   $ curl -OOOOOOOOO \
   https://raw.githubusercontent.com/cockroachdb/helm-charts/master/examples/client-secure.yaml
   
2. In the file, set the following values:
   • spec.serviceAccountName: my-release-cockroachdb
   • spec.image: cockroachdb/cockroach: {your CockroachDB version}
   • spec.volumes[0].project.sources[0].secret.name: my-release-cockroachdb-client-secret
3. Use the file to launch a pod and keep it running indefinitely:
   $ kubectl create -f client-secure.yaml
   
   pod "cockroachdb-client-secure" created
   
4. Get a shell into the pod and start the CockroachDB :
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=./cockroach-certs \
   --host=my-release-cockroachdb-public
   
   # Welcome to the cockroach SQL interface.
   # All statements must be terminated by a semicolon.
   # To exit: CTRL + D.
   #
   # Client version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   # Server version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   
   # Cluster ID: 256a8705-e348-4e3a-ab12-e1aba96857e4
   #
   # Enter \? for a brief introduction.
   #
   root@my-release-cockroachdb-public:26257/defaultdb>
   
   This pod will continue running indefinitely, so any time you need to reopen the built-in SQL client or run any other (e.g., cockroach node), repeat step 2 using the appropriate cockroach command. If you’d prefer to delete the pod and recreate it when needed, run kubectl delete pod cockroachdb-client-secure.
5. Run some basic :
   > CREATE DATABASE bank;
   
   > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
   
   > INSERT INTO bank.accounts VALUES (1, 1000.50);
   
   > SELECT * FROM bank.accounts;
   
     id | balance
   +----+---------+
      1 | 1000.50
   (1 row)
   
6. :
   > CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
   
   You will need this username and password to access the DB Console later.
7. Exit the SQL shell and pod:
   > \q
   

​

Step 4. Access the DB Console

1. On secure clusters, can only be accessed by admin users. Get a shell into the pod and start the CockroachDB :
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach/cockroach-certs \
   --host=cockroachdb-public
   
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   
   $ kubectl exec -it cockroachdb-client-secure
   — ./cockroach sql
   —certs-dir=/cockroach-certs
   —host=my-release-cockroachdb-public
2. Assign roach to the admin role (you only need to do this once):
   > GRANT admin TO roach;
   
3. Exit the SQL shell and pod:
   > \q
   
4. In a new terminal window, port-forward from your local machine to the cockroachdb-public service:
   $ kubectl port-forward service/cockroachdb-public 8080
   
   $ kubectl port-forward service/cockroachdb-public 8080
   
   $ kubectl port-forward service/my-release-cockroachdb-public 8080
   
   Forwarding from 127.0.0.1:8080 -> 8080
   
5. Go to https://localhost:8080 and log in with the username and password you created earlier.

1. Go to http://localhost:8080.
2. In the UI, verify that the cluster is running as expected:
   • View the to ensure that all nodes successfully joined the cluster.
   • Click the Databases tab on the left to verify that bank is listed.

​

Step 5. Stop the cluster

• If you plan to restart the cluster, use the minikube stop command. This shuts down the minikube virtual machine but preserves all the resources you created:
  minikube stop
  
  Stopping local Kubernetes cluster...
  Machine stopped.
  
  You can restore the cluster to its previous state with minikube start.
• If you do not plan to restart the cluster, use the minikube delete command. This shuts down and deletes the minikube virtual machine and all the resources you created, including persistent volumes:
  minikube delete
  
  Deleting local Kubernetes cluster...
  Machine deleted.
  

​

See also