Deploy CockroachDB on Red Hat OpenShift

This page shows you how to start and stop a secure 3-node CockroachDB cluster on the Red Hat OpenShift platform, using the CockroachDB Kubernetes Operator.

Prerequisites

  • A running OpenShift cluster
Note:

This article assumes you have already installed the OpenShift Container Platform as your Kubernetes cluster. For details on this, see the OpenShift documentation.

Step 1. Create a CockroachDB namespace

  1. Create a cockroachdb namespace. You will create the CockroachDB cluster in this namespace:

    Note:

    oc runs kubectl commands on OpenShift clusters, using the same syntax.

    icon/buttons/copy
    oc create namespace cockroachdb
    
    namespace/cockroachdb created
    
  2. Set cockroachdb as the default namespace:

    icon/buttons/copy
    oc config set-context --current --namespace=cockroachdb
    
    Context "admin" modified.
    

    Validate that this was successful:

    icon/buttons/copy
    oc config view --minify | grep namespace:
    
    namespace: cockroachdb
    

    This lets you issue oc commands without having to specify the namespace each time.

Step 2. Install the Operator

  1. Navigate to your OpenShift web console and click OperatorHub.

  2. Enter "cockroach" in the search box. There are two tiles called CockroachDB Operator. Find the tile without the Marketplace label (which requires a subscription).

    OpenShift OperatorHub

    Click the CockroachDB Operator tile and then Install.

  3. On the Install Operator page, select cockroachdb in the Installed Namespace dropdown and click Install.

  4. Confirm that the Operator is running:

    icon/buttons/copy

      $ oc get pods
    
    NAME                                  READY   STATUS    RESTARTS   AGE
    cockroach-operator-65c4f6df45-h5r5n   1/1     Running   0          51s
    

Step 3. Start CockroachDB

  1. When the Operator is ready, click View Operator to navigate to the Installed Operators page.

  2. In the CockroachDB Operator tile, click Create instance.

    OpenShift OperatorHub

  3. Make sure CockroachDB Version is set to an valid CockroachDB version. For a list of compatible image names, see spec.containers.env in the Operator manifest on GitHub.

  4. This will open the Create CrdbCluster page. By default, this deploys a 3-node secure cluster. Leave the other fields unchanged and click Create.

  5. Navigate to Workloads > Pods and observe the pods being created:

    OpenShift OperatorHub

  6. You can also use the command line to view the pods:

    icon/buttons/copy
    oc get pods
    
    NAME                                  READY   STATUS    RESTARTS   AGE
    cockroach-operator-65c4f6df45-h5r5n   1/1     Running   0          6m11s
    crdb-tls-example-0                    1/1     Running   0          3m15s
    crdb-tls-example-1                    1/1     Running   0          103s
    crdb-tls-example-2                    1/1     Running   0          89s
    
Note:

Due to a known issue, in rare cases the Operator can crash while installing CockroachDB. This causes the CockroachDB pods to fail to start, while the version checker job continues to run. If this happens, run oc get jobs to find the names of any running cockroachdb-vcheck jobs, and delete these jobs with oc delete job {cockroachdb-vcheck-job}. Then wait for the version checker job to restart and succeed.

Step 4. Create a secure client pod

To use the CockroachDB SQL client, first launch a secure pod running the cockroach binary.

This can be defined with the following YAML, which mounts the Operator's generated certificates:

Tip:

spec.containers.image should match the Image value that is displayed under the Containers section on the Pods page when you select a CockroachDB pod. Be sure to select a CockroachDB pod and not the Operator pod.

Note that OpenShift may display the image SHA instead of the tag. In this case, you should use the SHA for spec.containers.image.

icon/buttons/copy
apiVersion: v1
kind: Pod
metadata:
  name: crdb-client-secure
  labels:
    app.kubernetes.io/component: database
    app.kubernetes.io/instance: crdb-tls-example
    app.kubernetes.io/name: cockroachdb
spec:
  serviceAccountName: cockroach-operator-sa
  containers:
  - name: crdb-client-secure
    image: registry.connect.redhat.com/cockroachdb/cockroach:v20.2.8
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: client-certs
      mountPath: /cockroach/cockroach-certs/
    command:
    - sleep
    - "2147483648" # 2^31
  terminationGracePeriodSeconds: 0
  volumes:
  - name: client-certs
    projected:
        sources:
          - secret:
              name: crdb-tls-example-node
              items:
                - key: ca.crt
                  path: ca.crt
          - secret:
              name: crdb-tls-example-root
              items:
                - key: tls.crt
                  path: client.root.crt
                - key: tls.key
                  path: client.root.key
        defaultMode: 256
  1. On the Pods page, click Create Pod and replace the existing YAML with the above manifest.

  2. Click Create. Return to the Pods page and check that the client pod crdb-client-secure is running. This is also visible on the command-line:

    icon/buttons/copy
    oc get pods
    
    NAME                                  READY   STATUS    RESTARTS   AGE
    cockroach-operator-65c4f6df45-h5r5n   1/1     Running   0          6m11s
    crdb-client-secure                    1/1     Running   0          14m
    crdb-tls-example-0                    1/1     Running   0          3m15s
    crdb-tls-example-1                    1/1     Running   0          103s
    crdb-tls-example-2                    1/1     Running   0          89s
    

Step 5. Use the CockroachDB SQL client

  1. Start the CockroachDB built-in SQL client from the client pod:

    icon/buttons/copy
    oc exec -it crdb-client-secure -- ./cockroach sql --certs-dir=/cockroach/cockroach-certs/ --host=crdb-tls-example-public
    
    #
    # Welcome to the CockroachDB SQL shell.
    # All statements must be terminated by a semicolon.
    # To exit, type: \q.
    #
    # Server version: CockroachDB CCL v20.2.8   (x86_64-unknown-linux-gnu, built 2021/04/23 13:54:57, go1.13.14) (same version as client)
    # Cluster ID: 0813c343-c86b-4be8-9ad0-477cdb5db749
    #
    # Enter \? for a brief introduction.
    #
    root@crdb-tls-example-public:26257/defaultdb>
    

    Now you can run SQL commands against the cluster.

  2. Run some basic CockroachDB SQL statements:

    icon/buttons/copy
    > CREATE DATABASE bank;
    
    icon/buttons/copy
    > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
    
    icon/buttons/copy
    > INSERT INTO bank.accounts VALUES (1, 1000.50);
    
    icon/buttons/copy
    > SELECT * FROM bank.accounts;
    
      id | balance
    +----+---------+
       1 | 1000.50
    (1 row)
    
  3. Create a user with a password:

    icon/buttons/copy
    > CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
    

    You will need this username and password to access the DB Console later.

  4. Exit the SQL shell and pod:

    icon/buttons/copy
    > \q
    

Note: If you cannot access the SQL client, this may be related to your --certs-dir or --host flags.

  1. Shell into the client pod and check for the necessary certs in the --certs-dir directory:

    Tip:

    You can also access the client pod by selecting it on the Pods page and clicking Terminal.

    icon/buttons/copy
    oc exec -it crdb-client-secure sh
    
    icon/buttons/copy
    # cd /cockroach/cockroach-certs
    # ls
    
    ca.crt           client.root.key
    client.root.crt
    
  2. Check the name of the public service to use with the --host flag:

    icon/buttons/copy
    oc get services
    
    icon/buttons/copy
    NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
    crdb-tls-example          ClusterIP   None             <none>        26257/TCP,8080/TCP   14m
    crdb-tls-example-public   ClusterIP   172.25.180.197   <none>        26257/TCP,8080/TCP   14m
    

Step 6. Access the DB Console

To access the CockroachDB cluster's DB Console:

  1. On secure clusters, certain pages of the DB Console can only be accessed by admin users.

    Start the CockroachDB built-in SQL client from the client pod:

    icon/buttons/copy
    oc exec -it crdb-client-secure -- ./cockroach sql --certs-dir=/cockroach/cockroach-certs/ --host=crdb-tls-example-public
    
  2. Assign roach to the admin role (you only need to do this once):

    icon/buttons/copy
    > GRANT admin TO roach;
    
  3. Exit the SQL shell and pod:

    icon/buttons/copy
    > \q
    
  4. In a new terminal window, port-forward from your local machine to the crdb-tls-example-public service:

    icon/buttons/copy
    oc port-forward service/crdb-tls-example-public 8080
    
    Forwarding from [::1]:8080 -> 8080
    
  5. Go to https://localhost:8080 and log in with the username and password you created earlier.

    Note:

    If you are using Google Chrome, and you are getting an error about not being able to reach localhost because its certificate has been revoked, go to chrome://flags/#allow-insecure-localhost, enable "Allow invalid certificates for resources loaded from localhost", and then restart the browser. Enabling this Chrome feature degrades security for all sites running on localhost, not just CockroachDB's DB Console, so be sure to enable the feature only temporarily.

Step 7. Run a sample workload

To run a sample CockroachDB workload:

  1. Use the secure client pod to load the movr schema on one of the CockroachDB pods:

    icon/buttons/copy
    oc exec -it crdb-client-secure -- ./cockroach workload init movr 'postgresql://root@crdb-tls-example-0.crdb-tls-example.cockroachdb:26257?sslcert=%2Fcockroach%2Fcockroach-certs%2Fclient.root.crt&sslkey=%2Fcockroach%2Fcockroach-certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fcockroach%2Fcockroach-certs%2Fca.crt'
    
  2. Initialize and run the workload for 3 minutes:

    icon/buttons/copy
    oc exec -it crdb-client-secure -- ./cockroach workload run movr --duration=3m --tolerate-errors --max-rate=20 --concurrency=1 --display-every=10s 'postgresql://root@crdb-tls-example-0.crdb-tls-example.cockroachdb:26257?sslcert=%2Fcockroach%2Fcockroach-certs%2Fclient.root.crt&sslkey=%2Fcockroach%2Fcockroach-certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fcockroach%2Fcockroach-certs%2Fca.crt'
    
  3. Select one of the CockroachDB pods on the Pods page and click Logs. This will reveal the JDBC URL that your application can use to connect to CockroachDB:

    OpenShift OperatorHub

Step 8. Delete the cluster

Note:

If you want to continue using this cluster, see the documentation on configuring, scaling, monitoring, and upgrading the cluster.

  1. Go to the Installed Operators page and find the cluster name of the CockroachDB cluster. Select Delete CrdbCluster from the menu.

    OpenShift OperatorHub

This will delete the CockroachDB cluster being run by the Operator. It will not delete:

  • The persistent volumes that were attached to the pods. This can be done by deleting the PVCs via Storage > Persistent Volume Claims.
  • The opaque secrets used to authenticate the cluster. This can be done via Workloads > Secrets.
Warning:

If you want to delete the persistent volumes and free up the storage used by CockroachDB, be sure you have a backup copy of your data. Data cannot be recovered once the persistent volumes are deleted. For more information, see the Kubernetes documentation.

Note:

For more information on managing secrets, see the Kubernetes documentation.

YesYes NoNo