Resource management

This page explains how to configure Kubernetes cluster resources such as memory, CPU, and storage.

These settings override the defaults used when deploying CockroachDB on Kubernetes.

Note:

All kubectl steps should be performed in the namespace where you installed the Operator. By default, this is cockroach-operator-system.

Tip:

If you deployed CockroachDB on Red Hat OpenShift, substitute kubectl with oc in the following commands.

On a production cluster, the resources you allocate to CockroachDB should be proportionate to your machine types and workload. We recommend that you determine and set these values before deploying the cluster, but you can also update the values on a running cluster.

Tip:

Run kubectl describe nodes to see the available resources on the instances that you have provisioned.

Memory and CPU

You can set the CPU and memory resources allocated to the CockroachDB container on each pod.

Note:

1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, see the Production Checklist.

Specify CPU and memory values in resources.requests and resources.limits in the Operator's custom resource, which is used to deploy the cluster:

spec:
  resources:
    requests:
      cpu: "4"
      memory: "16Gi"
    limits:
      cpu: "4"
      memory: "16Gi"

Apply the new settings to the cluster:

icon/buttons/copy
$ kubectl apply -f example.yaml

Specify CPU and memory values in resources.requests and resources.limits in the StatefulSet manifest you used to deploy the cluster:

spec:
  template:
    containers:
    - name: cockroachdb
      resources:
        requests:
          cpu: "4"
          memory: "16Gi"
        limits:
          cpu: "4"
          memory: "16Gi"

Apply the new settings to the cluster:

icon/buttons/copy
$ kubectl apply -f {statefulset-manifest}.yaml

Specify CPU and memory values in resources.requests and resources.limits in the custom values file you created when deploying the cluster:

statefulset:
  resources:
    limits:
      cpu: "4"
      memory: "16Gi"
    requests:
      cpu: "4"
      memory: "16Gi"

Apply the custom values to override the default Helm chart values:

icon/buttons/copy
$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb

We recommend using identical values for resources.requests and resources.limits. When setting the new values, note that not all of a pod's resources will be available to the CockroachDB container. This is because a fraction of the CPU and memory is reserved for Kubernetes.

Note:

If no resource limits are specified, the pods will be able to consume the maximum available CPUs and memory. However, to avoid overallocating resources when another memory-intensive workload is on the same instance, always set resource requests and limits explicitly.

For more information on how Kubernetes handles resources, see the Kubernetes documentation.

Cache and SQL memory size

Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the Production Checklist.

Our Kubernetes manifests dynamically set cache size and SQL memory size each to 1/4 (the recommended fraction) of the available memory, which depends on the memory request and limit you specified for your configuration. If you want to customize these values, set them explicitly.

Specify cache and maxSQLMemory in the Operator's custom resource, which is used to deploy the cluster:

spec:
  cache: "4Gi"
  maxSQLMemory: "4Gi"

Apply the new settings to the cluster:

icon/buttons/copy
$ kubectl apply -f example.yaml
Note:

Specifying these values is equivalent to using the --cache and --max-sql-memory flags with cockroach start.

Cache and SQL memory size

Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the Production Checklist.

Our Kubernetes manifests dynamically set cache size and SQL memory size each to 1/4 (the recommended fraction) of the available memory, which depends on the memory request and limit you specified for your configuration. If you want to customize these values, set them explicitly.

Specify cache and maxSQLMemory in the custom values file you created when deploying the cluster:

conf:
  cache: "4Gi"
  max-sql-memory: "4Gi"

Apply the custom values to override the default Helm chart values:

icon/buttons/copy
$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb

Persistent storage

When you start your cluster, Kubernetes dynamically provisions and mounts a persistent volume into each pod. For more information on persistent volumes, see the Kubernetes documentation.

The storage capacity of each volume is set in pvc.spec.resources in the Operator's custom resource, which is used to deploy the cluster:

spec:
  dataStore:
    pvc:
      spec:
        resources:
          limits:
            storage: "60Gi"
          requests:
            storage: "60Gi"

The storage capacity of each volume is initially set in volumeClaimTemplates.spec.resources in the StatefulSet manifest you used to deploy the cluster:

volumeClaimTemplates:
  spec:
    resources:
      requests:
        storage: 100Gi

The storage capacity of each volume is initially set in the Helm chart's values file:

persistentVolume:
  size: 100Gi

You should provision an appropriate amount of disk storage for your workload. For recommendations on this, see the Production Checklist.

Expand disk size

If you discover that you need more capacity, you can expand the persistent volumes on a running cluster. Increasing disk size is often beneficial for CockroachDB performance.

Specify a new volume size in resources.requests and resources.limits in the Operator's custom resource, which is used to deploy the cluster:

spec:
  dataStore:
    pvc:
      spec:
        resources:
          limits:
            storage: "100Gi"
          requests:
            storage: "100Gi"

Apply the new settings to the cluster:

icon/buttons/copy
$ kubectl apply -f example.yaml

The Operator updates the StatefulSet and triggers a rolling restart of the pods with the new storage capacity.

To verify that the storage capacity has been updated, run kubectl get pvc to view the persistent volume claims (PVCs). It will take a few minutes before the PVCs are completely updated.

You can expand certain types of persistent volumes (including GCE Persistent Disk and Amazon Elastic Block Store) by editing their persistent volume claims.

Note:

These steps assume you followed the tutorial Deploy CockroachDB on Kubernetes.

  1. Get the persistent volume claims for the volumes:

    icon/buttons/copy
    $ kubectl get pvc
    
    NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    datadir-cockroachdb-0   Bound    pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       17m
    datadir-cockroachdb-1   Bound    pvc-75e143ca-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       17m
    datadir-cockroachdb-2   Bound    pvc-75ef409a-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       17m
    
  2. In order to expand a persistent volume claim, AllowVolumeExpansion in its storage class must be true. Examine the storage class:

    icon/buttons/copy
    $ kubectl describe storageclass standard
    
    Name:                  standard
    IsDefaultClass:        Yes
    Annotations:           storageclass.kubernetes.io/is-default-class=true
    Provisioner:           kubernetes.io/gce-pd
    Parameters:            type=pd-standard
    AllowVolumeExpansion:  False
    MountOptions:          <none>
    ReclaimPolicy:         Delete
    VolumeBindingMode:     Immediate
    Events:                <none>
    

    If necessary, edit the storage class:

    icon/buttons/copy
    $ kubectl patch storageclass standard -p '{"allowVolumeExpansion": true}'
    
    storageclass.storage.k8s.io/standard patched
    
  3. Edit one of the persistent volume claims to request more space:

    Note:

    The requested storage value must be larger than the previous value. You cannot use this method to decrease the disk size.

    icon/buttons/copy
    $ kubectl patch pvc datadir-cockroachdb-0 -p '{"spec": {"resources": {"requests": {"storage": "200Gi"}}}}'
    
    persistentvolumeclaim/datadir-cockroachdb-0 patched
    
  4. Check the capacity of the persistent volume claim:

    icon/buttons/copy
    $ kubectl get pvc datadir-cockroachdb-0
    
    NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    datadir-cockroachdb-0   Bound    pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       18m
    

    If the PVC capacity has not changed, this may be because AllowVolumeExpansion was initially set to false or because the volume has a file system that has to be expanded. You will need to start or restart a pod in order to have it reflect the new capacity.

    Tip:

    Running kubectl get pv will display the persistent volumes with their requested capacity and not their actual capacity. This can be misleading, so it's best to use kubectl get pvc.

  5. Examine the persistent volume claim. If the volume has a file system, you will see a FileSystemResizePending condition with an accompanying message:

    icon/buttons/copy
    $ kubectl describe pvc datadir-cockroachdb-0
    
    Waiting for user to (re-)start a pod to finish file system resize of volume on node.
    
  6. Delete the corresponding pod to restart it:

    icon/buttons/copy
    $ kubectl delete pod cockroachdb-0
    

    The FileSystemResizePending condition and message will be removed.

  7. View the updated persistent volume claim:

    icon/buttons/copy
    $ kubectl get pvc datadir-cockroachdb-0
    
    NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    datadir-cockroachdb-0   Bound    pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb   200Gi      RWO            standard       20m
    
  8. The CockroachDB cluster needs to be expanded one node at a time. Repeat steps 3 - 6 to increase the capacities of the remaining volumes by the same amount.

You can expand certain types of persistent volumes (including GCE Persistent Disk and Amazon Elastic Block Store) by editing their persistent volume claims.

Note:

These steps assume you followed the tutorial Deploy CockroachDB on Kubernetes.

  1. Get the persistent volume claims for the volumes:

    icon/buttons/copy
    $ kubectl get pvc
    
    NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    datadir-my-release-cockroachdb-0   Bound    pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       17m
    datadir-my-release-cockroachdb-1   Bound    pvc-75e143ca-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       17m
    datadir-my-release-cockroachdb-2   Bound    pvc-75ef409a-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       17m
    
  2. In order to expand a persistent volume claim, AllowVolumeExpansion in its storage class must be true. Examine the storage class:

    icon/buttons/copy
    $ kubectl describe storageclass standard
    
    Name:                  standard
    IsDefaultClass:        Yes
    Annotations:           storageclass.kubernetes.io/is-default-class=true
    Provisioner:           kubernetes.io/gce-pd
    Parameters:            type=pd-standard
    AllowVolumeExpansion:  False
    MountOptions:          <none>
    ReclaimPolicy:         Delete
    VolumeBindingMode:     Immediate
    Events:                <none>
    

    If necessary, edit the storage class:

    icon/buttons/copy
    $ kubectl patch storageclass standard -p '{"allowVolumeExpansion": true}'
    
    storageclass.storage.k8s.io/standard patched
    
  3. Edit one of the persistent volume claims to request more space:

    Note:

    The requested storage value must be larger than the previous value. You cannot use this method to decrease the disk size.

    icon/buttons/copy
    $ kubectl patch pvc datadir-my-release-cockroachdb-0 -p '{"spec": {"resources": {"requests": {"storage": "200Gi"}}}}'
    
    persistentvolumeclaim/datadir-my-release-cockroachdb-0 patched
    
  4. Check the capacity of the persistent volume claim:

    icon/buttons/copy
    $ kubectl get pvc datadir-my-release-cockroachdb-0
    
    NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    datadir-my-release-cockroachdb-0   Bound    pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb   100Gi      RWO            standard       18m
    

    If the PVC capacity has not changed, this may be because AllowVolumeExpansion was initially set to false or because the volume has a file system that has to be expanded. You will need to start or restart a pod in order to have it reflect the new capacity.

    Tip:

    Running kubectl get pv will display the persistent volumes with their requested capacity and not their actual capacity. This can be misleading, so it's best to use kubectl get pvc.

  5. Examine the persistent volume claim. If the volume has a file system, you will see a FileSystemResizePending condition with an accompanying message:

    icon/buttons/copy
    $ kubectl describe pvc datadir-my-release-cockroachdb-0
    
    Waiting for user to (re-)start a pod to finish file system resize of volume on node.
    
  6. Delete the corresponding pod to restart it:

    icon/buttons/copy
    $ kubectl delete pod my-release-cockroachdb-0
    

    The FileSystemResizePending condition and message will be removed.

  7. View the updated persistent volume claim:

    icon/buttons/copy
    $ kubectl get pvc datadir-my-release-cockroachdb-0
    
    NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    datadir-my-release-cockroachdb-0   Bound    pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb   200Gi      RWO            standard       20m
    
  8. The CockroachDB cluster needs to be expanded one node at a time. Repeat steps 3 - 6 to increase the capacities of the remaining volumes by the same amount.

Network ports

The Operator separates network traffic into three ports:

Protocol Default Description Custom Resource Field
gRPC 26258 Used for node connections grpcPort
HTTP 8080 Used to access the DB Console httpPort
SQL 26257 Used for SQL shell access sqlPort

Specify alternate port numbers in the Operator's custom resource (for example, to match the default port 5432 on PostgreSQL):

spec:
  sqlPort: 5432

Apply the new settings to the cluster:

icon/buttons/copy
$ kubectl apply -f example.yaml

The Operator updates the StatefulSet and triggers a rolling restart of the pods with the new port settings.

Warning:

Currently, only the pods are updated with new ports. To connect to the cluster, you need to ensure that the public service is also updated to use the new port. You can do this by deleting the service with kubectl delete service {cluster-name}-public. When service is recreated by the Operator, it will use the new port. This is a known limitation that will be fixed in an Operator update.

YesYes NoNo