Monitor CockroachDB on Kubernetes

Warning:
Cockroach Labs will stop providing Assistance Support for this version on November 18, 2022. Prior to that date, upgrade to a more recent version to continue receiving support. For more details, see the Release Support Policy.
Note:

This article assumes you have already deployed CockroachDB on a single Kubernetes cluster.

Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.

Configure Prometheus

Every node of a CockroachDB cluster exports granular timeseries metrics formatted for easy integration with Prometheus, an open source tool for storing, aggregating, and querying timeseries data. This section shows you how to orchestrate Prometheus as part of your Kubernetes cluster and pull these metrics into Prometheus for external monitoring.

This guidance is based on CoreOS's Prometheus Operator, which allows a Prometheus instance to be managed using built-in Kubernetes concepts.

Note:

If you're on Hosted GKE, before starting, make sure the email address associated with your Google Cloud account is part of the cluster-admin RBAC group, as shown in Deploy CockroachDB with Kubernetes.

  1. From your local workstation, edit the cockroachdb service to add the prometheus: cockroachdb label:

    icon/buttons/copy
    $ kubectl label svc cockroachdb prometheus=cockroachdb
    
    service/cockroachdb labeled
    

    This ensures that only the cockroachdb (not the cockroach-public service) is being monitored by a Prometheus job.

    icon/buttons/copy
    $ kubectl label svc cockroachdb prometheus=cockroachdb
    
    service/cockroachdb labeled
    

    This ensures that only the cockroachdb (not the cockroach-public service) is being monitored by a Prometheus job.

    icon/buttons/copy
    $ kubectl label svc my-release-cockroachdb prometheus=cockroachdb
    
    service/my-release-cockroachdb labeled
    

    This ensures that there is a Prometheus job and monitoring data only for the my-release-cockroachdb service, not for the my-release-cockroach-public service.

  2. Install CoreOS's Prometheus Operator:

    Note:

    We recommend checking for the latest Prometheus Operator release version and specifying this version in the following command.

    icon/buttons/copy
    $ kubectl apply \
    -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.47.1/bundle.yaml
    
    customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
    clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator configured
    clusterrole.rbac.authorization.k8s.io/prometheus-operator configured
    deployment.apps/prometheus-operator created
    serviceaccount/prometheus-operator configured
    service/prometheus-operator created
    
  3. Confirm that the prometheus-operator has started:

    icon/buttons/copy
    $ kubectl get deploy prometheus-operator
    
    NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
    prometheus-operator   1/1     1            1           27s
    
  4. Download our Prometheus manifest:

    icon/buttons/copy
    $ curl -O https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/prometheus.yaml
    
    Note:

    By default, this manifest uses the secret name generated by the CockroachDB Kubernetes Operator. If you generated your own certificates and keys when starting CockroachDB, be sure that ca.secret.name matches the name of the node secret you created.

  5. Apply the Prometheus manifest. This creates the various objects necessary to run a Prometheus instance:

    icon/buttons/copy
    $ kubectl apply -f prometheus.yaml
    
    serviceaccount/prometheus created
    clusterrole.rbac.authorization.k8s.io/prometheus created
    clusterrolebinding.rbac.authorization.k8s.io/prometheus created
    servicemonitor.monitoring.coreos.com/cockroachdb created
    prometheus.monitoring.coreos.com/cockroachdb created
    
  6. Access the Prometheus UI locally and verify that CockroachDB is feeding data into Prometheus:

    1. Port-forward from your local machine to the pod running Prometheus:

      icon/buttons/copy
      $ kubectl port-forward prometheus-cockroachdb-0 9090
      
    2. Go to http://localhost:9090 in your browser.

    3. To verify that each CockroachDB node is connected to Prometheus, go to Status > Targets. The screen should look like this:

      Prometheus targets

    4. To verify that data is being collected, go to Graph, enter the sys_uptime variable in the field, click Execute, and then click the Graph tab. The screen should like this:

      Prometheus graph

    Tip:

    Prometheus auto-completes CockroachDB time series metrics for you, but if you want to see a full listing, with descriptions, port-forward as described in Access the DB Console and then point your browser to http://localhost:8080/_status/vars.

    For more details on using the Prometheus UI, see their official documentation.

Configure Alertmanager

Active monitoring helps you spot problems early, but it is also essential to send notifications when there are events that require investigation or intervention. This section shows you how to use Alertmanager and CockroachDB's starter alerting rules to do this.

  1. Download our alertmanager-config.yaml configuration file:

    icon/buttons/copy
    $ curl -O \
    https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/alertmanager-config.yaml
    
  2. Edit the alertmanager-config.yaml file to specify the desired receivers for notifications. Initially, the file contains a placeholder web hook.

  3. Add this configuration to the Kubernetes cluster as a secret, renaming it to alertmanager.yaml and labelling it to make it easier to find:

    icon/buttons/copy
    $ kubectl create secret generic alertmanager-cockroachdb \
    --from-file=alertmanager.yaml=alertmanager-config.yaml
    
    secret/alertmanager-cockroachdb created
    
    icon/buttons/copy
    $ kubectl label secret alertmanager-cockroachdb app=cockroachdb
    
    secret/alertmanager-cockroachdb labeled
    
    Warning:

    The name of the secret, alertmanager-cockroachdb, must match the name used in the alertmanager.yaml file. If they differ, the Alertmanager instance will start without configuration, and nothing will happen.

  4. Use our alertmanager.yaml file to create the various objects necessary to run an Alertmanager instance, including a ClusterIP service so that Prometheus can forward alerts:

    icon/buttons/copy
    $ kubectl apply \
    -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/alertmanager.yaml
    
    alertmanager.monitoring.coreos.com/cockroachdb created
    service/alertmanager-cockroachdb created
    
  5. Verify that Alertmanager is running:

    1. Port-forward from your local machine to the pod running Alertmanager:

      icon/buttons/copy
      $ kubectl port-forward alertmanager-cockroachdb-0 9093
      
    2. Go to http://localhost:9093 in your browser. The screen should look like this:

      Alertmanager

  6. Ensure that the Alertmanagers are visible to Prometheus by opening http://localhost:9090/status. The screen should look like this:

    Alertmanager

  7. Add CockroachDB's starter alerting rules:

    icon/buttons/copy
    $ kubectl apply \
    -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/prometheus/alert-rules.yaml
    
    prometheusrule.monitoring.coreos.com/prometheus-cockroachdb-rules created
    
  8. Ensure that the rules are visible to Prometheus by opening http://localhost:9090/rules. The screen should look like this:

    Alertmanager

  9. Verify that the TestAlertManager example alert is firing by opening http://localhost:9090/alerts. The screen should look like this:

    Alertmanager

  10. To remove the example alert:

    1. Use the kubectl edit command to open the rules for editing:

      icon/buttons/copy
      $ kubectl edit prometheusrules prometheus-cockroachdb-rules
      
    2. Remove the dummy.rules block and save the file:

      - name: rules/dummy.rules
        rules:
        - alert: TestAlertManager
          expr: vector(1)
      

Yes No