> ## Documentation Index
> Fetch the complete documentation index at: https://www.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Essential Alerts for CockroachDB Advanced Deployments

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

## Storage

### Node storage capacity

A CockroachDB node will not able to operate if there is no free disk space on a CockroachDB <InternalLink path="cockroach-start#store">store</InternalLink> volume.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`capacity`</InternalLink>
<br /><InternalLink path="essential-metrics-advanced">`capacity.available`</InternalLink>

**Rule**
<br />Set alerts for each node:
<br />WARNING:  `capacity.available`/`capacity` is less than `0.30` for `24 hours`
<br />CRITICAL:  `capacity.available`/`capacity` is less than `0.10` for `1 hour`

**Action**

* Refer to <InternalLink path="common-issues-to-monitor#storage-capacity">Storage Capacity</InternalLink>.
* Increase the size of CockroachDB node storage capacity.  CockroachDB  storage volumes should not be utilized more than 60% (40% free space).
* In a "disk full" situation, you may be able to get a node "unstuck" by removing the <InternalLink path="cluster-setup-troubleshooting#automatic-ballast-files">automatically created emergency ballast file</InternalLink>.

## SQL

### Node not executing SQL

Send an alert when a node is not executing SQL despite having connections. `sql.conns` shows the number of connections as well as the distribution, or balancing, of connections across cluster nodes. An imbalance can lead to nodes becoming overloaded.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`sql.conns`</InternalLink>
<br />`sql.query.count`

**Rule**
<br />Set alerts for each node:
<br />WARNING:  `sql.conns` greater than `0` while `sql.query.count` equals `0`

**Action**

* Refer to <InternalLink path="connection-pooling">Connection Pooling</InternalLink>.

### SQL query failure

Send an alert when the query failure count exceeds a user-determined threshold based on their application's SLA.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`sql.failure.count`</InternalLink>

**Rule**
<br />WARNING:  `sql.failure.count` is greater than a threshold (based on the user’s application SLA)

**Action**

* Use the <InternalLink path="ui-insights-page">**Insights** page</InternalLink> to find failed executions with their error code to troubleshoot or use application-level logs, if instrumented, to determine the cause of error.

### SQL queries experiencing high latency

Send an alert when the query latency exceeds a user-determined threshold based on their application’s SLA.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`sql.service.latency`</InternalLink>
<br /><InternalLink path="essential-metrics-advanced">`sql.conn.latency`</InternalLink>

**Rule**
<br />WARNING:  (p99 or p90 of `sql.service.latency` plus average of `sql.conn.latency`) is greater than a threshold (based on the user’s application SLA)

**Action**

* Apply the time range of the alert to the <InternalLink path="monitoring-and-alerting#sql-activity-pages">**SQL Activity** pages</InternalLink> to investigate. Use the <InternalLink path="ui-statements-page">**Statements** page</InternalLink> P90 Latency and P99 latency columns to correlate <InternalLink path="ui-statements-page#sql-statement-fingerprints">statement fingerprints</InternalLink> with this alert.

## Changefeeds

During <InternalLink path="upgrade-cockroach-version">rolling maintenance</InternalLink>, <InternalLink path="change-data-capture-overview">changefeed jobs</InternalLink> restart following node restarts. **Mute changefeed alerts described in the following sections during routine maintenance procedures** to avoid unnecessary distractions.

### Changefeed failure

Changefeeds can suffer permanent failures (that the <InternalLink path="monitor-and-debug-changefeeds">jobs system</InternalLink> will not try to restart). Any increase in this metric counter should prompt investigative action.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`changefeed.failures`</InternalLink>

**Rule**
<br />CRITICAL:  If `changefeed.failures` is greater than `0`

**Action**

1. If the alert is triggered during cluster maintenance, mute it. Otherwise start investigation with the following query:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT job_id, status, ((high_water_timestamp/1000000000)::INT::TIMESTAMP) - NOW() AS "changefeed latency", created, LEFT(description, 60), high_water_timestamp FROM crdb_internal.jobs WHERE job_type = 'CHANGEFEED' AND status IN ('running', 'paused', 'pause-requested') ORDER BY created DESC;
   ```

2. If the cluster is not undergoing maintenance, check the health of <InternalLink path="changefeed-sinks">sink</InternalLink> endpoints. If the sink is <InternalLink path="changefeed-sinks#kafka">Kafka</InternalLink>, check for sink connection errors such as `ERROR: connecting to kafka: path.to.cluster:port: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`.

### Frequent changefeed restarts

Changefeeds automatically restart in case of transient errors. However too many restarts outside of a routine maintenance procedure may be due to a systemic condition and should be investigated.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`changefeed.error_retries`</InternalLink>

**Rule**
<br />WARNING:  If `changefeed.error_retries` is greater than `50` for more than `15 minutes`

**Action**

* Follow the action for a [changefeed failure](#changefeed-failure).

### Changefeed falling behind

Changefeed has fallen behind. This is determined by the end-to-end lag between a committed change and that change applied at the destination. This can be due to cluster capacity or changefeed sink availability.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`changefeed.commit_latency`</InternalLink>

**Rule**
<br />WARNING:  `changefeed.commit_latency` is greater than `10 minutes`
<br />CRITICAL:  `changefeed.commit_latency` is greater than `15 minutes`

**Action**

1. In the DB Console, navigate to **Metrics**, <InternalLink path="ui-cdc-dashboard">**Changefeeds** dashboard</InternalLink> for the cluster and check the maximum values on the <InternalLink path="ui-cdc-dashboard#commit-latency">**Commit Latency** graph</InternalLink>. Alternatively, individual changefeed latency can be verified by using the following SQL query:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT job_id, status, ((high_water_timestamp/1000000000)::INT::TIMESTAMP) - NOW() AS "changefeed latency", created, LEFT(description, 60), high_water_timestamp FROM crdb_internal.jobs WHERE job_type = 'CHANGEFEED' AND status IN ('running', 'paused', 'pause-requested') ORDER BY created DESC;
   ```

2. Copy the `job_id` for the changefeed job with highest `changefeed latency` and pause the job:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   PAUSE JOB 681491311976841286;
   ```

3. Check the status of the pause request by running the query from step 1. If the job status is `pause-requested`, check again in a few minutes.

4. After the job is `paused`, resume the job.

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   RESUME JOB 681491311976841286;
   ```

5. If the changefeed latency does not progress after these steps due to lack of cluster resources or availability of the changefeed sink, [contact Support](https://support.cockroachlabs.com).

### Changefeed has been paused a long time

Changefeed jobs should not be paused for a long time because <InternalLink path="protect-changefeed-data">the protected timestamp prevents garbage collection</InternalLink>. To protect against an operational error, this alert guards against an inadvertently forgotten pause.

**Metric**
<br /><InternalLink path="essential-metrics-advanced">`jobs.changefeed.currently_paused`</InternalLink>

**Rule**
<br />WARNING:  `jobs.changefeed.currently_paused` is greater than `0` for more than `15 minutes`
<br />CRITICAL:  `jobs.changefeed.currently_paused` is greater than `0` for more than `60 minutes`

**Action**

1. Check the status of each changefeed using the following SQL query:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT job_id, status, ((high_water_timestamp/1000000000)::INT::TIMESTAMP) - NOW() AS "changefeed latency",created, LEFT(description, 60), high_water_timestamp FROM crdb_internal.jobs WHERE job_type = 'CHANGEFEED' AND status IN ('running', 'paused','pause-requested') ORDER BY created DESC;
   ```

2. If all the changefeeds have status as `running`, one or more changefeeds may have run into an error and recovered. In the DB Console, navigate to **Metrics**, <InternalLink path="ui-cdc-dashboard">**Changefeeds** dashboard</InternalLink> for the cluster and check the <InternalLink path="ui-cdc-dashboard#changefeed-restarts">**Changefeed Restarts** graph</InternalLink>.

3. Resume paused changefeed(s) with the `job_id` using:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   RESUME JOB 681491311976841286;
   ```

### Changefeed experiencing high latency

Send an alert when the maximum latency of any running changefeed exceeds a specified threshold, which is less than the <InternalLink path="configure-replication-zones#replication-zone-variables">`gc.ttlseconds`</InternalLink> variable set in the cluster. This alert ensures that the changefeed progresses faster than the garbage collection TTL, preventing a changefeed's protected timestamp from delaying garbage collection.

**Metric**
<br /><InternalLink path="monitor-and-debug-changefeeds#metrics">`changefeed.checkpoint_progress`</InternalLink>

**Rule**
<br />WARNING: (current time minus `changefeed.checkpoint_progress`) is greater than a threshold (that is less than `gc.ttlseconds` variable)

**Action**

* Refer to <InternalLink path="monitor-and-debug-changefeeds#recommended-changefeed-metrics-to-track">Monitor and Debug Changefeeds</InternalLink>.

## See also

* <InternalLink path="monitoring-and-alerting#events-to-alert-on">Events to alert on</InternalLink>

* <InternalLink path="common-issues-to-monitor">Common Issues to Monitor</InternalLink>

* <InternalLink path="essential-metrics-advanced">Essential Metrics for CockroachDB Advanced Deployments</InternalLink>