> ## Documentation Index
> Fetch the complete documentation index at: https://www.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Physical Cluster Replication Monitoring

export const InlineImage = ({src, alt = "", height = "1.6em"}) => {
  return <img noZoom src={src} alt={alt} style={{
    display: "inline",
    verticalAlign: "start",
    height: height,
    margin: "0"
  }} />;
};

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

export const version = "v23.2";

<Note>
  **This feature is in <InternalLink path="cockroachdb-feature-availability">preview</InternalLink>** and subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com).
</Note>

<InlineImage alt="Megaphone" src="/images/common/icon-megaphone.png" /> New in v23.2: You can monitor a physical cluster replication stream using:

* [`SHOW VIRTUAL CLUSTER... WITH REPLICATION STATUS`](#sql-shell) in the SQL shell.
* The [Physical Replication dashboard](#db-console) on the DB Console.
* [Prometheus and Alertmanager](#prometheus) to track and alert on replication metrics.

When you complete a <InternalLink path="cutover-replication">cutover</InternalLink>, there will be a gap in the primary cluster's metrics whether you are monitoring via the [DB Console](#db-console) or [Prometheus](#prometheus).

The standby cluster will also require separate monitoring to ensure observability during the cutover period. You can use the DB console to track the relevant metrics, or you can use a tool like <InternalLink path="monitor-cockroachdb-with-prometheus#step-5-visualize-metrics-in-grafana">Grafana</InternalLink> to create two separate dashboards, one for each cluster, or a single dashboard with data from both clusters.

## SQL Shell

In the standby cluster's SQL shell, you can query `SHOW VIRTUAL CLUSTER... WITH REPLICATION STATUS` for detail on status and timestamps for planning <InternalLink path="cutover-replication">cutover</InternalLink>:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;
```

Refer to [Responses](#responses) for a description of each field.

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
id |        name        |     data_state     | service_mode | source_tenant_name |                                                     source_cluster_uri                                               | replication_job_id |        replicated_time        |         retained_time         | cutover_time
---+--------------------+--------------------+--------------+--------------------+----------------------------------------------------------------------------------------------------------------------+--------------------+-------------------------------+-------------------------------+---------------
3  | application        | replicating        | none         | application        | postgresql://{user}:{password}@{hostname}:26257?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted | 899090689449132033 | 2023-09-11 22:29:35.085548+00 | 2023-09-11 16:51:43.612846+00 |     NULL
(1 row)
```

### Responses

| Field                  | Response                                                                                                                                                                                                                                                                                                       |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`                   | The ID of a virtual cluster.                                                                                                                                                                                                                                                                                   |
| `name`                 | The name of the standby (destination) virtual cluster.                                                                                                                                                                                                                                                         |
| `data\_state`          | The state of the data on a virtual cluster. This can show one of the following: `initializing replication`, `ready`, `replicating`, `replication paused`, `replication pending cutover`, `replication cutting over`, `replication error`. Refer to [Data state](#data-state) for more detail on each response. |
| `service\_mode`        | The service mode shows whether a virtual cluster is ready to accept SQL requests. This can show one `none` or `shared`. When `shared`, a virtual cluster's SQL connections will be served by the same nodes that are serving the system virtual cluster.                                                       |
| `source\_tenant\_name` | The name of the primary (source) virtual cluster.                                                                                                                                                                                                                                                              |
| `source\_cluster\_uri` | The URI of the primary (source) cluster. The standby cluster connects to the primary cluster using this URI when <InternalLink path="set-up-physical-cluster-replication#step-4-start-replication">starting a replication stream</InternalLink>.                                                               |
| `replication\_job\_id` | The ID of the replication job.                                                                                                                                                                                                                                                                                 |
| `replicated\_time`     | The latest timestamp at which the standby cluser has consistent data — that is, the latest time you can cut over to. This time advances automatically as long as the replication proceeds without error. `replicated\_time` is updated periodically (every `30s`).                                             |
| `retained\_time`       | The earliest timestamp at which the standby cluster has consistent data — that is, the earliest time you can cut over to.                                                                                                                                                                                      |
| `cutover\_time`        | The time at which the cutover will begin. This can be in the past or the future. Refer to <InternalLink path="cutover-replication#cut-over-to-a-point-in-time">Cut over to a point in time</InternalLink>.                                                                                                     |
| `capability\_name`     | The <InternalLink path="create-virtual-cluster#capabilities">capability</InternalLink> name.                                                                                                                                                                                                                   |
| `capability\_value`    | Whether the <InternalLink path="create-virtual-cluster#capabilities">capability</InternalLink> is enabled for a virtual cluster.                                                                                                                                                                               |

#### Data state

| State                         | Description                                                                                                                                                                                                                                                         |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `initializing replication`    | The replication job is completing the initial scan of data from the primary cluster before it starts replicating data in real time.                                                                                                                                 |
| `ready`                       | A virtual cluster's data is ready for use.                                                                                                                                                                                                                          |
| `replicating`                 | The replication job has started and is replicating data.                                                                                                                                                                                                            |
| `replication paused`          | The replication job is paused due to an error or a manual request with <InternalLink path="alter-virtual-cluster">`ALTER VIRTUAL CLUSTER ... PAUSE REPLICATION`</InternalLink>.                                                                                     |
| `replication pending cutover` | The replication job is running and the cutover time has been set. Once the the replication reaches the cutover time, the cutover will begin automatically.                                                                                                          |
| `replication cutting over`    | The job has started cutting over. The cutover time can no longer be changed. Once cutover is complete, A virtual cluster will be available for use with <InternalLink path="alter-virtual-cluster">`ALTER VIRTUAL CLUSTER ... START SERVICE SHARED`</InternalLink>. |
| `replication error`           | An error has occurred. You can find more detail in the error message and the <InternalLink path="configure-logs">logs</InternalLink>. **Note:** A PCR job will retry for 3 minutes before failing.                                                                  |

## DB Console

You can access the <InternalLink path="ui-overview">DB Console</InternalLink> for your standby cluster at `https://{your IP or hostname}:8080/`. Select the **Metrics** page from the left-hand navigation bar, and then select **Physical Cluster Replication** from the **Dashboard** dropdown. The user that accesses the DB Console must have `admin` privileges to view this dashboard.

Use the **Graph** menu to display metrics for your entire cluster or for a specific node.

To the right of the Graph and Dashboard menus, a time interval selector allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected in the <InternalLink path="ui-overview#sql-activity">SQL Activity</InternalLink> pages. However, if you select 10 or 30 minutes, the interval defaults to 1 hour in SQL Activity pages.

Hovering your mouse pointer over the graph title will display a tooltip with a description and the <InternalLink path="essential-metrics-self-hosted">metrics</InternalLink> used to create the graph.

When hovering on graphs, crosshair lines will appear at your mouse pointer. The series' values corresponding to the given time in the cross hairs are displayed in the legend under the graph. Hovering the mouse pointer on a given series displays the corresponding value near the mouse pointer and highlights the series line (graying out other series lines). Click anywhere within the graph to freeze the values in place. Click anywhere within the graph again to cause the values to change with your mouse movements once more.

In the legend, click on an individual series to isolate it on the graph. The other series will be hidden, while the hover will still work. Click the individual series again to make the other series visible. If there are many series, a scrollbar may appear on the right of the legend. This is to limit the size of the legend so that it does not get endlessly large, particularly on clusters with many nodes.

<Note>
  The **Physical Cluster Replication** dashboard tracks metrics related to physical cluster replication jobs. This is distinct from the <InternalLink path="ui-replication-dashboard">**Replication** dashboard</InternalLink>, which tracks metrics related to how data is replicated across the cluster, e.g., range status, replicas per store, and replica quiescence.
</Note>

The **Physical Cluster Replication** dashboard contains graphs for monitoring:

### Logical bytes

<img src="https://mintcdn.com/cockroachlabs/bulMe5iV6qsi6_Jh/images/v23.2/ui-logical-bytes.png?fit=max&auto=format&n=bulMe5iV6qsi6_Jh&q=85&s=4dc7d0e14d51cfb0da379058cbe94abf" alt="DB Console Logical Bytes graph showing results over the past hour" width="1926" height="756" data-path="images/v23.2/ui-logical-bytes.png" />

The **Logical Bytes** graph shows you the throughput of the replicated bytes.

Hovering over the graph displays:

* The date and time.
* The number of logical bytes replicated in MiB.

<Note>
  When you <InternalLink path="set-up-physical-cluster-replication#step-4-start-replication">start a replication stream</InternalLink>, the **Logical Bytes** graph will record a spike of throughput as the initial scan completes.
</Note>

### SST bytes

<img src="https://mintcdn.com/cockroachlabs/bulMe5iV6qsi6_Jh/images/v23.2/ui-sst-bytes.png?fit=max&auto=format&n=bulMe5iV6qsi6_Jh&q=85&s=81dcdc608e157d2efb1b047828f2fdef" alt="DB Console SST bytes graph showing results over the past hour" width="1920" height="748" data-path="images/v23.2/ui-sst-bytes.png" />

The **SST Bytes** graph shows you the rate at which all <InternalLink path="architecture/storage-layer#ssts">SST</InternalLink> bytes are sent to the <InternalLink path="architecture/storage-layer">KV layer</InternalLink> by physical cluster replication jobs.

Hovering over the graph displays:

* The date and time.
* The number of SST bytes replicated in MiB.

## Prometheus

You can use Prometheus and Alertmanager to track and alert on physical cluster replication metrics. Refer to the <InternalLink path="monitor-cockroachdb-with-prometheus">Monitor CockroachDB with Prometheus</InternalLink> tutorial for steps to set up Prometheus.

We recommend tracking the following metrics:

* `physical_replication.logical_bytes`: The logical bytes (the sum of all keys and values) ingested by all physical cluster replication jobs.
* `physical_replication.sst_bytes`: The <InternalLink path="architecture/storage-layer#ssts">SST</InternalLink> bytes (compressed) sent to the KV layer by all physical cluster replication jobs.
* `physical_replication.replicated_time_seconds`: The <InternalLink path="physical-cluster-replication-technical-overview#cutover-and-promotion-process">replicated time</InternalLink> of the physical replication stream in seconds since the Unix epoch.

## Data verification

<Note>
  **This feature is in <InternalLink path="cockroachdb-feature-availability">preview</InternalLink>** and subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com).
</Note>

The `SHOW EXPERIMENTAL_FINGERPRINTS` statement verifies that the data transmission and ingestion is working as expected while a replication stream is running. Any checksum mismatch likely represents corruption or a bug in CockroachDB. Should you encounter such a mismatch, contact [Support](https://support.cockroachlabs.com/hc/en-us).
To verify that the data at a certain point in time is correct on the standby cluster, you can use the <InternalLink path="show-virtual-cluster#responses">current replicated time</InternalLink> from the replication job information to run a point-in-time fingerprint on both the primary and standby clusters. This will verify that the transmission and ingestion of the data on the standby cluster, at that point in time, is correct.

1. Retrieve the current replicated time of the replication job on the standby cluster with <InternalLink path="show-virtual-cluster">`SHOW VIRTUAL CLUSTER`</InternalLink>:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT replicated_time FROM [SHOW VIRTUAL CLUSTER standbyapplication WITH REPLICATION STATUS];
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
        replicated_time
   ----------------------------
   2024-01-09 16:15:45.291575+00
   (1 row)
   ```

   For detail on connecting to the standby cluster, refer to <InternalLink path="set-up-physical-cluster-replication#connect-to-the-standby-cluster-system-virtual-cluster">Set Up Physical Cluster Replication</InternalLink>.
2. From the **primary cluster's system virtual cluster**, specify a timestamp at or earlier than the current `replicated_time` to retrieve the fingerprint. This example uses the current `replicated_time`:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT * FROM [SHOW EXPERIMENTAL_FINGERPRINTS FROM VIRTUAL CLUSTER application] AS OF SYSTEM TIME '2024-01-09 16:15:45.291575+00';
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   tenant_name |             end_ts             |     fingerprint
   ------------+--------------------------------+----------------------
   application | 1704816945291575000.0000000000 | 2646132238164576487
   (1 row)
   ```

   For detail on connecting to the primary cluster, refer to <InternalLink path="set-up-physical-cluster-replication#connect-to-the-primary-cluster-system-virtual-cluster">Set Up Physical Cluster Replication</InternalLink>.
3. From the **standby cluster's system virtual cluster**, specify the same timestamp used on the primary cluster to retrieve the standby cluster's fingerprint:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT * FROM [SHOW EXPERIMENTAL_FINGERPRINTS FROM VIRTUAL CLUSTER standbyapplication] AS OF SYSTEM TIME '2024-01-09 16:15:45.291575+00';
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
       tenant_name     |             end_ts             |     fingerprint
   --------------------+--------------------------------+----------------------
   standbyapplication  | 1704816945291575000.0000000000 | 2646132238164576487
   (1 row)
   ```
4. Compare the fingerprints of the primary and standby clusters to verify the data. The same value for the fingerprints indicates the data is correct.

## See also

* <InternalLink path="ui-overview">DB Console Overview</InternalLink>
* <InternalLink path="monitoring-and-alerting">Monitoring and Alerting</InternalLink>
* <InternalLink path="physical-cluster-replication-overview">Physical Cluster Replication Overview</InternalLink>
* <InternalLink path="cluster-virtualization-overview">Cluster Virtualization Overview</InternalLink>