> ## Documentation Index
> Fetch the complete documentation index at: https://www.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cut Over from a Primary Cluster to a Standby Cluster

export const InlineImage = ({src, alt = "", height = "1.6em"}) => {
  return <img noZoom src={src} alt={alt} style={{
    display: "inline",
    verticalAlign: "start",
    height: height,
    margin: "0"
  }} />;
};

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

export const version = "v23.2";

<Note>
  **This feature is in <InternalLink path="cockroachdb-feature-availability">preview</InternalLink>** and subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com).
</Note>

Physical cluster replication is supported in CockroachDB self-hosted clusters.
<InlineImage alt="Megaphone" src="/images/common/icon-megaphone.png" /> New in v23.2: *Cutover* in <InternalLink path="physical-cluster-replication-overview">**physical cluster replication (PCR)**</InternalLink> allows you to switch from the active primary cluster to the passive standby cluster that has ingested replicated data. When you complete the replication stream to initiate a cutover, the job stops replicating data from the primary, sets the standby <InternalLink path="physical-cluster-replication-technical-overview">virtual cluster</InternalLink> to a point in time (in the past or future) where all ingested data is consistent, and then makes the standby virtual cluster ready to accept traffic.

*Cutback* using a new PCR stream switches operations back to the original primary cluster (or a new cluster) after a cutover event.

This page describes:

* [**Cutover**](#cutover) from the primary cluster to the standby cluster.
* [**Cutback**](#cut-back-to-the-primary-cluster) from the original standby cluster (after it was promoted during cutover) to the original primary cluster.
* [**Job management**](#job-management) after a cutover or cutback.

<Danger>
  Cutover and cutback do **not** redirect traffic automatically to the standby cluster. Once the cutover or cutback is complete, you must redirect application traffic to the standby (new) cluster. If you do not redirect traffic manually, writes to the primary (original) cluster may be lost.
</Danger>

## Cutover

The cutover is a two-step process on the standby cluster:

1. [Initiating the cutover](#step-1-initiate-the-cutover).
2. [Completing the cutover](#step-2-complete-the-cutover).

### Before you begin

During PCR, jobs running on the primary cluster will replicate to the standby cluster. Before you cut over to the standby cluster, or cut back to the original primary cluster, consider how you will manage running (replicated) jobs between the clusters. Refer to [Job management](#job-management) for instructions.

### Step 1. Initiate the cutover

To initiate a cutover to the standby cluster, you can specify the point in time for the standby's promotion. That is, the standby cluster's live data at the point of cutover. Refer to the following sections for steps:

* [`LATEST`](#cut-over-to-the-most-recent-replicated-time): The most recent replicated timestamp.
* [Point-in-time](#cut-over-to-a-point-in-time):
  * Past: A past timestamp within the <InternalLink path="physical-cluster-replication-technical-overview#cutover-and-promotion-process">cutover window</InternalLink>.
  * Future: A future timestamp for planning a cutover.

#### Cut over to the most recent replicated time

To initiate a cutover to the most recent replicated timestamp, you can specify `LATEST` when you start the cutover. The latest replicated time may be behind the actual time if there is <InternalLink path="physical-cluster-replication-technical-overview#cutover-and-promotion-process">*replication lag*</InternalLink> in the stream. Replication lag is the time between the most up-to-date replicated time and the actual time.

1. To view the current replication timestamp, use:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   id |     name           | data_state  | service_mode | source_tenant_name |                                                  source_cluster_uri                                 | replication_job_id |       replicated_time        |         retained_time         | cutover_time
   -----+--------------------+-------------+--------------+--------------------+-----------------------------------------------------------------------------------------------------+--------------------+------------------------------+-------------------------------+---------------
   5 | application        | replicating | none         | application        | postgresql://user:redacted@host?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted | 911803003607220225 | 2023-10-26 17:36:52.27978+00 | 2023-10-26 14:36:52.279781+00 |         NULL
   ```
2. Run the following from the standby cluster's SQL shell to start the cutover:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ALTER VIRTUAL CLUSTER application COMPLETE REPLICATION TO LATEST;
   ```

   The `cutover_time` is the timestamp at which the replicated data is consistent. The cluster will revert any replicated data above this timestamp to ensure that the standby is consistent with the primary at that timestamp:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
           cutover_time
   ----------------------------------
   1695922878030920020.0000000000
   (1 row)
   ```

#### Cut over to a point in time

You can control the point in time that the PCR stream will cut over to.

1. To select a <InternalLink path="as-of-system-time">specific time</InternalLink> in the past, use:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;
   ```

   The `retained_time` response provides the earliest time to which you can cut over.

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   id |        name        |     data_state     | service_mode | source_tenant_name |                                                     source_cluster_uri                                               | replication_job_id |        replicated_time        |         retained_time         | cutover_time
   ---+--------------------+--------------------+--------------+--------------------+----------------------------------------------------------------------------------------------------------------------+--------------------+-------------------------------+-------------------------------+---------------
   3  | application        | replicating        | none         | application        | postgresql://{user}:redacted@{hostname}:26257?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted    | 899090689449132033 | 2023-09-11 22:29:35.085548+00 | 2023-09-11 16:51:43.612846+00 |     NULL
   (1 row)
   ```
2. Specify a timestamp:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ALTER VIRTUAL CLUSTER application COMPLETE REPLICATION TO SYSTEM TIME '-1h';
   ```

   Refer to <InternalLink path="as-of-system-time#using-different-timestamp-formats">Using different timestamp formats</InternalLink> for more information.

   Similarly, to cut over to a specific time in the future:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ALTER VIRTUAL CLUSTER application COMPLETE REPLICATION TO SYSTEM TIME '+5h';
   ```

   A future cutover will proceed once the replicated data has reached the specified time.

To monitor for when the replication stream completes, use <InternalLink path="show-virtual-cluster">`SHOW VIRTUAL CLUSTER... WITH REPLICATION STATUS`</InternalLink> to find the replication stream's `replication_job_id`, which you can pass to `SHOW JOB WHEN COMPLETE job_id` as the `job_id`. Refer to the `SHOW JOBS` page for <InternalLink path="show-jobs#parameters">details</InternalLink> and an <InternalLink path="show-jobs#show-job-when-complete">example</InternalLink>.

### Step 2. Complete the cutover

1. The completion of the replication is asynchronous; to monitor its progress use:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SHOW VIRTUAL CLUSTER application WITH REPLICATION STATUS;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     id |        name         |         data_state          | service_mode | source_tenant_name |                                                source_cluster_uri                                                   | replication_job_id |       replicated_time        |         retained_time         |          cutover_time
   -----+---------------------+-----------------------------+--------------+--------------------+---------------------------------------------------------------------------------------------------------------------+--------------------+------------------------------+-------------------------------+---------------------------------
     4  | application         | replication pending cutover | none         | application        | postgresql://{user}:{password}@{hostname}:26257?options=-ccluster%3Dsystem&sslmode=verify-full&sslrootcert=redacted | 903895265809498113 | 2023-09-28 17:41:18.03092+00 | 2023-09-28 16:09:04.327473+00 | 1695922878030920020.0000000000
   (1 row)
   ```

   Refer to <InternalLink path="physical-cluster-replication-monitoring">Physical Cluster Replication Monitoring</InternalLink> for the <InternalLink path="physical-cluster-replication-monitoring#responses">Responses</InternalLink> and <InternalLink path="physical-cluster-replication-monitoring#data-state">Data state</InternalLink> of `SHOW VIRTUAL CLUSTER... WITH REPLICATION STATUS` fields.
2. Once complete, bring the standby's virtual cluster online with:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   ALTER VIRTUAL CLUSTER application START SERVICE SHARED;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     id |        name         |     data_state     | service_mode
   -----+---------------------+--------------------+---------------
     1  | system              | ready              | shared
     2  | template            | ready              | none
     3  | application         | ready              | shared
   (3 rows)
   ```
3. To make the standby's virtual cluster the default for connection strings, set the following <InternalLink path="cluster-settings">cluster setting</InternalLink>:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SET CLUSTER SETTING server.controller.default_target_cluster='application';
   ```

At this point, the primary and standby clusters are entirely independent. You will need to use your own network load balancers, DNS servers, or other network configuration to direct application traffic to the standby (now primary). To manage replicated jobs on the promoted standby, refer to [Job management](#job-management).

To enable PCR again, from the new primary to the original primary (or a completely different cluster), refer to [Cut back to the primary cluster](#cut-back-to-the-primary-cluster).

## Cut back to the primary cluster

After cutting over to the standby cluster, you may need to move back to the original primary cluster, or a completely different cluster. This process is manual and requires starting a new PCR stream.

For example, if you had <InternalLink path="set-up-physical-cluster-replication">set up PCR</InternalLink> between a primary and standby cluster and then cut over to the standby, the workflow to cut back to the original primary cluster would be as follows:

* Original primary cluster = Cluster A
* Original standby cluster = Cluster B

1. Cluster B is now serving application traffic after the cutover.
2. Drop the application virtual cluster from the cluster A with <InternalLink path="drop-virtual-cluster">`DROP VIRTUAL CLUSTER`</InternalLink>.
3. Start a PCR stream that sends updates from cluster B to cluster A. Refer to <InternalLink path="set-up-physical-cluster-replication#step-4-start-replication">Start replication</InternalLink>.

At this point, **Cluster A** is once again the primary and **Cluster B** is once again the standby. The clusters are entirely independent. To direct application traffic to the primary (**Cluster A**), you will need to use your own network load balancers, DNS servers, or other network configuration to direct application traffic to **Cluster A**. To manage replicated jobs on the promoted standby, refer to [Job management](#job-management).

To enable physical cluster replication again, from the primary to the standby (or a completely different cluster), refer to <InternalLink path="set-up-physical-cluster-replication">Set Up Physical Cluster Replication</InternalLink>.

## Job management

During a replication stream, jobs running on the primary cluster will replicate to the standby cluster. Once you have [completed a cutover](#step-2-complete-the-cutover) (or a [cutback](#cut-back-to-the-primary-cluster)), refer to the following sections for details on resuming jobs on the promoted cluster.

### Backup schedules

<InternalLink path="manage-a-backup-schedule">Backup schedules</InternalLink> will pause after cutover on the promoted cluster. Take the following steps to resume jobs:

1. Verify that there are no other schedules running backups to the same <InternalLink path="take-full-and-incremental-backups#backup-collections">collection of backups</InternalLink>, i.e., the schedule that was running on the original primary cluster.
2. Resume the backup schedule on the promoted cluster.

<Note>
  If your backup schedule was created on a cluster in v23.1 or earlier, it will **not** pause automatically on the promoted cluster after cutover. In this case, you must pause the schedule manually on the promoted cluster and then take the outlined steps.
</Note>

### Changefeeds

<InternalLink path="change-data-capture-overview">Changefeeds</InternalLink> will fail on the promoted cluster immediately after cutover to avoid two clusters running the same changefeed to one sink. We recommend that you recreate changefeeds on the promoted cluster.

<InternalLink path="create-schedule-for-changefeed">Scheduled changefeeds</InternalLink> will continue on the promoted cluster. You will need to manage <InternalLink path="pause-schedules">pausing</InternalLink> or <InternalLink path="drop-schedules">canceling</InternalLink> the schedule on the promoted standby cluster to avoid two clusters running the same changefeed to one sink.

## See also

* <InternalLink path="physical-cluster-replication-overview">Physical Cluster Replication Overview</InternalLink>
* <InternalLink path="physical-cluster-replication-technical-overview">Physical Cluster Replication Technical Overview</InternalLink>