> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Technical Advisory 106617

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

Publication date: August 15, 2023

## Description

A CockroachDB store with [encryption-at-rest](https://www.cockroachlabs.com/docs/v23.1/security-reference/encryption#encryption-at-rest) enabled maintains per-file metadata necessary for file decryption within an append-only log. When this log file grows large, the log file is rotated by writing a snapshot of the metadata to a new, replacement log file. The introduction of this log file in <InternalLink version="releases" path="v21.2">v21.2.0</InternalLink> contains a bug in log rotation that could result in the omission of a single file’s metadata from the persisted log file. When a node encounters this bug, it continues to operate normally until the process is restarted. If the omitted file still exists after a process restart, the [`cockroach` process](https://www.cockroachlabs.com/docs/v23.1/cockroach-node) cannot decrypt or read the file, and the local store appears to be corrupted. This bug most frequently causes the `cockroach` process on a node to exit shortly after it starts. An error such as “`pebble/table: invalid table (bad magic number: 0x08cf6caea91de887)`” is logged.

Only CockroachDB deployments using encryption-at-rest are affected. This issue is rare, given that the average lifespan of a file within CockroachDB’s [storage engine](https://www.cockroachlabs.com/docs/v23.1/architecture/storage-layer) is relatively short. If a file is omitted from the registry during a rotation, the file is frequently removed before the process is restarted.

The corruption is limited to a single store, allowing recovery through [decommissioning](https://www.cockroachlabs.com/docs/v23.1/node-shutdown?filters=decommission) and [replication](https://www.cockroachlabs.com/docs/v23.1/architecture/replication-layer). The restart of multiple processes simultaneously or in quick succession increases the risk of quorum loss through the simultaneous loss of nodes.

## Statement

This is resolved by a change in CockroachDB which fixes the ordering of steps during encryption-at-rest log rotation.

The fix has been applied to maintenance releases of CockroachDB <InternalLink version="releases" path="v23.1#v23-1-8">v23.1.8</InternalLink>, <InternalLink version="releases" path="v22.2">v22.2.13</InternalLink>, v22.1.22.

## Mitigation

Users of CockroachDB v21.2, ≤22.1.21, ≤22.2.12, ≤23.1.7 using [encryption-at-rest](https://www.cockroachlabs.com/docs/v23.1/security-reference/encryption#encryption-at-rest) are encouraged to upgrade to v22.1.22, v22.2.13, v23.1.8 or a later version. Users are encouraged to deploy the patch release within a best practice “rolling upgrade.” Nodes should be upgraded one at a time, pausing 15 minutes between nodes.

If a node encounters corruption, manifested as repeated node start failures, users should:

1. Pause the rolling upgrade.
2. Decommission the node with corruption.
3. Wait for all ranges to fully replicate.
4. Resume the rolling upgrade.

## Impact

Restarting a node with a store encrypted using [encryption-at-rest](https://www.cockroachlabs.com/docs/v23.1/security-reference/encryption#encryption-at-rest) may surface existing latent, local corruption of the store’s data, requiring the node to be decommissioned and replaced. Customers are encouraged to upgrade to a patched version through a rolling upgrade.

Questions about any technical alert can be directed to our [support team](https://support.cockroachlabs.com/).