Technical Advisory 88042

Publication date: October 24, 2022

Description

In CockroachDB v22.1.0 to v22.1.8 and v21.2.0 to v21.2.16, a RESTORE of an incremental backup may include rows that should not be restored, in a narrow set of circumstances relating to an ongoing IMPORT INTO job. This has been fixed in CockroachDB patch versions v22.1.9 and v21.2.17, available in Releases.

This issue primarily affects incremental backups taken after an IMPORT INTO operation is reverted. For this issue to occur, the following series of conditions need to occur:

1. First, a table must be backed up, typically as part of a full-cluster backup, while it is offline and being written to by an ongoing IMPORT INTO job. Rows which are being written to the table by the in-progress IMPORT INTO job are included in this backup.
2. Next, that IMPORT INTO job must fail or be canceled. This failure or cancellation triggers the roll-back process, which reverts rows which it had written, and then brings the table back online.
3. After the IMPORT INTO rollback completes and the table is back online, an incremental backup must be taken of the table. This incremental backup is appended to the original backup that was taken while the table was offline.
4. When this incremental backup is restored using an affected CockroachDB version, rows which were initially written by the IMPORT and backed up, but then later reverted when the IMPORT INTO was canceled before the incremental backup, will incorrectly appear in the restored table.

If a backup chain starts from a full backup subsequent to the IMPORT INTO job cancellation, it is unaffected.

The technical details can be found in the following GitHub issues:

• #88042
• #88043

Statement

This is resolved in CockroachDB by #87312.

The fix has been applied to the maintenance release of CockroachDB v22.1.9 and CockroachDB v21.2.17.

This public issue is tracked by #87305.

Mitigation

Users of CockroachDB v22.1.0 to v22.1.8 are encouraged to upgrade to v22.1.9 or a later version.

Users of CockroachDB v21.2.0 to v21.2.16 are encouraged to upgrade to v21.2.17 or a later version.

If a RESTORE must be run using prior versions, Cockroach Labs recommends that where possible, you restore only to full backups. If you must restore to an incremental backup taken on an affected version, restore only tables on which IMPORT INTO did not run or succeeded.

The best mitigation, however, is to upgrade to v22.1.9 or v21.2.17 as soon as possible.

Impact

A RESTORE of an incremental backup may include rows that should not be restored, in a narrow set of circumstances relating to an ongoing IMPORT INTO job. Versions affected include v22.1.0 to v22.1.8 and v21.2.0 to v21.2.16.

Please reach out to the support team if more information or assistance is needed.