Data encryption and decryption is the process of transforming plaintext data to cipher-text and vice versa using a key or password.
Encryption in flight
CockroachDB uses TLS 1.2 for inter-node and client-node authentication as well as setting up a secure communication channel. Once the secure channel is set up, all inter-node and client-node network communication is encrypted using a shared encryption key as per the TLS 1.2 protocol. This feature is enabled by default for all secure clusters and needs no additional configuration.
Encryption at Rest (Enterprise)
Encryption is performed in the storage layer and configured per store. All files used by the store, regardless of contents, are encrypted with the desired algorithm.
To allow arbitrary rotation schedules and ensure security of the keys, we use two layers of keys:
Store keys are provided by the user in a file. They are used to encrypt the list of data keys (see below). This is known as a key encryption key: its only purpose is to encrypt other keys. Store keys are never persisted by CockroachDB. Since very little data is encrypted using this key, it can have a very long lifetime without risk of reuse.
Data keys are automatically generated by CockroachDB. They are used to encrypt all files on disk. This is known as a data encryption key. Data keys are persisted in a key registry file, encrypted using the store key. The key has a short lifetime to avoid reuse.
Store keys are specified at node startup by passing a path to a locally readable file. The file must contain 32 bytes (the key ID) followed by the key (16, 24, or 32 bytes). The size of the key dictates the version of AES to use (AES-128, AES-192, or AES-256). For an example showing how to create a store key, see Generating key files below.
Also during node startup, CockroachDB uses a data key with the same length as the store key. If encryption has just been enabled, the key size has changed, or the data key is too old (default lifetime is one week), CockroachDB generates a new data key.
Any new file created by the store uses the currently-active data key. All data keys (both active and previous) are stored in a key registry file and encrypted with the active store key.
After startup, if the active data key is too old, CockroachDB generates a new data key and marks it as active, using it for all further encryption.
CockroachDB does not currently force re-encryption of older files but instead relies on normal storage engine churn to slowly rewrite all data with the desired encryption.
Key rotation is necessary for Encryption at Rest for multiple reasons:
- To prevent key reuse with the same encryption parameters (after encrypting many files).
- To reduce the risk of key exposure.
Store keys are specified by the user and must be rotated by specifying different keys.
This is done by setting the
key parameter of the
--enterprise-encryption flag to the path to the new key,
old-key to the previously-used key.
Data keys will automatically be rotated at startup if any of the following conditions are met:
- The active store key has changed.
- The encryption type has changed (different key size, or plaintext to/from encryption).
- The current data key is
rotation-periodold or more.
Data keys will automatically be rotated at runtime if the current data key is
rotation-period old or more.
Once rotated, an old store key cannot be made the active key again.
Upon store key rotation the data keys registry is decrypted using the old key and encrypted with the new key. The newly-generated data key is used to encrypt all new data from this point on.
Changing encryption type
The user can change the encryption type from plaintext to encryption, between different encryption algorithms (using various key sizes), or from encryption to plaintext.
When changing the encryption type to plaintext, the data key registry is no longer encrypted and all previous data keys are readable by anyone. All data on the store is effectively readable.
When changing from plaintext to encryption, it will take some time for all data to eventually be re-written and encrypted.
There are a number of considerations to keep in mind when running with encryption.
To prevent key leakage, production deployments should:
- Use encrypted swap, or disable swap entirely.
- Disable core files.
CockroachDB attempts to disable core files at startup when encryption is requested, but it may fail.
Key management is the most dangerous aspect of encryption. The following rules should be kept in mind:
- Make sure that only the UNIX user running the
cockroachprocess has access to the keys.
- Do not store the keys on the same partition/drive as the CockroachDB data. It is best to load keys at run time from a separate system (e.g., Keywhiz, Vault).
- Rotate store keys frequently (every few weeks to months).
- Keep the data key rotation period low (default is one week).
A few other recommendations apply for best security practices:
- Do not switch from encrypted to plaintext, this leaks data keys. When plaintext is selected, all previously encrypted data must be considered reachable.
- Do not copy the encrypted files, as the data keys are not easily available.
- If encryption is desired, start a node with it enabled from the first run, without ever running in plaintext.
Note that backups taken with the
BACKUP statement are not encrypted even if Encryption at Rest is enabled. Encryption at Rest only applies to the CockroachDB node's data on the local disk. If you want encrypted backups, you will need to encrypt your backup files using your preferred encryption method.
Generating store key files
Cockroach determines which encryption algorithm to use based on the size of the key file. The key file must contain random data making up the key ID (32 bytes) and the actual key (16, 24, or 32 bytes depending on the encryption algorithm).
|Algorithm||Key size||Key file size|
|AES-128||128 bits (16 bytes)||48 bytes|
|AES-192||192 bits (24 bytes)||56 bytes|
|AES-256||256 bits (32 bytes)||64 bytes|
Generating a key file can be done using the
$ cockroach gen encryption-key -s 128 /path/to/my/aes-128.key
Or the equivalent openssl CLI command:
$ openssl rand -out /path/to/my/aes-128.key 48
Starting a node with encryption
Encryption at Rest is configured at node start time using the
--enterprise-encryption command line flag.
The flag specifies the encryption options for one of the stores on the node. If multiple stores exist,
the flag must be specified for each store.
The flag takes the form:
--enterprise-encryption=path=<store path>,key=<key file>,old-key=<old key file>,rotation-period=<period>.
The allowed components in the flag are:
||Required||Path of the store to apply encryption to.|
||Required||Path to the key file to encrypt data with, or
||Required||Path to the key file the data is encrypted with, or
||Optional||How often data keys should be automatically rotated. Default: one week.|
old-key components must always be specified. They allow for transitions between
encryption algorithms, and between plaintext and encrypted.
Starting a node for the first time using AES-128 encryption can be done using:
$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=/path/to/my/aes-128.key,old-key=plain
Once specified for a given store, the
--enterprise-encryption flag must always be present.
Checking encryption status
Encryption status can be seen on the node's stores report, reachable through:
http(s)://nodeaddress:8080/#/reports/stores/local (or replace
local with the node ID). For example, if you are running a local cluster, you can see the node's stores report at https://localhost:8080/#/reports/stores/local.
The report shows encryption status for all stores on the selected node, including:
- Encryption algorithm.
- Active store key information.
- Active data key information.
- The fraction of files/bytes encrypted using the active data key.
CockroachDB relies on storage layer compactions to write new files using the latest encryption key. It may take several days for all files to be replaced. Some files are only rewritten at startup, and some keep older copies around, requiring multiple restarts. You can force storage compaction with the
cockroach debug compact command (the node must first be stopped).
Information about keys is written to the logs, including:
- Active/old key information at startup.
- New key information after data key rotation.
Alternatively, you can use the
cockroach debug encryption-active-key command to view information about a store's encryption algorithm and store key.
Changing encryption algorithm or keys
Encryption type and keys can be changed at any time by restarting the node.
To change keys or encryption type, the
key component of the
--enterprise-encryption flag is set to the new key,
while the key previously used must be specified in the
For example, we can switch from AES-128 to AES-256 using:
$ cockroach start --store=cockroach-data --enterprise-encryption=path=cockroach-data,key=/path/to/my/aes-256.key,old-key=/path/to/my/aes-128.key
Upon starting, the node will read the existing data keys using the old encryption key (
aes-128.key), then rewrite
the data keys using the new key (
aes-256.key). A new data key will be generated to match the desired AES-256 algorithm.
To check that the new key is active, use the stores report page in the DB Console to check the encryption status.
To disable encryption, specify
key=plain. The data keys will be stored in plaintext and new data will not be encrypted.
To rotate keys, specify
old-key=/path/to/my/old-aes-128.key. The data keys
will be decrypted using the old key and then encrypted using the new key. A new data key will also be generated.
Encrypted backups (Enterprise)
You can encrypt full or incremental backups with a passphrase by using the
encryption_passphrase option. Files written by the backup (including
BACKUP manifests and data files) are encrypted using the specified passphrase to derive a key. To restore the encrypted backup, the same
encryption_passphrase option (with the same passphrase) must be included in the
When used with incremental backups, the
encryption_passphrase option is applied to all the backup file URLs, which means the same passphrase must be used when appending another incremental backup to an existing backup. Similarly, when used with locality-aware backups, the passphrase provided is applied to files in all localities.
Encryption is done using AES-256-GCM, and GCM is used to both encrypt and authenticate the files. A random salt is used to derive a once-per-backup AES key from the specified passphrase, and then a random initialization vector is used per-file. CockroachDB uses PBKDF2 with 64,000 iterations for the key derivation.
RESTORE will use more memory when using encryption, as both the plain-text and cipher-text of a given file are held in memory during encryption and decryption.
For an example of an encrypted backup, see Create an encrypted backup.
Higher CPU utilization
Enabling Encryption at Rest might result in a higher CPU utilization. We estimate a 5-10% increase in CPU utilization.
Encryption for touchpoints with other services
- S3 backup encryption
- Encrypted comms with Kafka