Egress Perimeter Controls for CockroachDB Dedicated

On this page

This page describes how Egress Perimeter Controls can enhance the security of CockroachDB Dedicated clusters, and gives an overview of how to manage a cluster's egress rules.

Note:

Egress Perimeter Controls are not yet available for CockroachDB Dedicated on Azure.

Why use Egress Perimeter Controls

CockroachDB Dedicated clusters access external resources for many purposes:

Managing backups as part of a disaster recovery plan
Using Change data capture (CDC) changefeeds
Exporting data
Exporting logs

By default, clusters can access external resources via the internet without restriction, and even private clusters can access their private network. This potentially leaves a cluster open to a data exfiltration scenario, wherein an attacker, often a malicious insider, steals data by sending backups, changefeeds, data, or logs to a source that they control.

Operators of CockroachDB Dedicated clusters can mitigate against this risk by using Egress Perimeter Controls, which enable Cluster Administrators to restrict egress to a list of specified external destinations. This adds a strong layer of protection against malicious or accidental data exfiltration. Along with other measures such as Private Clusters, Egress Perimeter Controls are an important component in an overall strategy for maximizing network security.

Further reading: review how CockroachDB products differs in advanced security features.

Note:

Regardless of user-specific Egress Perimeter Control policy, egress is always permitted to services that are managed by Cockroach Labs and are essential to your cluster's functionality and ongoing operations.

Before you begin

Egress Perimeter Controls are supported on AWS and GCP for the following deployment types:
- CockroachDB Dedicated advanced with PCI-ready features.
- CockroachDB Dedicated Private Cluster.
Egress Perimeter Controls are not supported for CockroachDB Dedicated on Azure or for CockroachDB Serverless.
You need a service account with the Cluster Administrator role on clusters in your organization. You can provision service accounts and API keys in CockroachDB Cloud Console. Refer to Service Accounts.

Warning:

The operations described in this page require an API key with very broad permissions, such as the potential to modify a cluster's configuration to add malicious egress rules that could allow the type of attack that Egress Perimeter Controls are meant to prevent. Do not allow this key to be copied or transmitted in any form, including by capturing an image of your computer screen.

Initialize your shell with your API key and Cluster id

Inspect your cluster in the clusters page in the CockroachDB Cloud console.
Find your cluster's universally unique identifier (UUID). To do this, select your cluster from the clusters page in the console. The UUID will appear in the URL of the overview page for that specific cluster, in the format:

https://cockroachlabs.cloud/cluster/{ your cluster's UUID }/overview
Save your API key and cluster UUID as environment variables.
```
CC_API_KEY={ your API key }
CLUSTER_ID={ your cluster UUID }
```

Inspect your cluster with the Cloud API:

curl --request GET \
--header "Authorization: Bearer $CC_API_KEY" \
--url "https://management-staging.crdb.io/api/v1/clusters/$CLUSTER_ID"


{
  "id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
  "name": "docstest",
  "cockroach_version": "latest-v22.2-build(sha256:e42c4de8577556132120a9ab07efc1a2a96779c028ebab99223d862d9792428b)",
  "plan": "DEDICATED",
  "cloud_provider": "AWS",
  "account_id": "1234567890",
  "state": "CREATED",
  "creator_id": "b7d7bedc-b8f3-4a98-bd2c-4ba2bf9adbe1",
  "operation_status": "CLUSTER_STATUS_UNSPECIFIED",
  "config": {
    "dedicated": {
      "machine_type": "m5.large",
      "num_virtual_cpus": 2,
      "storage_gib": 15,
      "memory_gib": 8,
      "disk_iops": 225
    }
  },
  "regions": [
    {
      "name": "us-west-2",
      "sql_dns": "docstest-6qsh.aws-us-west-2.crdb.io",
      "ui_dns": "admin-docstest-6qsh.aws-us-west-2.crdb.io",
      "internal_dns": "",
      "node_count": 1
    }
  ],
  "created_at": "2022-10-27T18:03:47.862079Z",
  "updated_at": "2022-10-27T18:24:58.111198Z",
  "deleted_at": null
}

Use a deny-by-default egress traffic policy

Note:

Essential external traffic destined to resources managed by Cockroach Labs is always allowed, regardless of user-specified egress policy.

Make a POST request ordering the API to update your cluster's egress policy from default allow-all to deny-all. This allows you to add egress rules to explicitly allow egress traffic from the cluster to certain external destinations.
```
curl --request POST \
--header "Authorization: Bearer $CC_API_KEY" \
--header 'Cc-Version: latest' \
--url "https://management-staging.crdb.io/api/v1/clusters/$CLUSTER_ID/networking/egress-rules/egress-traffic-policy" \
--data '{"allow_all":false}'
```
```
{}
```

Create an egress rule to allow a destination

An egress rule is created with the following attributes:

name: A name for the rule.
type: Whether the destination will be specified as a fully-qualified domain name (FQDN), or as a range of IP addresses specified in CIDR notation. Value is "FQDN" or "CIDR".
destination: Either a fully qualified domain name, for example www.cockroachlabs.com, or a CIDR range, for example, 123.45.67.890/32.
ports: An array of allowed ports, for example [44,8080].
Warning:
By default, all ports are allowed.
paths: Deprecated. This field is ignored and will be removed in the future. An egress rule applies to all paths.
description: The intended purpose of egress to the destination.

The following steps create one FQDN rule and one CIDR rule.

First, create a YAML manifest for a FQDN-based rule. This example allows egress traffic from the cluster to storage.googleapis.com (the Google Cloud Platform (GCP) storage API) on ports 80 and 443.
```
---
name: "roach-buckets"
type: "FQDN"
destination: "storage.googleapis.com"
ports: [80, 443]
description: 'egress for GCP storage API'
```
Next, create a YAML manifest for a CIDR-based rule. This example adds egress traffic from the cluster to the IPv4 subnet 123.34.62.123/32 (which matches a single IPv4 address) on port 443, which may be, for example, the load balancer for a Kafka cluster running in your organization's internal network or private cloud.
```
--- # egress-rule2.yml
name: "roach-kafka"
type: "CIDR"
destination: "123.34.62.123/32" # replace with Kafka cluster CIDR range
ports: [443]
description: 'egress for Kafka'
```

Use Ruby (or another technique), to compile human-editable YAML into API-parsable JSON:

ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' < egress-rule1.yml > egress-rule1.json
ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' < egress-rule2.yml > egress-rule2.json

Use the shell utility jq to inspect the JSON payload:

Note:

On macOS, you can install jq from Homebrew: brew install jq

cat egress-rule1.json |jq
cat egress-rule2.json |jq

{
  "name": "roach-buckets",
  "type": "FQDN",
  "destination": "storage.googleapis.com",
  "ports": [
    443,
    80
  ],
  "description": "egress for GCP storage API"
}
{
  "name": "roach-kafka",
  "type": "CIDR",
  "destination": "123.34.62.123/32",
  "ports": [
    443
  ],
  "description": "egress for Kafka"
}

Make a POST request to create each rule from its JSON payload.

curl --request POST \
--header "Authorization: Bearer $CC_API_KEY" \
--url "https://management-staging.crdb.io/api/v1/clusters/$CLUSTER_ID/networking/egress-rules" \
--data "@egress-rule1.json"

curl --request POST \
--header "Authorization: Bearer $CC_API_KEY" \
--url "https://management-staging.crdb.io/api/v1/clusters/$CLUSTER_ID/networking/egress-rules" \
--data "@egress-rule2.json"

{
  "Rule": {
    "id": "564a25da-b99c-4e08-9ec3-483c0f1bc620",
    "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
    "name": "roach-buckets",
    "type": "FQDN",
    "state": "PENDING_CREATION",
    "crl_managed": false,
    "destination": "storage.googleapis.com",
    "ports": [
      443,
      80
    ],
    "description": "egress for GCP storage API",
    "created_at": "2022-10-27T19:35:51.435571Z"
  }
}
{
  "Rule": {
    "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
    "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
    "name": "roach-kafka",
    "type": "CIDR",
    "state": "PENDING_CREATION",
    "crl_managed": false,
    "destination": "123.34.62.123/32",
    "ports": [
      443
    ],
    "description": "egress for Kafka",
    "created_at": "2022-10-27T19:36:25.670162Z"
  }
}

Warning:

Your cluster's firewall behavior is enforced asynchronously after the API response. After submitting the request, check your egress rules to confirm that the new rules have been created.

Check the status of a rule

Note:

Refer to the list of rule statuses.

curl --request GET \
--header "Authorization: Bearer $CC_API_KEY" \
--header  'Cc-Version: 2022-09-20' \
--url "https://management-staging.crdb.io/api/v1/clusters/$CLUSTER_ID/networking/egress-rules/$RULE_ID"

{
  "rule": {
    "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
    "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
    "name": "roach-kafka",
    "type": "CIDR",
    "state": "ACTIVE",
    "crl_managed": false,
    "destination": "123.34.62.123/32",
    "ports": [
      443
    ],
    "description": "egress for Kafka",
    "created_at": "2022-10-27T19:36:25.670162Z"
  }

Check egress rules for a cluster

Note:

Consult the glossary of rule statuses.

curl --request GET \
--header "Authorization: Bearer $CC_API_KEY" \
--header  'Cc-Version: 2022-09-20' \
--url "https://management-staging.crdb.io/api/v1/clusters/$CLUSTER_ID/networking/egress-rules"

{
  "rules": [
    {
      "id": "564a25da-b99c-4e08-9ec3-483c0f1bc620",
      "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
      "name": "roach-buckets",
      "type": "FQDN",
      "state": "ACTIVE",
      "crl_managed": false,
      "destination": "storage.googleapis.com",
      "ports": [
        443,
        80
      ],
      "description": "egress for GCP storage API",
      "created_at": "2022-10-27T19:35:51.435571Z"
    },
    {
      "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
      "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
      "name": "roach-kafka",
      "type": "CIDR",
      "state": "ACTIVE",
      "crl_managed": false,
      "destination": "123.34.62.123/32",
      "ports": [
        443
      ],
      "description": "egress for Kafka",
      "created_at": "2022-10-27T19:36:25.670162Z"
    }
  ],
  "pagination": null
}

Remove a rule

To delete a rule, make DELETE request to the rule's path.

Warning:

Your cluster's firewall behavior is enforced asynchronously after the API response. After submitting the request, check your egress rules to confirm that the deletion is complete.

curl --request DELETE \
--header "Authorization: Bearer $CC_API_KEY" \
--header  'Cc-Version: 2022-09-20' \
--url "https://management-staging.crdb.io/api/v1/clusters/$CLUSTER_ID/networking/egress-rules/$RULE_ID"

{
  "Rule": {
    "id": "aa4f37d7-9aa4-456d-8df7-225d5c91c120",
    "cluster_id": "21f474d5-3e65-4a54-9317-e8d8803ef917",
    "name": "roach-kafka",
    "type": "CIDR",
    "state": "PENDING_DELETION",
    "crl_managed": false,
    "destination": "123.34.62.123/32",
    "ports": [
      443
    ],
    "description": "egress for Kafka",
    "created_at": "2022-10-27T19:36:25.670162Z"
  }
}

Rule statuses

The API displays the following statuses for a rule, when you [inspect an egress rule] or [list a cluster's egress rules]:

ACTIVE: The rule has been successfully enforced in your cluster's network firewall and egress is currently allowed to the specified destination.
PENDING_CREATION: Implementation of a new rule is in process. The behavior of the cluster's network firewall may not yet reflect the new rule, so egress may not yet be allowed to the selected destination.
CREATION_FAILED: Something went wrong in the process of implementing a rule. This is a rare occurrence and does indicate that you should contact the Cockroach Labs Support Team immediately, rather than attempting to retry rule creation, to avoid leaving your cluster's firewall in an unknown state.
PENDING_UPDATE: An PATCH request to update a rule is in process. The behavior may not yet be enforced in your cluster's firewall.
PENDING_DELETION: The rule-removal is being enforced. The behavior of the cluster's network firewall may still reflect the target rule, so egress may still be allowed to the destination.
DELETION_FAILED: A rule update or a rule removal has failed. The behavior of the cluster's network firewall may still reflect the previous state of the rule. This is a rare occurrence and does indicate that you should contact the Cockroach Labs Support Team immediately, rather than attempting to retry rule creation, to avoid leaving your cluster's firewall in an unknown state.
INCONSISTENT: This indicates that a rule cannot apply consistently across all of the regions or compute resources to which it is meant to apply. This is a rare occurrence and does indicate that you should contact the Cockroach Labs Support Team immediately, rather than attempting to retry rule creation, to avoid leaving your cluster's firewall in an unknown state.

Warning:

Generally, any kind of FAILED state, or an INCONSISTENT state, is a rare occurrence but can potentially leave the cluster in an unknown state. You should contact the Cockroach Labs Support Team immediately, rather than attempting to retry rule creation, to avoid leaving your cluster's firewall in an unknown state.

Was this helpful?

Cockroach
University

Docs Hub

Egress Perimeter Controls for CockroachDB Dedicated

Why use Egress Perimeter Controls

Before you begin

Initialize your shell with your API key and Cluster id

Use a deny-by-default egress traffic policy

Create an egress rule to allow a destination

Check the status of a rule

Check egress rules for a cluster

Remove a rule

Rule statuses

Cockroach University

Docs Hub

Cockroach University

Docs Hub

Egress Perimeter Controls for CockroachDB Dedicated

Why use Egress Perimeter Controls

Before you begin

Initialize your shell with your API key and Cluster id

Use a deny-by-default egress traffic policy

Create an egress rule to allow a destination

Check the status of a rule

Check egress rules for a cluster

Remove a rule

Rule statuses

Cockroach
University

Cockroach
University