Export Metrics From a CockroachDB Dedicated Cluster

On this page Carat arrow pointing down

CockroachDB Dedicated users can use the Cloud API to configure metrics export to AWS CloudWatch or Datadog. Once the export is configured, metrics will flow from all nodes in all regions of your CockroachDB Dedicated cluster to your chosen cloud metrics sink.

Exporting metrics to AWS CloudWatch is only available on CockroachDB Dedicated clusters which are hosted on AWS, and were created after August 11, 2022. Metrics export to Datadog is supported on all CockroachDB Dedicated clusters regardless of creation date.

Note:

This feature is in preview and is only available to enrolled organizations. To enroll your organization in the preview, contact your Cockroach Labs account team. This feature is subject to change.

The metricexport endpoint

To configure and manage metrics export for your CockroachDB Dedicated cluster, use the metricexport endpoint:

icon/buttons/copy
https://cockroachlabs.cloud/api/v1/clusters/{your_cluster_id}/metricexport

Access to the metricexport endpoint requires a valid CockroachDB Cloud service account with the appropriate permissions.

The following methods are available for use with the metricexport endpoint, and require the listed service account permissions:

Method Required permissions Description
GET ADMIN, EDIT, or READ Returns the current status of the metrics export configuration.
POST ADMIN or EDIT Enables metrics export, or updates an existing metrics export configuration.
DELETE ADMIN Disables metrics export, halting all metrics export to AWS CloudWatch or Datadog.

See Service accounts for instructions on configuring a service account with these required permissions.

Enable metrics export

Warning:

The metrics export feature is only available on CockroachDB Dedicated clusters created after August 11, 2022. If your CockroachDB Dedicated cluster was created before this date, you must create a new cluster to utilize metrics export, or export metrics to Datadog instead.

Perform the following steps to enable metrics export from your CockroachDB Dedicated cluster to AWS CloudWatch.

  1. Create the desired target AWS CloudWatch log group by following the Create a log group in CloudWatch instructions. If you already have a log group created, you may skip this step. When your CockroachDB Dedicated cluster emits metrics to AWS CloudWatch, they are written to this log group.

  2. Find your CockroachDB Dedicated cluster ID:

    1. Visit the CockroachDB Cloud console cluster page.
    2. Click on the name of your cluster.
    3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview. It should resemble f78b7feb-b6cf-4396-9d7f-494982d7d81e.
  3. Determine your cluster's cloud provider account ID. This command uses the third-party JSON parsing tool jq to isolate just the needed account_id field:

    icon/buttons/copy
    curl --request GET \
      --url https://cockroachlabs.cloud/api/v1/clusters/{your_cluster_id} \
      --header 'Authorization: Bearer {secret_key}' | jq .account_id
    

    Where:

    • {your_cluster_id} is the cluster ID of your CockroachDB Dedicated cluster as determined in step 2.
    • {secret_key} is your CockroachDB Dedicated API key. See API Access for more details.
  4. Create a cross-account IAM role in your AWS account:

    1. In the AWS console, visit the IAM page.
    2. Select Roles and click Create role.
    3. For Trusted entity type, select AWS account.
    4. Choose Another AWS account.
    5. For Account ID, provide the CockroachDB Dedicated cloud provider account ID from step 2.
    6. Finish creating the IAM role with a suitable name. These instructions will use the role name CockroachCloudMetricsExportRole. You do not need to add any permissions.
    Note:

    You will need the Amazon Resource Name (ARN) for your cross-account IAM role later in this procedure.

  5. Select the new role, and create a new policy for this role. These instructions will use the policy name CockroachCloudMetricsExportPolicy.

  6. Select the new policy, and paste the following into the Permissions tab, with the {} JSON option selected:

    icon/buttons/copy
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::{your_aws_acct_id}:root"
                },
                "Action": "sts:AssumeRole"
            },
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::{your_aws_acct_id}:root"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }
    

    Where:

    • {your_aws_acct_id} is the AWS Account ID of the AWS account where you created the CockroachCloudMetricsExportRole role, not the cloud provider account ID of your CockroachDB Dedicated cluster from step 2. You can find your AWS Account ID on the AWS IAM page.

    This defines the set of permissions that the CockroachDB Dedicated metrics export feature requires to be able to write metrics to CloudWatch.

  7. Copy the Amazon Resource Name (ARN) of the CockroachCloudMetricsExportRole role found under Summary, which is needed for the next step.

  8. Issue the following Cloud API command to enable metrics export for your CockroachDB Dedicated cluster:

    icon/buttons/copy
    curl --request POST \
      --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport \
      --header "Authorization: Bearer {secret_key}" \
      --data '{"cloudwatch":{"target_region": "{aws_region}}", "role_arn": "arn:aws:iam::{role_arn}:role/CockroachCloudMetricsExportRole", "log_group_name": "{log_group_name}"}}'
    

    Where:

    • {cluster_id} is your CockroachDB Dedicated cluster ID as determined in step 2.
    • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.
    • {aws_region} is your AWS region, like us-east-1.
    • {role_arn} is the ARN for the CockroachCloudMetricsExportRole role you copied in step 7. If you used a different role name there, be sure to use your role name in place of CockroachCloudMetricsExportRole in the above command.
    • {log_group_name} is the target AWS CloudWatch log group you created in step 1.

    Specifying an AWS region (to {aws_region}) that you do not have a cluster in, or a region that only partially covers your cluster's nodes will result in missing metrics.

  9. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

    icon/buttons/copy
    curl --request GET \
      --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport \
      --header "Authorization: Bearer {secret_key}"
    

    Run the command periodically until the command returns a status of ENABLED, at which point the configuration across all nodes is complete, and metrics will begin appearing in CloudWatch under the log group you created in step 1. Since the configuration is applied to cluster nodes in a rolling fashion, you may see some metrics appear even before the GET command returns an ENABLED status.

  10. Once metrics export has been enabled, you can access metrics from your CockroachDB Dedicated cluster directly in AWS CloudWatch.

To enable metrics export for your CockroachDB Dedicated cluster to Datadog, you can either enable the Datadog integration in your CockroachDB Dedicated Cloud Console, or on the command line via the Cloud API:

  1. Find your CockroachDB Dedicated cluster ID:

    1. Visit the CockroachDB Cloud console cluster page.
    2. Click on the name of your cluster.
    3. Find your cluster ID in the URL of the single cluster overview page: https://cockroachlabs.cloud/cluster/{your_cluster_id}/overview. It should resemble f78b7feb-b6cf-4396-9d7f-494982d7d81e.
  2. Determine your cluster's cloud provider account ID. This command uses the third-party JSON parsing tool jq to isolate just the needed account_id field:

    icon/buttons/copy
    curl --request GET \
      --url https://cockroachlabs.cloud/api/v1/clusters/{your_cluster_id} \
      --header 'Authorization: Bearer {secret_key}' | jq .account_id
    

    Where:

    • {your_cluster_id} is the cluster ID of your CockroachDB Dedicated cluster as determined in step 1.
    • {secret_key} is your CockroachDB Dedicated API key. See API Access for more details.
  3. Determine the Datadog API key you'd like to use. If you don't already have one, follow the steps to add a new Datadog API key.

  4. Issue the following Cloud API command to enable metrics export for your CockroachDB Dedicated cluster:

    icon/buttons/copy
    curl --request POST \
      --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport \
      --header "Authorization: Bearer {secret_key}" \
      --data '{"datadog":{"site": "{datadog_site}", "api_key": "{datadog_api_key}}"}}'
    

    Where:

    • {cluster_id} is your CockroachDB Dedicated cluster ID as determined in step 1, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
    • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.
    • {datadog_site} is your Datadog site. Valid sites are: US1, US3, US5, US1_GOV, and EU1.
    • {datadog_api_key} is the Datadog API key determined in step 3.
  5. Depending on the size of your cluster and how many regions it spans, the configuration may take a moment. You can monitor the ongoing status of the configuration using the following Cloud API command:

    icon/buttons/copy
    curl --request GET \
      --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport \
      --header "Authorization: Bearer {secret_key}"
    

    Run the command periodically until the command returns a status of ENABLED, at which point the configuration across all nodes is complete, and metrics will begin appearing in the Datadog interface. Since the configuration is applied to cluster nodes in a rolling fashion, you may see some metrics appear even before the GET command returns an ENABLED status.

  6. Once metrics export has been enabled, you can access metrics from your CockroachDB Dedicated cluster directly in Datadog's Metrics Explorer, or via Datadog's notebook or dashboard features.

Monitor the status of a metrics export configuration

To check the status of an existing CockroachDB Dedicated metrics export configuration, use the following Cloud API command:

icon/buttons/copy
curl --request GET \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport \
  --header "Authorization: Bearer {secret_key}"

Where:

  • {cluster_id} is your CockroachDB Dedicated cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
  • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.

Update an existing metrics export configuration

To update an existing CockroachDB Dedicated metrics export configuration, make any necessary changes to your cloud provider configuration (e.g., AWS CloudWatch or Datadog), then issue the same POST Cloud API command as shown in the Enable metrics export instructions for your cloud provider with the desired updated configuration. Follow the Monitor the status of a metrics export configuration instructions to ensure the update completes successfully.

Disable metrics export

To disable an existing CockroachDB Dedicated metrics export configuration, and stop sending metrics to a cloud metrics sink, use the following Cloud API command:

icon/buttons/copy
curl --request DELETE \
  --url https://cockroachlabs.cloud/api/v1/clusters/{cluster_id}/metricexport?type={metrics_export_type} \
  --header "Authorization: Bearer {secret_key}"

Where:

  • {cluster_id} is your CockroachDB Dedicated cluster's cluster ID, which can be found in the URL of your Cloud Console for the specific cluster you wish to configure, resembling f78b7feb-b6cf-4396-9d7f-494982d7d81e.
  • {metrics_export_type} is the metrics export cloud sink target you are using, either: CLOUDWATCH or DATADOG.
  • {secret_key} is your CockroachDB Dedicated API key. See API Access for instructions on generating this key.

Limitations

  • Metrics export to AWS CloudWatch is currently only available on CockroachDB Dedicated clusters created after August 11, 2022.
  • Metrics export to AWS CloudWatch is only available on CockroachDB Dedicated clusters which are hosted on AWS. If your CockroachDB Dedicated cluster is hosted on GCP, you can export metrics to Datadog instead.
  • AWS CloudWatch does not currently support histograms. Any histogram-type metrics emitted from your CockroachDB Dedicated cluster are dropped by CloudWatch. See Prometheus metric type conversion for more information, and Logging dropped Prometheus metrics for instructions on tracking dropped histogram metrics in CloudWatch.

Troubleshooting

AWS CloudWatch

Be sure you are providing your own AWS Account ID as shown on the AWS IAM page to step 6, not the AWS cloud provider account ID as returned from step 2.

If you are using an existing AWS role, or are otherwise using a role name different from the example name used in this tutorial, be sure to use your own role name in step 8 in place of CockroachCloudMetricsExportRole.

Your CockroachDB Dedicated cluster must be running on AWS (not GCP), and must have been created after August 11, 2022 to make use of metrics export to AWS CloudWatch.


Yes No
On this page

Yes No