The configurations and settings explained on this page will significantly impact a changefeed's behavior.
The following sections describe performance, settings, configurations, and details to tune changefeeds:
Some options for the
webhook_sink_config parameters are discussed on this page. However, for more information on specific tuning for Kafka and Webhook sinks, refer to the following pages:
New in v23.1: By default, changefeeds are integrated with elastic CPU, which helps to prevent changefeeds from affecting foreground traffic. For example, changefeed backfills and initial scans can be CPU-intensive. This integration will result in a cluster prioritizing SQL traffic over changefeeds. Since this may affect changefeed latency, you can monitor your cluster's admission control system on the Overload Dashboard and changefeed latency on the Changefeed Dashboard.
This is controlled by the following cluster settings, which are by default enabled:
For a more technical explanation of elastic CPU, refer to the Rubber control theory on the Go scheduler blog post.
Tuning for high durability delivery
When designing a system that relies on high durability message delivery—that is, not missing any message acknowledgement at the downstream sink—consider the following settings and configuration in this section:
- Pausing changefeeds and garbage collection
- Defining Kafka message acknowledgment
- Choosing changefeed sinks
- Defining schema change behavior
Before tuning these settings, we recommend reading details on our changefeed at-least-once-delivery guarantee.
Pausing changefeeds and garbage collection
By default, protected timestamps will protect changefeed data from garbage collection up to the time of the checkpoint. Protected timestamps will protect changefeed data from garbage collection if the downstream changefeed sink is unavailable until you either cancel the changefeed or the sink becomes available once again.
However, if the changefeed lags too far behind, the protected changes could lead to an accumulation of garbage. This could result in increased disk usage and degraded performance for some workloads.
For more detail on changefeeds and protected timestamps, refer to Garbage collection and changefeeds.
To balance protecting change data and prevent the over-accumulation of garbage, Cockroach Labs recommends creating a changefeed with options to define your protection duration and monitoring your changefeed for protected timestamp record collection.
Protecting change data on pause
Create changefeeds with the following options so that your changefeed protects data when it is paused:
protect_data_from_gc_on_pause: to protect changes while the changefeed is paused until you resume the changefeed.
on_error=pause: to pause the changefeed when it encounters an error. By default, changefeeds treat errors as retryable apart from some exceptions.
- New in v23.1:
gc_protect_expires_after: to automatically expire the protected timestamp records that are older than your defined duration and cancel the changefeed job.
Monitoring protected timestamp records
New in v23.1: You can monitor changefeed jobs for protected timestamp usage. We recommend setting up monitoring for the following metrics:
jobs.changefeed.protected_age_sec: Tracks the age of the oldest protected timestamp record protected by changefeed jobs. We recommend monitoring if
protected_age_secis greater than
protected_age_secincreases, garbage accumulation increases. Garbage collection will not progress on a table, database, or cluster if the protected timestamp record is present.
jobs.changefeed.currently_paused: Tracks the number of changefeed jobs currently considered paused. Since paused changefeed jobs can accumulate garbage, it is important to monitor the number of paused changefeeds.
jobs.changefeed.expired_pts_records: Tracks the number of expired protected timestamp records owned by changefeed jobs. You can monitor this metric in conjunction with the
jobs.changefeed.protected_record_count: Tracks the number of protected timestamp records held by changefeed jobs.
Defining Kafka message acknowledgment
To determine what a successful write to Kafka is, you can configure the
kafka_sink_config option. The
'RequiredAcks' field specifies what a successful write to Kafka is. CockroachDB guarantees at least once delivery of messages—the
'RequiredAcks' value defines the delivery.
For high durability delivery, Cockroach Labs recommends setting:
ALL provides the highest consistency level. A quorum of Kafka brokers that have committed the message must be reached before the leader can acknowledge the write.
You must also set
ALL in your server-side Kafka configuration for this to provide high durability delivery.
Choosing changefeed sinks
Use Kafka or cloud storage sinks when tuning for high durability delivery in changefeeds. Both Kafka and cloud storage sinks offer built-in advanced protocols, whereas the webhook sink, while flexible, requires an understanding of how messages are acknowledged and committed by the particular system used for the webhook in order to ensure the durability of message delivery.
New in v23.1:
changefeed.new_webhook_sink_enabled cluster setting to improve the throughput of changefeeds emitting to webhook sinks.
Defining schema change behavior
Ensure that data is ingested downstream in its new format after a schema change by using the
schema_schange_policy options. For example, setting
schema_change_policy=stop will trigger an error to the
cockroach.log file on a schema change and the changefeed to fail.
Tuning for high throughput
When designing a system that needs to emit a lot of changefeed messages, whether it be steady traffic or a burst in traffic, consider the following settings and configuration in this section:
- Setting the
- Batching and buffering messages
- Configuring file and message format
- Configuring for tables with many ranges
- Adjusting concurrent changefeed work
When a changefeed emits a resolved message, it force flushes all outstanding messages that have buffered, which will diminish your changefeed's throughput while the flush completes. Therefore, if you are aiming for higher throughput, we suggest setting the duration higher (e.g., 10 minutes), or not using the
If you are setting the
resolved option when you are aiming for high throughput, you must also consider the
min_checkpoint_frequency option, which defaults to
30s. This option controls how often nodes flush their progress to the coordinating changefeed node. As a result,
resolved messages will not be emitted more frequently than the configured
min_checkpoint_frequency. Set this option to at least as long as your
resolved option duration.
Batching and buffering messages
- Batch messages to your sink:
- For a Kafka sink, refer to the
Flushparameter for the
- For a cloud storage sink, use the
FILE_SIZEparameter to flush a file when it exceeds the specified size.
- For a webhook sink, refer to the
Flushparameter for the
- For a Kafka sink, refer to the
- Set the
changefeed.memory.per_changefeed_limitcluster setting to a higher limit to give more memory for buffering changefeed data. This setting influences how often the changefeed will flush buffered messages. This is useful during heavy traffic.
Configuring file and message format
avroas the emitted message format option with Kafka sinks; JSON encoding can potentially create a slowdown.
- Use the
compressionoption when you create a changefeed emitting data files to a cloud storage sink. For larger files, set
- Use the
snappycompression format to emit messages to a Kafka sink. If you're intending to do large batching for Kafka, use the
To configure changefeeds emitting to cloud storage sinks for high throughput, you should consider:
- Increasing the
FILE_SIZEparameter to control the size of the files that the changefeed sends to the sink. The default is
16MB. To configure for high throughput, we recommend
128MB. Note that this is not a hard limit, and a changefeed will flush the file when it reaches the specified size.
- When you compress a file, it will contain many more events.
- File size is also dependent on what kind of data the changefeed job is writing. For example, large JSON blobs will quickly fill up the
FILE_SIZEvalue compared to small rows.
- When you change or increase
FILE_SIZE, ensure that you adjust the
changefeed.memory.per_changefeed_limitcluster setting, which has a default of
512MiB. Buffering messages can quickly reach this limit if you have increased the file size.
Configuring for tables with many ranges
If you have a table with 10,000 or more ranges, you should consider increasing the following two cluster settings. We strongly recommend increasing these settings slowly. That is, increase the setting and then monitor its impact before adjusting further:
kv.rangefeed.catchup_scan_concurrency: The number of catchups a rangefeed can execute concurrently. The default is
kv.rangefeed.concurrent_catchup_iterators: The number of rangefeed catchup iterators a store will allow concurrently before queuing. The default is
Adjusting concurrent changefeed work
- Increase the
changefeed.backfill.concurrent_scan_requestssetting, which controls the number of concurrent scan requests per node issued during a backfill event. The default behavior, when this setting is at
0, is that the number of scan requests will be 3 times the number of nodes in the cluster (to a maximum of 100). While increasing this number will allow for higher throughput, it will increase the cluster load overall, including CPU and IO usage.
onby default. This causes rangefeeds to use time-bound iterators for catch-up scans when possible. Catch-up scans are run for each rangefeed request. This setting improves the performance of changefeeds during some range-split operations.