Description
In all versions of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3, changefeeds could emit events on the same row out of order in some cases, which violates changefeed ordering guarantees. This issue was caused by a bug in the sarama Kafka client library, a third-party library that CockroachDB uses to talk to Kafka clusters. The bug manifested when a workload had mutations to the same key in rapid succession, because it was possible for sarama to re-order the messages if it encountered a retryable Kafka error. To resolve this issue, CockroachDB now uses a new version of the Kafka sink that uses franz-go, a different third-party Kafka client library that does not have this bug. The new Kafka sink can be enabled using the cluster settingchangefeed.new_kafka_sink.enabled.
Statement
This is resolved by a fix in CockroachDB which replaces the Kafka client library sarama with franz-go. The fix has been applied to maintenance releases of CockroachDB v23.2.10, v24.1.4, and v24.2.0-rc.1.Mitigation
Users of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3 are encouraged to upgrade to , , , or a later version, and enable the cluster settingchangefeed.new_kafka_sink.enabled:
changefeed.new_kafka_sink.enabled will be enabled by default in versions v24.2.1, v24.3, and all later versions.