Integrate CockroachDB with Confluent Cloud using Change Data Capture

Last edited on May 10, 2024

0 minute read

    We’re excited to announce an integration that brings together two cloud-native powerhouses: CockroachDB and Confluent. Businesses can now seamlessly connect the distributed SQL capabilities of CockroachDB with the real-time event streaming features of Confluent using change data capture. Whether it’s real-time analytics, event-driven architectures, or significantly simplified data migrations, this integration is your ticket to a more streamlined, efficient, and powerful data architecture.

    CockroachDB and ConfluentCopy Icon

    Using Confluent and Cockroach together creates leverage in your data architecture in five significant areas:

    1. Reliability

    The distributed nature of CockroachDB and Kafka ensures that your applications remain resilient against failures, providing a truly uninterrupted service. Distributed data stores mean that if one node goes down, service can continue to function without any hiccups.

    2. Scalability

    Confluent Cloud and CockroachDB Cloud help you to future proof your data architecture. They are both built for native horizontal scalability. As your data streams and datasets grow, you won’t need to worry about infrastructure or scalability issues.

    3. Cloud-Native Synergy

    The true magic happens when you leverage Confluent Cloud and CockroachDB Cloud together. Both platforms offer fully managed platforms, meaning less time tinkering and more time innovating.The cloud-native architecture of both platforms ensures auto-scaling and resilience. Using them together as-a-service removes operational complexity and the burden of maintenance

    4. Global Reach

    CockroachDB Cloud offers geo-replicated clusters, and when combined with Confluent’s global presence, you get a solution that’s truly global.

    5. Cost Efficiency

    Both services offer pay-as-you go pricing models. So, you only pay for what you use, driving down costs and ensuring optimal resource utilization.

    Integrate CockroachDB Cloud with Confluent Cloud in just 3 steps Copy Icon

    In just three steps, you can set up a resilient, enterprise-grade data pipeline from your CockroachDB cluster to Confluent Cloud, using CockroachDB's built-in data streaming and bulk export capabilities.

    1. Create a topic from the Confluent Cloud console:

    Go to the Confluent Cloud Console -> Your Cluster -> Topics -> Create Topic. This is the topic we will send our changefeed messages to. You can choose a name and number of partitions for your topic. You can also choose more advanced configuration options for your topic such as the retention time of your messages.

    confluent cluster settings

    2. Get Connection Information for your Confluent Cloud Cluster

    Next, to connect to your cluster, you need to get the endpoint and authentication information for the cluster. First step is to create an API key for the cluster. Copy the key and secret information to be used later.

    confluent endpoint

    Next, copy the “Bootstrap server endpoint” from the cluster settings page:

    confluent topic-create

    You’re now ready to stream data into your Confluent Cloud Cluster!

    3. Create changefeed into Confluent Cloud

    You’re now ready to stream data into your Confluent Cloud Cluster! Go to the SQL shell for your CockroachDB cluster, ensure your user has the required privileges to create changefeeds, and issue a CREATE CHANGEFEED SQL statement.  Your statement should look like this:

    CREATE CHANGEFEED FOR TABLE [your tables] INTO 'confluent-cloud://[bootstrap server endpoint]?api_key=[your key]&api_secret=[your secret]&topic_name=[your topic name]'

    You can add additional configuration options when you create the changefeed. Note that the API secret will need to be URL-encoded.

    That’s it. You’re done.Copy Icon

    The integration of CockroachDB with Confluent, powered by changefeeds, opens doors to endless possibilities. With the cloud advantages of both platforms at your fingertips, you’re geared up for unparalleled efficiency, scalability, and reliability - without the headaches of deploying and managing your own cloud software.

    Happy streaming!

    Distributed SQL