Technical Advisory 82309

​

Description

​

Statement

• You observe that your downstream sink (such as Kafka) is not receiving data.
• After the upgrade begins, metrics related to the number of messages ( changefeed.emitted_messages ) and message sizes ( changefeed.emitted_bytes ) will be close to 0. This metric will begin dropping immediately, as soon as nodes start upgrading.
• The decay in the number of emitted messages may be gradual (depending on the upgrade speed), but once all nodes have upgraded, 0 messages will be emitted.

​

Mitigation

• In the default case, creating a changefeed will emit (scan) the entire table to your sink, and then begin emitting all new events. This does not require any special options, as it is the default.
  CREATE CHANGEFEED FOR TABLE name, name2, name3
  INTO '{sink URI}'
  WITH '{your existing options}';
  
• Alternatively, you may wish to be more precise in restarting (catching up) from the point in time where the failure began. This is likely preferable for large tables where you do not wish to incur the full initial scan.
  • To do this, you will need to choose a point in time. We recommend choosing a few minutes before your upgrade began.
    The usual method of retrieving the high-water timestamp cursor is invalid in the presence of this bug.
    CREATE CHANGEFEED FOR TABLE name, name2, name3
    INTO '{sink URI}'
    WITH '{your existing options}', cursor='{see below}';
    
  • To get a cursor, note the time your upgrade began in UTC. You can use either:
    • An absolute point in time (e.g., 2022-06-02 13:59:00+00 in this example):
      SELECT 1e9*('2022-06-02 13:59:00+00'::timestamptz)::int;
      
    • A relative time interval since the upgrade began (e.g., 3h10m in this example):
      SELECT 1e9*(now()- interval '3h10m')::int;
      

CREATE CHANGEFEED FOR TABLE name, name2, name3
INTO '{sink URI}'
WITH '{your existing options}', no_initial_scan;

​

Impact