Follower reads are a mechanism that CockroachDB uses to provide faster reads in situations where you can afford to read data that may be slightly less than current (using
AS OF SYSTEM TIME). Normally, reads have to be serviced by a replica's leaseholder. This can be slow, since the leaseholder may be geographically distant from the gateway node that is issuing the query.
The shortest interval at which
AS OF SYSTEM TIME can serve follower reads is 4.8 seconds. In prior versions of CockroachDB, the interval was 48 seconds.
For instructions showing how to use follower reads to get low latency, historical reads in multi-region deployments, see the Follower Reads Topology Pattern.
This is an enterprise feature.
How follower reads work
Each CockroachDB node tracks a property called its "closed timestamp", which means that no new writes can ever be introduced below that timestamp. The closed timestamp advances forward by some target interval behind the current time. If the replica receives a write at a timestamp less than its closed timestamp, it rejects the write.
With follower reads enabled, any replica on a node can serve a read for a key as long as the time at which the operation is performed (i.e., the
AS OF SYSTEM TIME value) is less or equal to the node's closed timestamp.
When a gateway node in a cluster with follower reads enabled receives a request to read a key with a sufficiently old
AS OF SYSTEM TIME value, it forwards the request to the closest node that contains a replica of the data–– whether it be a follower or the leaseholder.
When to use follower reads
You should not use follower reads when your application cannot tolerate reading data that was current as of the follower read timestamp, since the results of follower reads may not reflect the latest writes against the tables you are querying.
In addition, follower reads are "read-only" operations; they cannot be used in any way in read-write transactions.
Using follower reads
Enable/disable follower reads
SET CLUSTER SETTING to set
trueto enable follower reads (default)
falseto disable follower reads
> SET CLUSTER SETTING kv.closed_timestamp.follower_reads_enabled = false;
If you have follower reads enabled, you may want to verify that follower reads are happening.
If follower reads are enabled, but the time-travel query is not using
AS OF SYSTEM TIME far enough in the past (as defined by the follower read timestamp), CockroachDB does not perform a follower read. Instead, the read accesses the leaseholder replica. This adds network latency if the leaseholder is not the closest replica to the gateway node.
Verify that follower reads are happening
To verify that your cluster is performing follower reads:
- Make sure that follower reads are enabled.
- Go to the Custom Chart Debug Page in the Admin UI and add the metric
follower_read.success_countto the time series graph you see there. The number of follower reads performed by your cluster will be shown.
Run queries that use follower reads
To simplify this calculation, we've added a convenience function that will always set the
AS OF SYSTEM TIME value to the minimum required for follower reads,
SELECT ... FROM ... AS OF SYSTEM TIME experimental_follower_read_timestamp();
Run read-only transactions that use follower reads
You can set the
AS OF SYSTEM TIME value for all operations in a read-only transaction:
BEGIN; SET TRANSACTION AS OF SYSTEM TIME experimental_follower_read_timestamp(); SELECT ... SELECT ... COMMIT;
Note that follower reads are "read-only" operations; they cannot be used in any way in read-write transactions.
Follower reads and long-running writes
Long-running write transactions will create write intents with a timestamp near when the transaction began. When a follower read encounters a write intent, it will often end up in a "transaction wait queue", waiting for the operation to complete; however, this runs counter to the benefit follower reads provides.
To counteract this, you can issue all follower reads in explicit transactions set with
BEGIN PRIORITY HIGH AS OF SYSTEM TIME experimental_follower_read_timestamp(); SELECT ... SELECT ... COMMIT;
- Cluster Settings Overview
- Load-Based Splitting
- Network Latency Page
- Enterprise Features
- Follower Reads Topology Pattern