Follower reads are a mechanism that CockroachDB uses to provide faster reads in situations where you can afford to read data that may be slightly less than current (using AS OF SYSTEM TIME). Normally, reads have to be serviced by a replica's leaseholder. This can be slow, since the leaseholder may be geographically distant from the gateway node that is issuing the query.
A follower read is a read taken from the closest replica, regardless of the replica's leaseholder status. This can result in much better latency in geo-distributed, multi-region deployments.
The shortest interval at which AS OF SYSTEM TIME can serve follower reads is 4.8 seconds. In prior versions of CockroachDB, the interval was 48 seconds.
For instructions showing how to use follower reads to get low latency, historical reads in multi-region deployments, see the Follower Reads Topology Pattern.
This is an Enterprise feature.
Watch the demo
How follower reads work
Each CockroachDB node tracks a property called its "closed timestamp", which means that no new writes can ever be introduced below that timestamp. The closed timestamp advances forward by some target interval behind the current time. If the replica receives a write at a timestamp less than its closed timestamp, it rejects the write.
With follower reads enabled, any replica on a node can serve a read for a key as long as the time at which the operation is performed (i.e., the AS OF SYSTEM TIME value) is less or equal to the node's closed timestamp.
When a gateway node in a cluster with follower reads enabled receives a request to read a key with a sufficiently old AS OF SYSTEM TIME value, it forwards the request to the closest node that contains a replica of the data–– whether it be a follower or the leaseholder.
When to use follower reads
As long as your SELECT operations can tolerate reading data that was current as of the follower read timestamp, follower reads can reduce read latencies and increase throughput.
You should not use follower reads when your application cannot tolerate reading data that was current as of the follower read timestamp, since the results of follower reads may not reflect the latest writes against the tables you are querying.
In addition, follower reads are "read-only" operations; they cannot be used in any way in read-write transactions.
Using follower reads
Enable/disable follower reads
Use SET CLUSTER SETTING to set kv.closed_timestamp.follower_reads_enabled to:
trueto enable follower reads (default)falseto disable follower reads
> SET CLUSTER SETTING kv.closed_timestamp.follower_reads_enabled = false;
If you have follower reads enabled, you may want to verify that follower reads are happening.
If follower reads are enabled, but the time-travel query is not using AS OF SYSTEM TIME far enough in the past (as defined by the follower read timestamp), CockroachDB does not perform a follower read. Instead, the read accesses the leaseholder replica. This adds network latency if the leaseholder is not the closest replica to the gateway node.
Verify that follower reads are happening
To verify that your cluster is performing follower reads:
- Make sure that follower reads are enabled.
- Go to the Custom Chart Debug Page in the DB Console and add the metric
follower_read.success_countto the time series graph you see there. The number of follower reads performed by your cluster will be shown.
Run queries that use follower reads
Any SELECT statement with an AS OF SYSTEM TIME value at least 4.8 seconds in the past can be a follower read (i.e., served by any replica).
To simplify this calculation, we've added the convenience function follower_read_timestamp(). follower_read_timestamp() returns the TIMESTAMP statement_timestamp() - 4.8s (known as the follower read timestamp). Using this function in an AS OF SYSTEM TIME statement will set the time as close as possible to the present time while remaining safe for follower reads:
SELECT ... FROM ... AS OF SYSTEM TIME follower_read_timestamp();
Run read-only transactions that use follower reads
You can set the AS OF SYSTEM TIME value for all operations in a read-only transaction:
BEGIN;
SET TRANSACTION AS OF SYSTEM TIME follower_read_timestamp();
SELECT ...
SELECT ...
COMMIT;
Note that follower reads are "read-only" operations; they cannot be used in any way in read-write transactions.
Using the SET TRANSACTION statement as shown in the example above will make it easier to use the follower reads feature from drivers and ORMs.
Follower reads and long-running writes
Long-running write transactions will create write intents with a timestamp near when the transaction began. When a follower read encounters a write intent, it will often end up in a "transaction wait queue", waiting for the operation to complete; however, this runs counter to the benefit follower reads provides.
To counteract this, you can issue all follower reads in explicit transactions set with HIGH priority:
BEGIN PRIORITY HIGH AS OF SYSTEM TIME follower_read_timestamp();
SELECT ...
SELECT ...
COMMIT;
See Also
- Cluster Settings Overview
- Load-Based Splitting
- Network Latency Page
- Enterprise Features
- Follower Reads Topology Pattern
Was this helpful?
