Description
Cockroach Labs has discovered a race condition, where a long running read request submitted during replica removal may be evaluated on the removed replica, returning an empty result. Common causes of replica removal are range merges and replica rebalancing in the cluster. When a replica is removed there is a small window, where the data is already deleted but a read request will still see it as a valid replica. In this case a read request will always return an empty result for the part of the query that relied on this range. This is a rare scenario that can only be encountered if a number of conditions are met:- A read request checks that the node is holding the lease and begins processing
- The node loses the lease before the read request is able to lock the replica for reading
- The read request checks the state of the replica and makes sure it is alive
- The new leaseholder decides to immediately remove the old leaseholder replica from the range, due to a rebalance or range merge.
- The removal is processed before the read requests starts reading the data from the replica
- The read request observes the replica in a post-deletion state returning an empty result set.