Publication date: February 12, 2020
Description
When running SHOW JOBS or viewing the Jobs page in the Admin UI, high memory usage can be incurred on a node to the point it could crash.
Statement
SHOW JOBS and the Jobs page in the Admin UI internally load all the job descriptions from the cluster in RAM before displaying them.
Under reasonable production settings, a single backup job payload may exceed 5MB in size. Considering an hourly backup and default property for jobs.retention_time set to 336h, a single use of SHOW JOBS or a single user of the Jobs page in the Admin UI can incur ~1.7GB of memory utilization. This allocation is then multiplied by the number of concurrent accesses to the jobs table.
Starting in CockroachDB v19.2.3, new jobs payloads are reduced in size. A later version will also avoid loading old job entries in memory when viewing recent jobs.
This public issue is tracked as #44166.
Mitigation
It is possible to reduce the number of job entries overall by setting the jobs.retention_time cluster setting to a value closer to 48h or 24h.
For example:
SET CLUSTER SETTING jobs.retention_time='48:00:00'.
Additionally, if the nodes are observed to crash due to excessive memory usage, it may be necessary to truncate the job history. This can be achieved, for example, with:
DELETE from system.jobs
WHERE status = 'succeeded'
AND created < (now() - '2 days'::interval);
Impact
All deployments running CockroachDB v19.2.0 to v19.2.2 are affected.
Questions about any technical alert can be directed to our support team.
Was this helpful?
