Automatic Go Execution Tracer

The automatic Go execution tracer allows operators and support engineers to investigate internal system behavior over time. Use this feature to gain fine-grained insight into goroutine scheduling, blocking events, garbage collection, and runtime activity.

Execution tracing introduces a performance overhead. This feature is primarily intended for use under the guidance of Cockroach Labs Support.

When enabled, the automatic Go execution tracer captures runtime execution traces on a recurring schedule and stores them in the node’s log directory. These traces are collected in the background and can be analyzed using Go’s standard tooling (e.g., go tool trace). To prevent disk overuse, older traces are automatically garbage collected based on a configurable storage limit.

Configuration

You can configure automatic Go execution trace capture with the following :

Cluster Setting	Description	Default Value
`obs.execution\_tracer.interval`	Enables periodic execution tracing on each node and defines how often traces are collected. Set to a positive duration (e.g., `10m`, `1h`) to enable. Set to `0` to disable (default).	`0`
`obs.execution\_tracer.duration`	Sets how long each execution trace runs. Use short durations (e.g., 5-15 seconds) to produce traces that are easier to analyze.	`10s`
`obs.execution\_tracer.total\_dump\_size\_limit`	Sets a soft limit on the total disk space used by execution trace files. When the limit is exceeded, the oldest files are deleted.	`4GiB`

Accessing Go execution traces

The automatic Go execution tracer saves the captured traces to disk on each node’s file system in the . The default path is cockroach-data/logs/executiontrace_dump.
To access these Go execution traces, you must use the node file system.
Each file represents a single trace session, includes the executiontrace. prefix and .out extension, and is timestamped with the session end time to support correlation with other logs and profiles. For example, the trace file could be named executiontrace.2025-07-14T18_41_22.216.out.
Enabling the automatic Go execution tracer does add Go execution traces to in the nodes/*/executiontrace paths.

Best Practices

Trace duration should be short (e.g., 5s–15s). Shorter trace files are easier to parse and load into external tools.
Storage limit should be generous to allow sufficient trace history. Avoid overly tight disk caps unless necessary.
Enable tracing temporarily when troubleshooting runtime behavior or under support guidance, as it may slightly impact performance.

Performance Impact

Enabling execution tracing incurs a low overhead, estimated at 1–2% based on upstream Go benchmarks (Go blog, 2024). No additional CockroachDB-specific benchmarks are available at this time.

Example

To enable 10-second traces every 15 minutes with a storage limit of 8GiB, configure the following settings:

SET CLUSTER SETTING obs.execution_tracer.interval = '15m';
SET CLUSTER SETTING obs.execution_tracer.duration = '10s';
SET CLUSTER SETTING obs.execution_tracer.total_dump_size_limit = '8GiB';

Troubleshooting

Automatic Go Execution Tracer

Configuration

Accessing Go execution traces

Best Practices

Performance Impact

Example

See also

​Configuration

​Accessing Go execution traces

​Best Practices

​Performance Impact

​Example

​See also

Configuration

Accessing Go execution traces

Best Practices

Performance Impact

Example

See also