2 min read

Waiting for go1.21: Execution Tracing with < 1% Overhead

Learn about the upcoming go1.21 performance improvements that reduce the overhead of execution tracing from up to 20% to less than 1% for most applications.
Waiting for go1.21: Execution Tracing with < 1% Overhead

Go’s execution tracer (aka go tool trace) has suffered from high overhead since its inception in 2014. Historically this has forced potential users to worry about up to 20% of CPU overhead when turning it on. Due to this, it's mostly been used in test environments or tricky situations rather than gaining adoption as a continuous profiling signal in production.

However, in early 2023 my colleague Nick Ripley and I started to become convinced that we could change this by implementing frame pointer unwinding. You can read more about the details in the blog post below:

Reducing Go Execution Tracer Overhead With Frame Pointer Unwinding
Learn how frame pointer unwinding could significantly reduce the CPU overhead of the Go execution tracer.

Since then we’ve been actively collaborating with the Go runtime team at Google, in particular Michael Knyszek, Michael Pratt and Cherry Mui in order to make this a reality. And at this point it looks like we’ve succeeded: The go1.21 release coming out in ~August will feature a new version of the tracer which should provide less than 1% overhead for most applications 🥳 (see upcoming release notes).

Below is a list of directly or indirectly related patches that contributed to this effort:

So if you're into Go performance and observability, this improvement is definitely something to look forward to.

And this is just the beginning:

  1. Michael Knyszek has proposed a lot of exciting improvements that will further improve execution tracing in upcoming releases.
  2. Dominik Honnef's gotraceui viewer keeps improving.
  3. We've also been busy with integrating the execution tracer into our profiling product at Datadog. We'll share more information about this soon.