Profiling is often associated with performance optimization, but it has many other valuable use cases. Here are 9 of my favorites:
- Detect Workload Shifts: Handler frames get wider when they get more work.
- Detect Data Nesting: Frames often get deeper when processing more nested data.
- Detect Dead Code: If a profile shows a function, you can reject the hypothesis that the code is unused. This can also help with prioritizing security fixes for new vulnerabilities.
- Debug Unresponsiveness: I once used profiling to diagnose a PostgreSQL vacuum process stuck in an infinite loop due to index corruption. Loops with O(N^2) scaling issues also usually stand out in profiles.
- Identify Configuration Issues: Profiling can be used to identify bottlenecks caused by misconfiguration. For example spending a lot of CPU time in read/write syscalls is often caused by a lack of buffering. Similarly, a high amount of CPU time in TLS can indicate connection churn problems.
- Debug Errors: If your error rates get high, your profile will show you the error path in the code.
- Verify Deployments: New frames appear, old frames disappear, regressions become visible.
- Improve Maintainability: The call hierarchies in your profiles can reveal deficiencies in the structure of your application.
- Faster Onboarding: Studying the hot code paths of an application is a good way to ramp up quickly.
Profiling may not be the ideal tool for all of these use cases, but no observability signal can give you such a broad visibility. Logging, tracing and metrics all depend on having the right instrumentation at the right time.
Disclaimer: I work on profiling for Datadog, but I'm not barking on behalf of my employer here. I've been meaning to write this down for a while now since it often comes up in conversations.