Tracing in Linux
How Bpftrace Can Help you to Resolve any Bug in Production
When a harmful bug occurs in the production environment, what’s the best way to eliminate it once you’ve checked the logs in the application, as well as the container layer, but still haven’t found any helpful information? Perhaps you could restart the server but there’s actually another alternative available for finding the root cause of the issue.
Bpf-trace is a high-level tracing language for Linux enhanced Berkeley Packet Filter (eBPF) available in recent Linux kernels (4.x). It uses LLVM as a backend to compile scripts to BPF-bytecode and makes use of BCC for interacting with the Linux BPF system, as well as existing Linux tracing capabilities, kernel dynamic tracing (kprobes), user-level dynamic tracing (uprobes), and tracepoints. Linux already has many performance tools, but they are often counter-based and have limited visibility. For example, iostat or a monitoring agent may tell you your average disk latency but not the distribution of this latency. Distributions can reveal multiple modes or outliers, either of which may be the real cause of your performance problems. Bpftrace is suited for this kind of analysis: decomposing metrics into distributions or per-event logs and creating new metrics for visibility into blind spots.
Terminology
Probe types
Getting hands-on with bpftrace
- Check Linux kernel version
This is what the official page of bpftrace said for installing:
It is recommended that you are running a Linux 4.9 kernel or higher. Some tools may work on older kernels, but these old kernels are no longer tested. To explain this requirement, these are the kernel versions where major features were added:
4.1 — kprobes
4.3 — uprobes
4.6 — stack traces, count and hist…
