XRay is a tool, originally developed at Google and now integrated in LLVM, that instruments the program such that when it runs it produces a trace of executed functions and their timestamps. This article explains how to instrument WiredTiger, collect the XRay traces, and analyze them. As an example, we will show how to trace wtperf
.
LLVM and graphviz packages are needed to build and run llvm-xray
command inside the wtperf_xray.sh script.
LLVM needs to be version 10 or higher. Check the version like this:
If your distribution's default llvm-config
isn't from the 10 or higher series, you'll need to move one with a major version of 10 or higher into the $PATH
such that it gets invoked instead.
The llvm-config
command line tool is used to supply the required compiler and linker flags to build programs such like xray_to_optrack
on top of LLVM.
Use the script wtperf_xray.sh
to launch wtperf
from the directory containing the wtperf
binary. The first argument to the script must be the benchmark configuration file. For example:
In general the usage is:
The wtperf_xray.sh
produces a few outputs to help analyze performance:
wtperf_account.txt:
The top 10 functions where the workload is spending the most time with a count, min, max and some percentiles for each one.wtperf_stacks.txt:
The top 10 stack traces where the workload is spending the most time. This calculation is done separately per thread.wtperf_graph.svg:
A function call graph showing what functions call each other. The edges are labeled and colored proportionally to represent the ratio of time spent in each function call.wtperf_flame.svg:
A graph visualizing stack traces and the time spent within each stack frame. If FLAME_GRAPH_PATH
is not specified, this graph won't be generated.The wtperf_xray.sh
script uses a few optional environment variables to modify its behavior:
XRAY_BINARY:
The binary to use to inspect the XRay log. The script defaults to using llvm-xray
however, if you compiled with a particular clang
version, you should use the corresponding llvm-xray
version. For example, if you selected clang-10
like the configuration above, you should set XRAY_BINARY
to llvm-xray-10
.FLAME_GRAPH_PATH:
As part of the script's analysis phase, it can optionally produce a FlameGraph. The FLAME_GRAPH_PATH
variable must be set to your copy of Brendan Gregg's FlameGraph script (flamegraph.pl
).After running a program instrumented with XRay, a log file will be produced containing performance information. There is a tool called xray_to_optrack
which is designed to convert this log to the Operation Tracking format.
To build the xray_to_optrack
utility, we'll need to reconfigure and build with a new set of flags. Before doing that, we'll need to keep a copy of the XRay log and the wtperf
binary that we used to generate it.
Supply -DENABLE_LLVM=1
to your configuration and have a clang toolchain installed in order to build.
Take care NOT to customize CC
or CXX
. Customizing either of these variables will cause C++ programs such as workgen
or xray_to_optrack
to be skipped since we can't reliably link object files emitted by C and C++ compilers unless they are the system's default cc
and c++
.
To process the traces, use the xray_to_optrack
tool in the tools/xray_to_optrack
directory.
Then, run the tool like this:
xray_instrumented_binary
is the binary that produced the log, wtperf
in our case, and xray_log
is the log file.
The script will produce one or more files with a prefix optrack
. You can view these files with optrack tools, described in the optrack documentation.