/systemtap-lwtools

SystemTap Lightweight Tools

GNU General Public License v2.0GPL-2.0

SystemTap Lightweight Tools (lwtools)

A miscellaneous collection of in-development and unsupported SystemTap tools and one-liners, which use a limited set of SystemTap capabilities (lightweight). These provide advanced observability for the analysis of system and cloud performance.

An aim of this collection is to explore what SystemTap can do using basic features and without the need for kernel debuginfo, which can be an onerous requirement. For a similar set of ftrace- and perf_events-based tools, see the perf tools collection, which is more developed. An advantage of these SystemTap tools is that they can perform custom in-kernel aggregations (eg, latency distributions) with low overhead.

Some expert assembly will likely be required with these SystemTap tools. Also see warnings.

Warnings

SystemTap is in-development kernel-level software, so there is the possibility of kernel panics or freezes. I've developed these for use in a lab environment, where panics can be tolerated.

Also, tools that use the inherently unstable dynamic tracing interface may either not work at all (print errors), or work and show bogus data, or work and show plausible but still incorrect data. The same is true for any dynamic tracing-based scripts (including my DTrace ones).

Note that the panics may be due to SystemTap or other kernel origins. Use crash dump analysis to determine the real cause, and please report bugs.

Prerequisites

SystemTap, for the stap command, and an environment that is fault tolerant (see Warnings). It is strongly recommended that you try the latest version of SystemTap, which probably means compiling from source. See git://sourceware.org/git/systemtap.git, and the build readme.

Where possible, these tools avoid the requirement of having kernel debuginfo.

Contents

There are also man pages under man/man8.

Screenshots

Histogram of syscall writes by requested size (one-liner):

# stap -e 'global a; probe nd_syscall.write { a <<< int_arg(3); } probe end { print(@hist_log(a)); }'
^Cvalue |-------------------------------------------------- count
    0 |                                                    0
    1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                35
    2 |@                                                   1
    4 |@                                                   1
    8 |                                                    0
   16 |@@@@@@@@@@@@@@@@@@@@@@                             22
   32 |@@@@@@@@@@@@@@@                                    15
   64 |@@@@@@@@@@@@@@@@@                                  17
  128 |@@                                                  2
  256 |@@                                                  2
  512 |                                                    0
 1024 |                                                    0

Tracing exec() syscalls to show new process execution:

# ./execsnoop-nd.stp
TIME                        UID   PPID    PID           COMM ARGS
Sat Jan 31 20:39:37 2015      0    878   5744           sshd /usr/sbin/sshd -R
Sat Jan 31 20:39:38 2015      0   5744   5746           bash -bash
Sat Jan 31 20:39:38 2015      0   5747   5748             id id -un
Sat Jan 31 20:39:38 2015      0   5749   5750       hostname /bin/hostname
Sat Jan 31 20:39:38 2015      0   5751   5752            tty tty -s
Sat Jan 31 20:39:38 2015      0   5751   5753           tput tput colors
Sat Jan 31 20:39:38 2015      0   5754   5755      dircolors dircolors --sh /etc/DIR_COLORS.256color
Sat Jan 31 20:39:38 2015      0   5746   5756           grep grep -qi ^COLOR.*none /etc/DIR_COLORS.256color
Sat Jan 31 20:39:40 2015      0   5746   5757             ls ls --color=auto
Sat Jan 31 20:39:40 2015      0   5746   5758           date date

Tracing open() syscalls with process and pathnames:

# ./opensnoop-nd.stp
   UID    PID             COMM   FD PATH
     0  13858           iostat    3 /etc/ld.so.cache
     0  13858           iostat    3 /lib64/libc.so.6
     0  13858           iostat    3 /usr/lib/locale/locale-archive
     0  13858           iostat    3 /sys/devices/system/cpu
     0  13858           iostat    3 /proc/diskstats
     0  13858           iostat    3 /etc/localtime
     0  13858           iostat    3 /proc/uptime
     0  13858           iostat    3 /proc/stat
     0  13858           iostat    3 /proc/diskstats
     0  13858           iostat    4 /etc/sysconfig/sysstat.ioconf
     0  13858           iostat    3 /proc/uptime
     0  13858           iostat    3 /proc/stat
     0  13858           iostat    3 /proc/diskstats
[...]

Summarizing read() and write() syscall latency for processes named "node":

# ./rwtime-nd.stp node
Tracing read/write syscalls for processes named "node"... Hit Ctrl-C to end.
^C
syscall read latency (ns):
value |-------------------------------------------------- count
  512 |                                                     0
 1024 |                                                     0
 2048 |                                                     2
 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     278
 8192 |                                                     1
16384 |                                                     5
32768 |                                                     0
65536 |                                                     0

syscall write latency (ns):
 value |-------------------------------------------------- count
  4096 |                                                     0
  8192 |                                                     0
 16384 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      135
 32768 |@@                                                   8
 65536 |                                                     0
131072 |                                                     0

Internals

These tools are designed to be short, simple, and documented, and use as few SystemTap capabilities as possible. They also avoid using kernel debuginfo, so that these scripts can be used in environments that lack it (eg, cloud instances that are created and destroyed quickly, where including debuginfo can be onerous). This can complicate or obfuscate some scripts.

Each tool has an examples file under examples, a man page under man/man8, a symlink in bin, and a link in this README.

See Also

For SystemTap tools that use its full capabilities, see the SystemTap examples page, particularly the "best examples" section, as well as the examples in the SystemTap source under share/doc/systemtap/examples. Most of these are likely to require kernel debuginfo.