Favourite strace options

I will try to avoid writing next article about Linux developer/admin swiss army knife, which is obviously strace. Instead, I would like to focus on most useful options:

Follow forks -f

If you try to debug your multithreading application, don’t forget about -f option. After that, strace will track all new threads and child processes.

This is really useful for checking what’s going on, after any sort of deamon, like sshd or getty, accept new client.

For example, I want to catch any file operations made by new, logged on user on SSH:

# pgrep sshd # (1) returns sshd PID -> 256
256
# strace -p 256 -e trace=file -f
 ... a lot of spam (checking user authentity in files like /etc/passwd...
[pid  9883] open("/root/.ash_history", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0600) = 3
[pid  9883] stat64("/bin/cat", {st_mode=S_IFREG|S_ISUID|0755, st_size=655180, ...}) = 0
Process 9894 attached
[pid  9894] execve("/bin/cat", ["cat", "/etc/network/interfaces"], [/* 20 vars */]) = 0
...ld and dynamic linking stuff...
[pid  9894] open("/etc/network/interfaces", O_RDONLY|O_LARGEFILE) = 3
[pid  9894] +++ exited with 0 +++
[pid  9883] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=9894, si_uid=0, si_status=0, si_utime=1, si_stime=3} ---
# # (4)

1. If you don’t have pgrep, just use top or htop. pgrep might return several PIDs, which are main sshd process and it’s children (existing sessions). Probably it will be first one.
2. -e trace=fileis my next favourite option. It tracks all syscalls related to file manipulation and reading.
3. This guy executed cat /etc/network/interfaces

Filtering out syscalls

strace have great filtering options. Embedded systems quite often has some limits on it, but it’s still quite useful

• Track file related syscalls: -e trace=file
• Filtering precise syscall: -e trace=mmap2.
• Track network stuff, but not sendto/recvfrom: -e trace=network -e trace=\!sendto -e trace=\!recvfrom – some shells need backslash before exclamation mark
• Last resort, if those filters not fit, you can use grep. This example show syscalls which fails on errno EAGAIN: strace -p XXX 2>&1 | grep EAGAIN – remember, that strace output is taken into stderr. To pipe it together with stdout you need add 2>&1. grep is much more flexible, however there is big efficiency impact without using strace builtin filters. Each process usually make huge amount of syscalls, and all of them are piped.

Performance analysis

Really nice feature is measuring time spent in each syscall. You can make it with -T option, and check how many seconds it lasts. Remember that if your process is preempted on a syscall, the time of other jobs is accumulated into measurement. That’s beacuse time is measured between entering syscall and returning it’s exit code.

Usually useful is adding -tt option which prefixes all records with microsecond timestamp. It lets you correlate syscalls with other logs.

Worth noting here is possibility to track several processes. You can use -p option multiple times and give several PIDs. I’ve used this option recently to track how threads connected through IPC are preempting each other.

Don’t forget…

…that ever debug tool have impact on system you follow. In this case every syscall generates huge console output, which slows down execution.

Strace implementation are different. Especially on embedded systems not all options are available

Obviously you must have root privileges to use strace.