How to Trace System Calls with ltrace and strace

When an application behaves unexpectedly — crashing, hanging, or producing wrong output — tracing its system calls and library calls can reveal exactly what is happening under the hood. strace and ltrace are essential debugging tools for any Linux administrator.

strace: System Call Tracing

strace intercepts and records system calls made by a process — every file open, read, write, network connection, and memory allocation.

Basic Usage

# Trace a command
strace ls /var/www

# Trace with timestamps
strace -t ls /var/www

# Trace with relative timestamps (time between calls)
strace -r ls /var/www

# Follow child processes (fork/clone)
strace -f nginx

# Trace a running process by PID
sudo strace -p 1234

# Save output to a file
strace -o /tmp/trace.log ls /var/www

Filtering System Calls

# Trace only file-related calls
strace -e trace=file ls /var/www

# Trace only network calls
strace -e trace=network curl http://example.com

# Trace only process management calls
strace -e trace=process bash -c "ls"

# Trace specific system calls
strace -e open,read,write cat /etc/hostname

# Available trace categories:
# file     — open, stat, chmod, unlink, rename
# process  — fork, exec, wait, exit
# network  — socket, connect, accept, send, recv
# signal   — signal, sigaction, kill
# ipc      — shmget, semget, msgget
# memory   — mmap, brk, mprotect

Practical strace Examples

# Find out which config files a program reads
strace -e openat nginx -t 2>&1 | grep "openat"

# Diagnose a segfault
strace -f ./crashed-program 2>&1 | tail -20

# Find out why a program hangs
sudo strace -p $(pgrep hung-process)
# The last system call shown is where it is stuck

# Measure time spent in system calls
strace -c ls /var/www
# Shows a summary table:
# % time  seconds  usecs/call  calls  errors  syscall
# 45.23   0.001234    12        100           read
# 30.12   0.000823     8        100           write

# Find permission denied errors
strace -e openat nginx -t 2>&1 | grep "EACCES"

ltrace: Library Call Tracing

ltrace traces calls to shared libraries (libc, libssl, etc.), showing function names and arguments.

# Install ltrace
sudo apt install ltrace

# Trace library calls
ltrace ls /var/www

# Trace specific library
ltrace -l libssl.so.3 openssl s_client -connect example.com:443

# Show only specific functions
ltrace -e malloc+free ls

# Count library calls
ltrace -c ls /var/www

When to Use strace vs ltrace

• strace — Low-level debugging: file access, network, permissions, system errors
• ltrace — Application-level debugging: library function calls, memory allocation, string operations

Performance Considerations

# WARNING: Tracing significantly slows down the traced process
# strace can make a process 10-100x slower
# Never use strace on production processes for extended periods

# For production debugging, consider:
# - perf trace (much lower overhead)
# - bpftrace (eBPF-based, minimal overhead)
# - SystemTap

# Quick production-safe trace (limited to 5 seconds)
timeout 5 sudo strace -c -p $(pgrep nginx | head -1)

Common Debugging Patterns

# Why does my app fail to start?
strace -f -e trace=file,network myapp 2>&1 | grep -i "error|fail|denied"

# What DNS queries is my app making?
strace -e trace=network -f myapp 2>&1 | grep -i "connect"

# What files does a program modify?
strace -e trace=write,rename,unlink -f myapp 2>&1

# Is my app reading the right config?
strace -e openat myapp 2>&1 | grep ".conf|.cfg|.ini|.env"