Sysdig for Container Troubleshooting

Sysdig is a powerful system exploration and troubleshooting tool that provides deep visibility into container and system activity. It captures system calls, network connections, and file operations at the kernel level, making it invaluable for debugging containerized applications. This guide covers using sysdig and its UI tool csysdig for container troubleshooting.

Installation

# Ubuntu/Debian
curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

# Docker (no host installation needed)
docker run -d --name sysdig \
    --privileged \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /src:/src:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    sysdig/sysdig

Basic Usage

# See all system calls
sudo sysdig

# Filter by container
sudo sysdig container.name=web

# See network connections for a container
sudo sysdig -c netstat container.name=web

# See file opens for a container
sudo sysdig "container.name=web and evt.type=open"

# Top processes by CPU in a container
sudo sysdig -c topprocs_cpu container.name=web

# Top files by I/O
sudo sysdig -c topfiles_bytes container.name=db

# Top network connections
sudo sysdig -c topconns container.name=api

csysdig (Interactive UI)

# Launch interactive UI
sudo csysdig

# Container-specific views
sudo csysdig -v containers          # Container overview
sudo csysdig -v container_resources # Resources per container
sudo csysdig container.name=web     # Focus on specific container

# Key bindings:
# F2    — switch view
# Enter — drill down
# F4    — filter
# F7    — legend
# q     — quit

Troubleshooting Scenarios

Network Issues

# See all network connections for a container
sudo sysdig -c netstat container.name=api

# Monitor DNS lookups
sudo sysdig -A "container.name=api and fd.sport=53"

# Track connection timeouts
sudo sysdig "container.name=api and evt.type=connect and evt.res=ETIMEDOUT"

# Capture network traffic
sudo sysdig -w capture.scap "container.name=api and fd.type=ipv4"
# Analyze later
sudo sysdig -r capture.scap -c netstat

File System Issues

# See all file writes
sudo sysdig "container.name=app and evt.type=write and fd.type=file" -p "%evt.time %proc.name %fd.name %evt.buflen"

# Find which process is filling up disk
sudo sysdig -c topfiles_bytes container.name=app

# Monitor log file writes
sudo sysdig "container.name=app and fd.name contains /var/log" -p "%evt.time %proc.name → %fd.name (%evt.buflen bytes)"

Process Issues

# See all processes in a container
sudo sysdig -c topprocs_cpu container.name=app

# Track process creation
sudo sysdig "container.name=app and evt.type=execve" -p "%evt.time %proc.name %proc.args"

# Monitor failed operations
sudo sysdig "container.name=app and evt.failed=true" -p "%evt.time %proc.name %evt.type %evt.res"

# See what a process is doing
sudo sysdig "container.name=app and proc.name=node" -p "%evt.time %evt.type %evt.info"

Capture and Replay

# Capture system activity to file (for offline analysis)
sudo sysdig -w /tmp/capture.scap container.name=problematic-app

# Replay capture
sudo sysdig -r /tmp/capture.scap -c topprocs_cpu
sudo sysdig -r /tmp/capture.scap -c topfiles_bytes
sudo csysdig -r /tmp/capture.scap

Falco (Security Monitoring)

# Falco (from Sysdig) detects abnormal container behavior
# Detect shell in container
- rule: Terminal shell in container
  condition: container and proc.name = bash and evt.type = execve
  output: "Shell spawned in container (container=%container.name command=%proc.cmdline)"
  priority: WARNING

Best Practices

• Use sysdig for deep troubleshooting when container logs are insufficient
• Capture system calls to a file during incidents for post-mortem analysis
• Use csysdig for interactive exploration of container behavior
• Monitor failed system calls (evt.failed=true) to find permission and resource issues
• Use Falco for continuous security monitoring of container runtime behavior
• Run sysdig temporarily during troubleshooting — it has overhead on busy systems