How to Respond to a Server Compromise: Incident Response Playbook

When you suspect your server has been compromised, every minute counts. Having a pre-defined incident response playbook ensures you respond quickly and effectively, minimizing damage and preserving evidence.

Phase 1: Detection and Confirmation

# Signs of compromise:
# - Unexpected processes or services running
# - Modified system files (check with tripwire/aide)
# - Unusual network connections
# - Unexplained CPU/memory usage (cryptomining)
# - New user accounts or SSH keys
# - Modified crontabs
# - Suspicious log entries or cleared logs

# Quick triage commands:
w                          # Who is logged in right now?
last -20                   # Recent logins
ps auxf                    # Process tree (look for anomalies)
ss -tlnp                   # Listening ports (anything unexpected?)
crontab -l                 # Cron jobs for current user
sudo cat /etc/crontab      # System cron
find /tmp -type f -executable  # Executables in /tmp
cat /etc/passwd | grep ":0:"   # Check for extra root users

Phase 2: Containment

# CRITICAL: Do NOT shut down the server yet
# You need to preserve evidence in memory

# Option A: Network isolation (preferred)
# Block all traffic except your SSH session
sudo iptables -I INPUT -s YOUR_IP -j ACCEPT
sudo iptables -I OUTPUT -d YOUR_IP -j ACCEPT
sudo iptables -A INPUT -j DROP
sudo iptables -A OUTPUT -j DROP

# Option B: If available, take a snapshot first
# Use your provider portal to snapshot the running server
# This preserves the complete state for forensics

Phase 3: Evidence Collection

# Collect evidence BEFORE any cleanup
# Save to an external location if possible

# 1. Process listing
ps auxf > /tmp/evidence/processes.txt

# 2. Network connections
ss -tlnp > /tmp/evidence/listening.txt
ss -tnp > /tmp/evidence/connections.txt

# 3. User accounts
cat /etc/passwd > /tmp/evidence/passwd.txt
cat /etc/shadow > /tmp/evidence/shadow.txt

# 4. SSH authorized keys for all users
find /home -name authorized_keys -exec echo "=== {} ===" \; -exec cat {} \; > /tmp/evidence/ssh-keys.txt

# 5. Cron jobs
for user in $(cut -f1 -d: /etc/passwd); do
  echo "=== $user ===" >> /tmp/evidence/crontabs.txt
  crontab -u $user -l 2>/dev/null >> /tmp/evidence/crontabs.txt
done

# 6. Recently modified files
find / -mtime -2 -type f 2>/dev/null > /tmp/evidence/recent-files.txt

# 7. Log files
tar czf /tmp/evidence/logs.tar.gz /var/log/

# 8. Full memory dump (if tools available)
# sudo dd if=/dev/mem of=/tmp/evidence/memory.dump

Phase 4: Eradication and Recovery

1. Rebuild from clean image — Never trust a compromised server. Deploy a new server.
2. Restore from known-good backup — Use backups from BEFORE the compromise
3. Change ALL credentials — SSH keys, database passwords, API tokens, SSL certificates
4. Patch the vulnerability — Identify how the attacker got in and fix it
5. Verify the backup — Scan restored data for backdoors or modifications

Phase 5: Post-Incident

• Document the full timeline of the incident
• Identify the root cause and entry vector
• Update security controls to prevent recurrence
• Review and improve monitoring and alerting
• Notify affected parties if data was exposed
• Conduct a lessons-learned meeting

Incident Response Checklist

1. Confirm the compromise (not a false alarm)
2. Notify the incident response team
3. Isolate the affected server
4. Take a snapshot for forensics
5. Collect evidence (processes, connections, logs)
6. Identify the entry vector
7. Deploy a clean replacement server
8. Restore from clean backups
9. Rotate all credentials
10. Monitor for re-compromise
11. Document and report