Troubleshoot systemd Service Failures

systemd manages virtually every service on modern Linux systems, so understanding how to troubleshoot service failures is essential. This guide provides a systematic approach to diagnosing why a systemd service won't start, keeps crashing, or behaves unexpectedly.

Step 1: Check Service Status

# Get full service status with recent log output
sudo systemctl status nginx.service

# Key fields to look at:
# Active: failed (Result: exit-code)  → service crashed
# Active: inactive (dead)             → service is stopped
# Active: activating (auto-restart)   → service is crash-looping

# Check the exit code
# Main PID: 12345 (code=exited, status=1/FAILURE)
# status=1: general error
# status=2: misuse of shell command
# status=127: command not found
# status=137: killed by signal 9 (OOM)
# status=203: exec format error (wrong binary)

# Check if service is enabled
systemctl is-enabled nginx.service

Step 2: Read the Logs

# View service logs
journalctl -u nginx.service -n 50 --no-pager

# Follow logs in real-time
journalctl -u nginx.service -f

# View logs from the last failed start
journalctl -u nginx.service --since "5 minutes ago"

# View logs from a specific boot
journalctl -u nginx.service -b -1  # Previous boot

# Show only error-level messages
journalctl -u nginx.service -p err

# If journald logs are missing, check syslog
grep nginx /var/log/syslog
grep nginx /var/log/messages

Step 3: Validate the Service File

# Show the service file
systemctl cat nginx.service

# Verify service file syntax
systemd-analyze verify nginx.service

# Check for common issues:
# - ExecStart path doesn't exist
# - User/Group doesn't exist
# - Working directory doesn't exist
# - Missing dependencies

# Check the actual path of the binary
which nginx
ls -la /usr/sbin/nginx

# Test the binary directly
sudo /usr/sbin/nginx -t  # Nginx config test
sudo /usr/bin/node /opt/app/server.js  # Run manually

Step 4: Diagnose Common Failure Patterns

Service Starts Then Immediately Stops

# Usually: the process forks/backgrounds itself, but systemd thinks it exited
# Fix: Set the correct Type in the service file

# For processes that fork:
[Service]
Type=forking
PIDFile=/run/nginx.pid

# For processes that stay in foreground:
[Service]
Type=simple  # or Type=exec

# For processes that notify systemd when ready:
[Service]
Type=notify

# Test: run the ExecStart command manually and observe behavior

Service Crash-Loops

# Check restart settings
systemctl show nginx.service | grep -E "Restart|StartLimit"

# Restart=always with StartLimitBurst=5 and StartLimitIntervalSec=10
# means: if it fails 5 times in 10 seconds, stop trying

# Reset failed state
sudo systemctl reset-failed nginx.service

# Adjust restart policy
[Service]
Restart=on-failure
RestartSec=5
StartLimitBurst=3
StartLimitIntervalSec=60

Permission Denied Errors

# Check file permissions
ls -la /opt/app/server.js
ls -la /var/log/app/

# Check the User/Group the service runs as
systemctl show nginx.service | grep -E "^User|^Group"

# Common fixes:
sudo chown -R www-data:www-data /opt/app/
sudo chmod 755 /opt/app/server.js

# For binding to ports below 1024:
# Option 1: Use capabilities
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Option 2: Run as root (not recommended)
# Option 3: Use a reverse proxy on port 80, app on port 3000+

Port Already in Use

# Find what's using the port
sudo ss -tlnp | grep :80
sudo lsof -i :80

# Kill the conflicting process
sudo kill $(sudo lsof -t -i :80)
# Or
sudo fuser -k 80/tcp

# Then restart the service
sudo systemctl restart nginx

Step 5: Environment and Dependencies

# Check if environment variables are set correctly
systemctl show nginx.service | grep Environment

# View the full execution environment
sudo systemd-run --scope -p User=www-data env

# Check dependency order
systemctl list-dependencies nginx.service

# If the service needs network, database, etc:
[Unit]
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

Step 6: SELinux/AppArmor Issues

# Check if SELinux is blocking
sudo ausearch -m AVC -ts recent
sudo sealert -a /var/log/audit/audit.log

# Check AppArmor
sudo aa-status
sudo journalctl | grep -i apparmor | tail -20

# Temporarily disable to test
sudo setenforce 0  # SELinux permissive
sudo aa-complain /usr/sbin/nginx  # AppArmor complain mode

Useful Debugging Commands

# Analyze boot timing
systemd-analyze blame
systemd-analyze critical-chain nginx.service

# List all failed units
systemctl --failed

# Show all properties of a service
systemctl show nginx.service

# Start service with debugging
sudo SYSTEMD_LOG_LEVEL=debug systemctl start nginx.service
journalctl -u nginx.service -n 100

Best Practices

• Always check logs first: journalctl -u service -n 50 gives you the answer 90% of the time
• Test the binary manually with the same user and environment as the service file
• Use systemd-analyze verify to catch syntax errors in service files
• Set appropriate Restart and RestartSec values to handle transient failures
• Use Type=exec (newer systemd) or Type=simple for foreground processes
• Check systemctl --failed regularly — failed services may go unnoticed