How to Debug systemd Service Failures

Introduction

When a service fails to start or crashes on your Breeze, systemd provides comprehensive tools to diagnose the problem. Understanding how to read service status, check logs, and analyze unit files is essential for keeping your server running smoothly.

Step 1: Check Service Status

# Detailed status output
sudo systemctl status myapp

# Key things to look for:
# - "Active: failed" or "Active: inactive"
# - The "Main PID" and exit code
# - The last few log lines shown at the bottom

Step 2: Read the Full Logs

# Full logs for the service
journalctl -u myapp -n 100 --no-pager

# Only errors
journalctl -u myapp -p err

# Logs since last boot
journalctl -u myapp -b

# Follow in real time while you try to start
journalctl -u myapp -f &
sudo systemctl start myapp

Step 3: Examine the Unit File

# Show the full unit file
systemctl cat myapp

# Show effective configuration (including overrides)
systemctl show myapp

# Key properties to check
systemctl show myapp -p ExecStart,WorkingDirectory,User,Group,Environment

Common Failure Reasons

1. Wrong ExecStart Path or Missing Binary

# Verify the binary exists and is executable
ls -la /usr/local/bin/myapp
which myapp

# Test running the command manually as the service user
sudo -u myapp-user /usr/local/bin/myapp

2. Permission Issues

# Check that the service user can access required files
sudo -u www-data ls -la /var/www/myapp/
sudo -u www-data cat /etc/myapp/config.yml

# Fix ownership
sudo chown -R www-data:www-data /var/www/myapp/

3. Port Already in Use

# Check what is using the port
sudo ss -tlnp | grep :8080
sudo lsof -i :8080

# Kill the conflicting process or change the port

4. Missing Dependencies

# Check if required services are running
systemctl is-active mysql
systemctl is-active redis

# Check the unit file for dependency declarations
systemctl cat myapp | grep -E "After=|Requires=|Wants="

5. Environment Variables Not Set

# Check environment in the unit file
systemctl show myapp -p Environment

# Add environment variables via override
sudo systemctl edit myapp
# Add:
# [Service]
# Environment="DB_HOST=localhost"
# Environment="DB_PORT=3306"
# Or use an environment file:
# EnvironmentFile=/etc/myapp/env

6. Service Restarting Too Quickly

# Check for "start-limit-hit"
systemctl status myapp | grep "start-limit"

# Reset the failure counter
sudo systemctl reset-failed myapp

# Adjust restart limits in the unit file
# [Unit]
# StartLimitIntervalSec=60
# StartLimitBurst=5
# [Service]
# RestartSec=5

Useful Debug Techniques

# Run the service in foreground mode for debugging
sudo -u myapp-user /usr/local/bin/myapp --foreground --debug

# Check for core dumps
coredumpctl list
coredumpctl info PID

# Validate unit file syntax
systemd-analyze verify /etc/systemd/system/myapp.service

# Check boot order and timing
systemd-analyze blame
systemd-analyze critical-chain myapp.service

Creating a Service Override

# Create an override without editing the original
sudo systemctl edit myapp

# This creates /etc/systemd/system/myapp.service.d/override.conf
# Add your overrides:
[Service]
Environment="DEBUG=true"
ExecStartPre=/usr/bin/echo "Starting myapp"
TimeoutStartSec=120

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart myapp

Troubleshooting Checklist

• Read systemctl status and note the exit code
• Check journalctl -u service for detailed error messages
• Verify the binary path, permissions, and working directory
• Test running the command manually as the service user
• Check for port conflicts and missing dependencies
• Review environment variables and configuration files
• Use systemd-analyze verify to validate unit file syntax