Fix No Space Left on Device Caused by Journal Logs

"No space left on device" errors can crash services, prevent logging, and make your server unmanageable. Often the culprit isn't obvious — hidden files, journal logs, or inode exhaustion can fill your disk invisibly. This guide covers finding and fixing all causes of disk space exhaustion.

Identify What's Using Space

# Check disk usage
df -h /
# If 100% used, investigate

# Find largest directories
du -h --max-depth=2 / 2>/dev/null | sort -rh | head -20

# Check systemd journal size
journalctl --disk-usage

# Journal is often the biggest culprit!
# Default: no size limit, can grow to gigabytes

# Clean journal immediately
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

Find Hidden Space Usage

# Deleted files still open (space not freed)
sudo lsof +L1 | head -20
# These files were deleted but a process still has them open
# The space won't be freed until the process closes the file

# Fix: Restart the process holding the deleted file
sudo systemctl restart rsyslog  # Common offender

# Or truncate the file descriptor
sudo truncate -s 0 /proc/PID/fd/FD_NUMBER

# Check for files in /tmp, /var/tmp
du -sh /tmp /var/tmp

# Check for old log files
find /var/log -name "*.gz" -mtime +30 | xargs du -ch | tail -1
# Clean them
find /var/log -name "*.gz" -mtime +30 -delete

# Check for old kernels
dpkg -l 'linux-image-*' | grep ^ii
sudo apt autoremove --purge

Inode Exhaustion

# Disk has space but "no space left"? Check inodes
df -i /
# If IUsed is at 100%, you're out of inodes (not disk space)

# Find directories with the most files
find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

# Common causes: session files, mail queue, temp files
find /tmp -type f | wc -l
find /var/spool -type f | wc -l
find /home -name ".sess_*" | wc -l

# Clean up
find /tmp -type f -mtime +7 -delete
find /var/spool/postfix/maildrop -type f -delete

Prevent Recurrence

# Configure journal size limit
sudo cat > /etc/systemd/journald.conf.d/size.conf  /etc/logrotate.d/myapp