How to Recover from a Failed Kernel Update

The Problem

A failed kernel update can leave your Breeze unable to boot. This can happen due to power loss during the update, a bug in the new kernel, incompatible drivers, or a full /boot partition. Knowing how to recover is critical for server administration.

Symptoms

• Server does not come back after reboot
• Kernel panic during boot
• Blank screen or GRUB rescue prompt
• Boot hangs at a specific point

Method 1: Boot into the Previous Kernel via GRUB

If you have console access (VNC or IPMI/iDRAC):

# At the GRUB menu (hold Shift during boot if hidden):
# Select "Advanced options for Ubuntu"
# Choose the previous working kernel version
# The system should boot normally

Method 2: Fix from Recovery Mode

# Boot into recovery mode from GRUB
# Select "Advanced options" → kernel version with "(recovery mode)"
# Choose "root - Drop to root shell prompt"

# Remount filesystem as read-write
mount -o remount,rw /

# Check /boot space
df -h /boot

# Remove failed kernel packages
apt remove --purge linux-image-X.X.X-XX-generic
apt remove --purge linux-headers-X.X.X-XX-generic

# Reinstall the working kernel
apt install --reinstall linux-image-$(uname -r)

# Update GRUB
update-grub

# Reboot
reboot

Method 3: Rescue from Live ISO

# Boot from a live USB/ISO
# Mount the root filesystem
sudo mount /dev/sda1 /mnt
# If using LVM:
sudo vgchange -ay
sudo mount /dev/mapper/vg-root /mnt

# Mount required virtual filesystems
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys
sudo mount /dev/sda2 /mnt/boot    # if separate /boot

# Chroot into the system
sudo chroot /mnt

# Now fix the kernel
apt update
apt install --reinstall linux-image-generic
update-grub

# Exit and reboot
exit
sudo umount -R /mnt
sudo reboot

Fixing a Full /boot Partition

# Check /boot usage
df -h /boot

# List installed kernels
dpkg --list | grep linux-image

# Remove old kernels (keep current and one previous)
sudo apt autoremove --purge

# If autoremove is blocked, manually remove
sudo dpkg --purge linux-image-5.4.0-100-generic

# Clean up
sudo apt -f install
sudo update-grub

Setting the Default Boot Kernel

# List available kernels in GRUB
grep -E "menuentry " /boot/grub/grub.cfg | head -10

# Set default to a specific entry
sudo grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-150-generic"
sudo update-grub

# Verify
grub-editenv list

Prevention

• Monitor /boot space — keep at least 200 MB free
• Run apt autoremove regularly to clean old kernels
• Test kernel updates on staging before production
• Keep console access (VNC/IPMI) available for emergencies
• Consider pinning a known-good kernel version on critical production servers