Ubuntu krenel panic

One of my hobbies now is selfhosting, I have many VPS instances that I use for different purposes. They are all ubuntu instances that is based on ubuntu LTS 20.04. Two months ago I did an ordinary system update which contained krenel upgrade. Suddenly I got later some errors with my services tracker that they are offline. I tried a lot of debugging but then rebooted the system. Nothing happened and worse that this, I did not manage to access to the instance via ssh connection. I used VNC access provided by my cloud provider and found that the system is in panic mode. I spent a couple of hours trying to fix that thought first that the problem with just with the missing initramfs for the updated krenel. But to my surprise I found that the last update filled my /boot parition so I thought that I want now to move everything to the root system including /boot. Fournately this worked and I was able to boot the system again. I will describe the steps that I did to fix the problem.

We need first to to install the krenel and modify grub, we also need a recovery system to be able to boot the system. Many providers provide a recovery system or if you are are doing this on a home machine/server you can use a live USB stick. I will use the recovery system provided by my cloud provider but it doesn’t matter from now on I assume that you are already in the recovery system.

List the paritions

You will need to know what paritions do you have and you can run fdisk -l to see the paritions. you will find different paritions

Mount the root parition

The next step is to mount root parition to /mnt which in my case was mount /dev/sda3 /mnt but that might be different for you.

Chroot Setup

Now we need to invoke a chroot environment by running the following command

for i in dev proc sys run; do mount -o bind /$i /mnt/$i; done

The idea of this code is that it sets a loop that will iterate over each of the directories listed: /dev, /proc, /sys, and /run. The mount -o bind /$i /mnt/$i; part mounts the directory listed in $i to a corresponding directory under /mnt. The -o bind option tells the mount command to create a bind mount, which means that the directory is mounted in such a way that any changes made to it will also be reflected in the original directory. So, for example, if a file is created in /dev, it will also appear in /mnt/dev.

Chroot to the root parition

Now lets enter the chroot command on the mounted /mnt file system

chroot /mnt

Update grub

First we need to update the ubuntu package list

apt-get update

Then we need to install linux krenel. For example, installing v5.4.0-137

apt-get --reinstall install linux-image-5.4.0-137-generic

Now we are ready to install grub

Install/Repair grub

We can run the following to install vmliuz and initrd necessary files to install grub

Let’s update grup bootloader configuration files based on the current state of the system

sudo update-grub && sudo update-grub2

Next we need to update the initramfs image used by the system during boot. The initramfs is a temporary filesystem loaded into memory at boot time that contains essential system files and drivers needed to mount the root filesystem. The initramfs image is generated from the files in the /etc/initramfs-tools directory. We can update the initramfs image by running the following command:

sudo update-initramfs -u -vvvvv

The -vvvvv option provides verbose output so you can see what’s happening during the update process.

And now we need to install grub

grub-install

This command installs the grup bootloader to the designated disk. The default disk is /dev/sda, but you can specify a different disk by using the --root-directory option. For example, if you wanted to install GRUB to /dev/sdb, you would run the following command:

grub-install --root-directory=/dev/sdb

Finally we just need to verify that the latest kernel has a corresponding initrd or initramfs image. You can do this by running the following command:

ls /boot/initrd.img-5.4.0-137-generic

Then lets close the chroot environment by running exit and then umount /mnt/{dev,proc,run,sys,} to unmount the directories we mounted earlier.

Reboot

Reboot the system and you should be able to boot the system again.

reboot