CentOS booting from old kernel even after update?

Have you run "yum -y update", to make sure the latest kernel is actually installed?

I appreciate the suggestion. Yes, I have run "yum update" and there are two newer kernels. I don't experience this issue with physical servers. I also don't experience this issue with servers I host with my own in-house KVM farm. Trying to figure out what is different in this Linode environment that keeps me from booting CentOS v7's latest kernel?

well, here are the steps I follow in my Linode servers and they all work fine and boot the vanilla centos kernel, take a look:

(make sure to install kernel and grub2)

yum install kernel.x86_64 grub2

(edit /etc/default/grub)

GRUB_TIMEOUT=5

GRUBCMDLINELINUX="console=ttyS0,19200n8"

GRUBDISABLELINUX_UUID=true

GRUBSERIALCOMMAND="serial –speed=19200 --unit=0 --word=8 --parity=no --stop=1"

(in case its not already made)

mkdir /boot/grub

(prep grub, must be run after every kernel update)

grub2-mkconfig -o /boot/grub/grub.cfg

thats it, now I can set GRUB2 (via the linode profile) as the boot option and enable SELinux.

This is about a three month old Linode that by default is KVM. By default it uses the CentOS kernel and GRUB2. It seems since day one, it is always using the same old kernel to boot from. I wanted to change it to use current updated CentOS v7 kernels and did the following.

In /etc/default/grub there was;

GRUB_DEFAULT=saved

I changed it to;

GRUB_DEFAULT=0

Thinking this would boot the latest kernel.

I then ran;

grub2-mkconfig –output=/boot/grub2/grub.cfg

Then rebooted.

Unfortunately, it still boots from the same CentOS v7 kernel which is now two kernel updates behind. It does not seem to want to boot from the newer CentOS v7 kernels that are installed. I was going to create a support ticket, but was recommended to try the forum first so I I brought it here. Any other suggestions. Is there something in the Linode profile I should be looking at?

[root@linode404 grub2]# rpm -qa | grep kernel

kernel-3.10.0-514.6.1.el7.x86_64

kernel-3.10.0-514.2.2.el7.x86_64

kernel-tools-libs-3.10.0-514.6.1.el7.x86_64

kernel-tools-3.10.0-514.6.1.el7.x86_64

kernel-3.10.0-327.36.3.el7.x86_64

[root@linode404 grub2]# rpm -qa | grep grub2

grub2-tools-2.02-0.44.el7.centos.x86_64

grub2-2.02-0.44.el7.centos.x86_64

[root@linode404 grub2]# yum install kernel.x86_64

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

• base: mirrors.linode.com

• extras: mirrors.linode.com

• updates: mirrors.linode.com

Package kernel-3.10.0-514.6.1.el7.x86_64 already installed and latest version

Nothing to do

[root@linode404 grub2]# uname -a

Linux linode404 3.10.0-327.36.3.el7.x8664 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux

Simply:

yum -y remove kernel-3.10.0-514.2.2.el7.x86_64

delete the contents of your /etc/default/grub and just put the 4 lines I'm using above

re-run the grub2-mkconfig command and boot (notice that my parameters are different to yours. but grub/grub.cfg should be a symlink to grub2/grub.cfg)

once you boot, make sure to

yum -y remove kernel-3.10.0-327.36.3.el7.x86_64

I don't want my dog to explode…. It's a production server at the moment.

Do you see anything wrong with the extra stuff in the Linode supplied /etc/default/grub that would warrant such a move?

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

GRUB_DEFAULT=0

GRUBDISABLESUBMENU=true

GRUB_TERMINAL="serial"

GRUBCMDLINELINUX="crashkernel=auto rhgb console=ttyS0,19200n8"

GRUBDISABLERECOVERY="true"

GRUBDISABLEOS_PROBER="true"

GRUBSERIALCOMMAND="serial --speed=19200 --unit=0 --word=8 --parity=no --stop=1"

GRUBDISABLELINUX_UUID="true"

GRUBGFXPAYLOADLINUX="text"

I think the problem is with GRUB_DEFAULT, which tries to boot kernel "0", which probably is always the old kernel. If you remove that line completely, it will boot the last kernel installed.

Also, look at the output of the grub2-mkconfig line, it should show you the order of the kernels as they've been installed.

It originally had;

GRUB_DEFAULT=saved

I chose 0 based on the output of;

[root@linode404 ~]# grep '^menuentry' /boot/grub2/grub.cfg

menuentry 'CentOS Linux (3.10.0-514.6.1.el7.x8664) 7 (Core)' –class centos --class gnu-linux --class gnu --class os --unrestricted $menuentryidoption 'gnulinux-3.10.0-514.6.1.el7.x8664-advanced-f409ba6a-b715-4444-a33c-d6167e51695f' {

menuentry 'CentOS Linux (3.10.0-514.2.2.el7.x8664) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentryidoption 'gnulinux-3.10.0-514.2.2.el7.x8664-advanced-f409ba6a-b715-4444-a33c-d6167e51695f' {

menuentry 'CentOS Linux (3.10.0-327.36.3.el7.x8664) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentryidoption 'gnulinux-3.10.0-327.36.3.el7.x8664-advanced-f409ba6a-b715-4444-a33c-d6167e51695f' {

menuentry 'CentOS Linux (0-rescue-7cd9c0d80e994cf7a81ea6da998990d7) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentryidoption 'gnulinux-0-rescue-7cd9c0d80e994cf7a81ea6da998990d7-advanced-f409ba6a-b715-4444-a33c-d6167e51695f' {

Which matches the output of the grub2-mkconfig line;

Generating grub configuration file …

Found linux image: /boot/vmlinuz-3.10.0-514.6.1.el7.x86_64

Found initrd image: /boot/initramfs-3.10.0-514.6.1.el7.x86_64.img

Found linux image: /boot/vmlinuz-3.10.0-514.2.2.el7.x86_64

Found initrd image: /boot/initramfs-3.10.0-514.2.2.el7.x86_64.img

Found linux image: /boot/vmlinuz-3.10.0-327.36.3.el7.x86_64

Found initrd image: /boot/initramfs-3.10.0-327.36.3.el7.x86_64.img

Found linux image: /boot/vmlinuz-0-rescue-7cd9c0d80e994cf7a81ea6da998990d7

Found initrd image: /boot/initramfs-0-rescue-7cd9c0d80e994cf7a81ea6da998990d7.img

done

If you think that line is still the culprit, I might spin up another linode and test it there before testing it on my production linode.

Yes I think that is the problem, but I've been known to make mistakes in the past ;)

Testing in another linode is a good idea, I always do my tests first in a local VM (virtualbox) then in a test linode that I later delete.

I dug a little deeper.

It seems that if you install CentOS v7/RHEL v7 via an ISO image downloaded from CentOS or Red Hat, the default is to use /boot/grub2/grub.cfg.

For some reason, the CentOS v7 instance that Linode provides defaults to /boot/grub/grub.cfg. Notice "grub" not "grub2".

However, when yum installs an updated kernel, they update /boot/grub2/grub.cfg.

Any idea why the difference?

I believe older centos images used /boot/grub but was later changed to /boot/grub2

but, it should make no difference, because on newer images, the file /boot/grub/grub.cfg should be a symbolic link to /boot/grub2/grub.cfg

@IfThenElse:

I believe older centos images used /boot/grub but was later changed to /boot/grub2

but, it should make no difference, because on newer images, the file /boot/grub/grub.cfg should be a symbolic link to /boot/grub2/grub.cfg

I checked that on over 10 CentOS 7 systems I have here, and none of them have that symbolic link. The only thing they have in /boot/grub is;

-rw-r–r-- 1 root root 1350 Nov 15 2011 splash.xpm.gz

I'll spin up a new CentOS v7 Linode and if it has the same problem, I will open up a support ticket.

Okay, I figured it out.

I have had a problem in the past with Linode's host kernels so I like to stay with CentOS supplied kernels. This requires prepping the Linode to use GRUB2. Linode's documentation updated Tuesday, June 7, 2016 has a typo in it;

https://www.linode.com/docs/tools-refer … l-with-kvm">https://www.linode.com/docs/tools-reference/custom-kernels-distros/run-a-distribution-supplied-kernel-with-kvm

At this moment, when you build a CentOS v7 64 bit Linode, the kernel and grub files needed are already installed. The /boot/grub and /boot/grub2 directories are already there. Out of the box, /boot/grub/grub.cfg is a symbolic link to /boot/grub2/grub.cfg. If you run their command in step 2 for CentOS 7 "grub2-mkconfig -o /boot/grub/grub.cfg", you will break the symbolic link. Not sure why the symbolic link is needed for a Linode , it is not needed for physical servers or KVM guest VMs that I have built.

The correct command should be;

grub2-mkconfig -o /boot/grub2/grub.cfg (Notice "/grub2/" not "/grub/")

When CentOS v7 updates to a new kernel, it places the new entry in /boot/grub2/grub.cfg. However, during boot up, it looks for /boot/grub/grub.cfg (which was no longer a symbolic link to /boot/grub2/grub.cfg) and was never seeing the newer CentOS v7 kernels.

I spun up a new CentOS v7 Linode, ran the correct command, installed OS updates and now each time there is a new kernel, it will automatically boot from it on reboot.

Interesting, let me talk to some of the docs guys about this. Sorry you've had to go through this, but maybe we can make sure no one else has to as well.

I think the symbolic link to /boot/grub/grub.cfg is needed because Linode's version of Grub2 searches there by default. Red Hat's searches /boot/grub2. It sounds like Red Hat and Linode are using different versions of grub2 to boot the system. It's OK, if it works, but you are spot on, I just ran into this problem with RHEL 7 because I believe something in the copy disk over images process [1] clobbered my /boot/grub/grub.cfg symbolic link. I just checked and it was an old version that looks like it was created when I first converted the image from RHEL 7.2. I changed back to a symbolic link, and now things are back to working perfectly.

Probably the best thing for Linode would be to patch their version of Grub to search /boot/grub and /boot/grub2 and choose whichever one is newest - but I don't want to write that patch :-)

[1]: https://linode.com/docs/platform/disk-i … -over-ssh/">https://linode.com/docs/platform/disk-images/copying-a-disk-image-over-ssh/