Kernel

Abstract

This is a very very basic attempt to describe how the kernel is implemeted on redhat/fedora/centos for those people whom are in environments where someone tried to be "helpful" by removing the native distros kernel and replace it with their own. This is NOT an article for VPS users, because you dont have a kernel and theres nothing you can do. As a side note, this is not a replacement for the real documentation in the linux kernel, they are just my personal observations and are subject to error. This is not linux canon. You have been warned.

Overview

The kernel installation/upgrade process is for the most part completely automated. Upgrade events are capable of reading the existing configuration information, and adding them to the boot loader.

Major components

Kernel, usually /boot/vmlinuz-XXXX

Ram Disk, usually /boot/initrd-XXXX

Modules, usually /lib/modules/XXXX

Boot loader, usually Grub (/etc/grub.conf which is a symlink to /boot/grub/menu.lst)

Module configuration information, /etc/modprobe.conf, or /etc/modprobe.d/*

Minor componets

New-kernel-pkg, /sbin/new-kernel-pkg

Kenel configuration file (used by new-kernel-pkg), /etc/sysconfig/kernel

mkinitrd config directory (depends on distro), /etc/sysconfig/mkinitrd/

Boot Process

1) Grub is configured to load a specific kernel

2) Kernel boots, this loads the initrd ramdisk

3) initrd contains the modules needed to read core hardware, like the disks. This is generally where mayhem happens.

4) System mounts disks and goes multi-user, remaining modules like the ethernet controller, lm_sensors, firewall modules, are loaded dynamically

Initial Installation Process

This is all pretty amazing and I'm completely underselling it. Basically the Red Hat OS installer does some really impressive work, its called anaconda and you can read more about it here. In brief, the system boots off of an installer kernel, and anaconda interrogates the system to populate the modprobe configuration file mentioned above. Its the only thing I know of that can actually figure out what the hardware is on a box without human intervention.

Modprobe is what loads the kernel drivers for everything on the box, like the disks or the network card, and the modprobe.conf/modprobe.d part is only configured through anaconda. If you dont go through anaconda, then you dont get these configs for your hardware, and if you dont have these configs a normal upgrade wont work.

Let me reiterate this, if you're in an environment where someone thought they were doing you a favor by not putting their own kernel on a system from a disk image (1&1, and ovh, I mean YOU), well they arent. You're about to take the first step down the long road of becoming a kernel/hardware expert. There is no automated way, short of re-imaging the box (and thereby running anaconda again) to resolve this.

Upgrade Process

Assuming you've got a healthy, normal system, its largely automated. At most I generally only interact with the system at the grub level, using tricks like --once to boot kernels in test mode. That aside, heres what happens when you upgrade the kernel with yum.

1) Yum installs an additional kernel on the system. It may be configured to remove older kernels, but in nearly every environment it is just adding kernels to the system.

2) Inside the kernel rpm there is a post processing macro called %post, this is a shell script that actually does all the magic. Heres what it looks like from Fedora 10's 2.6.27.x kernels:

 %{expand:%%post %{?-v*}}\
 %{-s:\
 if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] &&\
    [ -f /etc/sysconfig/kernel ]; then\
   /bin/sed -i -e 's/^DEFAULTKERNEL=%{-s*}$/DEFAULTKERNEL=%{-r*}/' /etc/sysconfig/kernel || exit $?\
 fi}\
 /sbin/new-kernel-pkg --package kernel%{?-v:-%{-v*}} --mkinitrd --kernel-args="selinux=0 acpi=off" --depmod --install %{?1} %{KVERREL}%{?-v*} || exit $?\
 %{nil}

3) Breaking this down, the first step is to update /etc/sysconfig/kernel. The sed line above is passing in information on which base kernel is being used. This is not the version, rather its saying that the default kernel for this system is the kernel-PAE or kernel- package. /etc/sysconfig/kernel itself is one of the config files used by the next step.

4) new-kernel-pkg is run. This is what creates the initrd ramdisk, and add the kernel to grub. Getting back to the whole modprobe.conf issue, new-kernel-pkg reads /etc/modprobe.conf or /etc/modprobe.d for the information it needs to create the /boot/initrd ramdisk. If this information is wrong, or missing, the kernel will not boot. What you'll get is a kernel panic.

5) Once the initrd is created, new-kernel-pkg adds the entries to the bootloader (/etc/grub.conf or /boot/grub/menu.lst), again using /etc/sysconfig/kernel as its config file. You'll see a setting in there UPDATEDEFAULT=yes. If this is set to yes, then whenever new-kernel-pkg is run, the system will automatically configure that kernel as the default.

6) Sometimes during this process you will see warnings that certain modules dont exist. This gets back to modprobe.conf again, as the kernel evolves sometimes the module names will change, or even go away. The entries still exist in modpobe.conf, and so new-kernel-pkg will still try to add them. Its generally a bad sign, and something you *should* worry about, especially if its referencing something important like a network or disk controller. Because of this its a good idea to always boot your kernels in testing mode with the --once flag.

Troubleshooting

Kernel Panic

Setting which kernel to boot

If your system is using grub:

Modify this file as root:

/etc/grub.conf

A typical grub.conf file will look similar to this:

# Note that you do not have to rerun grub after making changes to this file

1. NOTICE: You have a /boot partition. This means that
2. all kernel and initrd paths are relative to /boot/, eg.
3. root (hd0,0)
4. kernel /vmlinuz-version ro root=/dev/md1
5. initrd /initrd-version.img
6. boot=/dev/md0

default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.27.7-9.art.i686) root (hd0,0) kernel /vmlinuz-2.6.27.7-9.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000 initrd /initrd-2.6.27.7-9.art.i686.img title CentOS (2.6.26.6-1.art.i686) root (hd0,0) kernel /vmlinuz-2.6.26.6-1.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000 initrd /initrd-2.6.26.6-1.art.i686.img title CentOS (2.6.25.4-4.art.i686) root (hd0,0) kernel /vmlinuz-2.6.25.4-4.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000 initrd /initrd-2.6.25.4-4.art.i686.img

The system will boot into the kernel based on the configuration of this line in /etc/grub/conf:

default=0

Where "0" is the number of the kernel to boot into. In grub the first position is "0", the second position is "1" and so on. So the kernel the system will boot into, in the example above, is:

title CentOS (2.6.27.7-9.art.i686) root (hd0,0) kernel /vmlinuz-2.6.27.7-9.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000 initrd /initrd-2.6.27.7-9.art.i686.img

If you want the system to boot into a different kernel, for example if we wanted to boot into the second kernel (in position 1, remember the first position is 0, not 1):

title CentOS (2.6.26.6-1.art.i686) root (hd0,0) kernel /vmlinuz-2.6.26.6-1.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000 initrd /initrd-2.6.26.6-1.art.i686.img

The this line:

default=0

Must be changed to:

default=1