Difference between revisions of "Kernel"

Revision as of 14:23, 15 May 2012

Abstract

This is a very very basic attempt to describe how the kernel is implemeted on redhat/fedora/centos for those people whom are in environments where someone tried to be "helpful" by removing the native distros kernel and replace it with their own. This is NOT an article for VPS users, because you dont have a kernel and theres nothing you can do. As a side note, this is not a replacement for the real documentation in the linux kernel, they are just my personal observations and are subject to error. This is not linux canon. You have been warned.

Overview

The kernel installation/upgrade process is for the most part completely automated. Upgrade events are capable of reading the existing configuration information, and adding them to the boot loader.

Major components

Kernel, usually /boot/vmlinuz-XXXX

Ram Disk, usually /boot/initrd-XXXX

Modules, usually /lib/modules/XXXX

Boot loader, usually Grub (/etc/grub.conf which is a symlink to /boot/grub/menu.lst)

Module configuration information, /etc/modprobe.conf, or /etc/modprobe.d/*

Minor componets

New-kernel-pkg, /sbin/new-kernel-pkg

Kenel configuration file (used by new-kernel-pkg), /etc/sysconfig/kernel

mkinitrd config directory (depends on distro), /etc/sysconfig/mkinitrd/

Boot Process

1) Grub is configured to load a specific kernel

2) Kernel boots, this loads the initrd ramdisk

3) initrd contains the modules needed to read core hardware, like the disks. This is generally where mayhem happens.

4) System mounts disks and goes multi-user, remaining modules like the ethernet controller, lm_sensors, firewall modules, are loaded dynamically

Initial Installation Process

This is all pretty amazing and I'm completely underselling it. Basically the Red Hat OS installer does some really impressive work, its called anaconda and you can read more about it here. In brief, the system boots off of an installer kernel, and anaconda interrogates the system to populate the modprobe configuration file mentioned above. Its the only thing I know of that can actually figure out what the hardware is on a box without human intervention.

Modprobe is what loads the kernel drivers for everything on the box, like the disks or the network card, and the modprobe.conf/modprobe.d part is only configured through anaconda. If you dont go through anaconda, then you dont get these configs for your hardware, and if you dont have these configs a normal upgrade wont work.

Let me reiterate this, if you're in an environment where someone thought they were doing you a favor by not putting their own kernel on a system from a disk image (1&1, and ovh, I mean YOU), well they arent. You're about to take the first step down the long road of becoming a kernel/hardware expert. There is no automated way, short of re-imaging the box (and thereby running anaconda again) to resolve this.

Upgrade Process

Assuming you've got a healthy, normal system, its largely automated. At most ASL only interacts with the system at the grub level, using tricks like --once to boot kernels in test mode. That aside, heres what happens when you upgrade the kernel with yum.

1) Yum installs an additional kernel on the system. It may be configured to remove older kernels, but in nearly every environment it is just adding kernels to the system.

2) Inside the kernel rpm there is a post processing macro called %post, this is a shell script that actually does all the magic. Heres what it looks like from Fedora 10's 2.6.27.x kernels:

 %{expand:%%post %{?-v*}}\
 %{-s:\
 if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] &&\
    [ -f /etc/sysconfig/kernel ]; then\
   /bin/sed -i -e 's/^DEFAULTKERNEL=%{-s*}$/DEFAULTKERNEL=%{-r*}/' /etc/sysconfig/kernel || exit $?\
 fi}\
 /sbin/new-kernel-pkg --package kernel%{?-v:-%{-v*}} --mkinitrd --kernel-args="selinux=0 acpi=off" --depmod --install %{?1} %{KVERREL}%{?-v*} || exit $?\
 %{nil}

3) Breaking this down, the first step is to update /etc/sysconfig/kernel. The sed line above is passing in information on which base kernel is being used. This is not the version, rather its saying that the default kernel for this system is the kernel-PAE or kernel- package. /etc/sysconfig/kernel itself is one of the config files used by the next step.

4) new-kernel-pkg is run. This is what creates the initrd ramdisk, and add the kernel to grub. Getting back to the whole modprobe.conf issue, new-kernel-pkg reads /etc/modprobe.conf or /etc/modprobe.d for the information it needs to create the /boot/initrd ramdisk. If this information is wrong, or missing, the kernel will not boot. What you'll get is a kernel panic.

5) Once the initrd is created, new-kernel-pkg adds the entries to the bootloader (/etc/grub.conf or /boot/grub/menu.lst), again using /etc/sysconfig/kernel as its config file. You'll see a setting in there UPDATEDEFAULT=yes. If this is set to yes, then whenever new-kernel-pkg is run, the system will automatically configure that kernel as the default.

6) Sometimes during this process you will see warnings that certain modules dont exist. This gets back to modprobe.conf again, as the kernel evolves sometimes the module names will change, or even go away. The entries still exist in modpobe.conf, and so new-kernel-pkg will still try to add them. Its generally a bad sign, and something you *should* worry about, especially if its referencing something important like a network or disk controller. Because of this its a good idea to always boot your kernels in testing mode with the --once flag.

Troubleshooting

Kernel Panic

Checking to see if the ASL kernel is installed

Check your rpm repository to see if the kernel is installed:

rpm -qa | grep kernel

If you have the ASL kernel installed, you will see a series of kernel names with "art" in them, similar to this:

kernel.x86_64 1:2.6.25.4-4.art installed
kernel.x86_64 1:2.6.26.6-1.art installed
kernel.x86_64 1:2.6.27.7-9.art installed
kernel.x86_64 1:2.6.29.6-1.art installed
kernel.x86_64 1:2.6.32.8-1.art installed
kernel-devel.x86_64 1:2.6.25.4-4.art installed
kernel-devel.x86_64 1:2.6.26.6-1.art installed
kernel-devel.x86_64 1:2.6.27.7-9.art installed
kernel-devel.x86_64 1:2.6.29.6-1.art installed
kernel-devel.x86_64 1:2.6.32.8-1.art installed
kernel-headers.x86_64 1:2.6.32.8-1.art installed

If you do not see this, then you need to install the ASL kernel. Simply run this command as root:

yum install kernel

If you are running a PAE system, run this command as root:

yum install kernel-PAE

Then check to see if you have the ASL kernel installed by running the "rpm -qa | grep kernel" command again.

How to tell if you are running the ASL kernel

You can tell if you are running an ASL kernel with this command:

uname -a

You should see a kernel name with "art" in the title, for example:

Linux www3.atomicorp.com 2.6.32.21-3.art.x86_64 #1 SMP Tue Sep 7 16:57:34 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

If you do not see a kernel name with "art" in the title then you are not running the ASL kernel. If you see "art" in the title, then you are running the ASL kernel.

What to do if the kernel is not installed or won't upgrade

If the kernel is not installed after running the ASL installer or the kernel will not upgrade, that can mean two things:

1) If your system is a VPS, such as when using the Virtuozzo and Openvz products, you can not install a kernel. VPS systems do not have their own kernel and use the single kernel shared by the system. Please contact your hosting provider or sysadmin to install the ASL kernel on the root system.

2) If this is not a VPS, the ASL installer will not install the kernel if your system is configured to not allow this. No distribution does this by default. If you system is configured this way, it usually means yum is configured on your system to not allow kernels to be installed. Please check your yum configuration to make sure it is not configured to exclude kernels. Some places to check are your /etc/yum.conf file for a line similar to this:

exclude=kernel*

If you see any lines like that, remove them. Also check in your /etc/yum.repos.d directory to make sure someone did not set this in a repository.

ASL honors your systems configuration, so if you configure you system to prevent kernels from being installed via yum, ASL will not install its kernel. If you do not have any exclude lines configured for yum, please contact the parties that configured/setup your system and ask them to fix your system to allow kernels to be installed.

You can also try manually installing the kernel, but keep in mind that if you have to do this something else is wrong with your system. Please run this command as root to see if yum is listing the ASL kernels as being available to install on your system:

yum list | grep kernel

(This is not a listing of the kernels installed on your system, just the kernels available via yum that may be installed on your system)

You should see kernel names similar to this (note the "art" in the rpm name, your architecture may be different, but an ASL kernel will have "art" in the name):

kernel.x86_64 1:2.6.25.4-4.art installed
kernel.x86_64 1:2.6.26.6-1.art installed
kernel.x86_64 1:2.6.27.7-9.art installed
kernel.x86_64 1:2.6.29.6-1.art installed
kernel.x86_64 1:2.6.32.8-1.art installed
kernel-devel.x86_64 1:2.6.25.4-4.art installed
kernel-devel.x86_64 1:2.6.26.6-1.art installed
kernel-devel.x86_64 1:2.6.27.7-9.art installed
kernel-devel.x86_64 1:2.6.29.6-1.art installed
kernel-devel.x86_64 1:2.6.32.8-1.art installed
kernel-headers.x86_64 1:2.6.32.8-1.art installed

If you see art kernels listed, yum upgrade should work, in that case you would run this command as root:

yum upgrade kernel

Or if you have a 32 bit system, and need a PAE kernel, run this command as root:

yum upgrade kernel-PAE

Installing the kernel

If you have not run the ASL installed

If the kernel is not installed, and you have not run the ASL installer, run the ASL installer.

If you have run the ASL installer

If you have run the ASL installer, and the kernel is not installed follow this process:

1) If you are using a VPS technology, such as virtuzzo or openvz, and you are installing ASL inside a VPS you will not be able to install any kernel (not just the ASL kernel). VPS' do not have their own kernel, they share the hosts single kernel. Install ASL on the host. (VPS technologies are not to be confused with technologies like VMWare, KVM, qemu, ESXi, Parallels, and other virtualization technologies. Those technologies do allow you to install your own kernel, and you can install the ASL kernel or any other kernel inside those virtual machines)

2) Check to make sure you do not have the kernel excluded from your yum configuration. Please read this entire article as it covers how to determine if your system is setup this way.

3a) If your system is 64 bit, run this command to install the kernel:

yum --disableexcludes=all upgrade kernel

3b) If your system is 32 bit, run this command to install the kernel:

yum --disableexcludes=all upgrade kernel-PAE

If you do not already have a PAE kernel installed, then you will need to use this command:

yum --disableexcludes=all install kernel-PAE

If this does not work, please contact your system administrator or hosting company. This means that your system has been configured to not allow you to install kernels, or something is seriously wrong with the software management in your OS.

Upgrading the kernel

Step 1) Just run this command as root:

yum upgrade kernel

Or if you have a 32 bit system, then you will want to install the PAE kernel. Use this command:

yum upgrade kernel-PAE

Note: As in the examples above, use yum upgrade, not yum update to upgrade a kernel.

If your system reports that there is no kernel to upgrade to, it is possible you already have the latest kernel installed and you simply need to reboot into the kernel. Please see the #Checking_to_see_if_the_ASL_kernel_is_installed section for information about how to check what kernels you have installed.

If your system reports that there is no kernel to upgrade to, and the ASL kernel is not installed, it is possible your system is configured to not install kernels. Check your yum configuration for any "exclude" lines. These tell your system to not install certain software.

Step 2) Reboot the system into the new kernel.

Run this command as root:

reboot

Setting which kernel to boot

Linux uses a boot loader to select which kernel to boot into (this is not part of ASL). In most cases your system will use a boot loader called "grub". If your system is using lilo, the older (really really old) boot loader we recommend you use grub.

ASL will not replace your existing kernel. ASL will install an additional secure kernel you can boot into, and will set that new secure kernel to boot by default. Should you need to use your older kernel just select your old kernel to boot into.

If your system is using grub:

Modify this file as root:

/etc/grub.conf

A typical grub.conf file will look similar to this:

# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/md1
#          initrd /initrd-version.img
#boot=/dev/md0
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.27.7-9.art.i686)
	root (hd0,0)
	kernel /vmlinuz-2.6.27.7-9.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
	initrd /initrd-2.6.27.7-9.art.i686.img
title CentOS (2.6.26.6-1.art.i686)
	root (hd0,0)
	kernel /vmlinuz-2.6.26.6-1.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
	initrd /initrd-2.6.26.6-1.art.i686.img
title CentOS (2.6.25.4-4.art.i686)
	root (hd0,0)
	kernel /vmlinuz-2.6.25.4-4.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
	initrd /initrd-2.6.25.4-4.art.i686.img

The system will boot into the kernel based on the configuration of this line in /etc/grub/conf:

default=0

Where "0" is the number of the kernel to boot into. In grub the first position is "0", the second position is "1" and so on. So the kernel the system will boot into, in the example above, is:

title CentOS (2.6.27.7-9.art.i686)
	root (hd0,0)
	kernel /vmlinuz-2.6.27.7-9.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
	initrd /initrd-2.6.27.7-9.art.i686.img

If you want the system to boot into a different kernel, for example if we wanted to boot into the second kernel (in position 1, remember the first position is 0, not 1):

title CentOS (2.6.26.6-1.art.i686)
	root (hd0,0)
	kernel /vmlinuz-2.6.26.6-1.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
	initrd /initrd-2.6.26.6-1.art.i686.img

The this line:

default=0

Must be changed to:

default=1

If you want do not want yum to install the secure ASL kernel, and would prefer to stick with your distributions stock kernel, put 'exclude=kernel*' in the [asl-2.0] section in /etc/yum.repos.d/asl.repo. ASL will actively test and report vulnerabilities in all kernels, so if your stock kernel is reported to have vulnerabilities these are not false positives. We do not recommend you use any kernel other than the ASL kernel.

Schedulers

The ASL kernel includes four I/O schedulers to suit various system needs and configurations.

Scheduler Types

Completely Fair Queuing (CFQ)

This is the default algorithm. As the name implies, CFQ maintains a scalable per-process I/O queue and attempts to distribute the available I/O bandwidth equally among all I/O requests. CFQ is well suited for mid-to-large multi-processor systems and for systems which require balanced I/O performance over multiple LUNs and I/O controllers.

Deadline

The Deadline elevator uses a deadline algorithm to minimize I/O latency for a given I/O request. The scheduler provides near real-time behavior and uses a round robin policy to attempt to be fair among multiple I/O requests and to avoid process starvation. Using five I/O queues, this scheduler will aggressively re-order requests to improve I/O performance.

NOOP

This scheduler is a simple FIFO queue and uses the minimal amount of CPU/instructions per I/O to accomplish the basic merging and sorting functionality to complete the I/O. It assumes performance of the I/O has been or will be optimized at the block device (memory-disk) or with an intelligent HBA or externally attached controller.

Anticipatory

The Anticipatory elevator introduces a controlled delay before dispatching the I/O to attempt to aggregate and/or re-order requests improving locality and reducing disk seek operations. This algorithm is intended to optimize systems with small or slow disk subsystems. One artifact of using the AS scheduler can be higher I/O latency.

Changing the scheduler

If the default scheduler does not meet your needs, you can change the scheduler by logging in as root and changing the scheduler for the device in this manner:

echo {SCHEDULER-NAME} > /sys/block/{DEVICE-NAME}/queue/scheduler

Example:

echo noop > /sys/block/hda/queue/scheduler

You can also view the scheduler you are using this this command:

cat /sys/block/{DEVICE-NAME}/queue/scheduler