Common kernel problems
From FedoraProject
This page should be revised or reconstructed to better suit the needs of Fedora.
This page documents common problems with the Linux kernel in Fedora.
How to set kernel boot options
Kernel boot options are contained in the file /etc/grub.conf. Each installed kernel has a group of lines called a stanza describing what kernel to boot, where to find the root file system, the name of the initrd to load, and additional kernel options. A typical stanza looks something like this:
title Fedora Core (2.6.22.9-61.fc6) root (hd0,1) kernel /vmlinuz-2.6.22.9-61.fc6 ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.22.9-61.fc6.img
Kernel options are placed at the end of the "kernel" line and are separated by spaces. When having problems, it is usually a good idea to remove the "quiet" option so that the full set of kernel messages is shown during boot. Here is an example with the "quiet" and "rhgb" options removed and some other options added:
title Fedora Core (2.6.22.9-61.fc6) root (hd0,1) kernel /vmlinuz-2.6.22.9-61.fc6 ro root=LABEL=/ pci=nomsi,nommconf nohz=off initrd /initrd-2.6.22.9-61.fc6.img
NOTE: The full list of kernel options is in the file /usr/share/doc/kernel-doc-<version>/Documentation/kernel-parameters.txt, which is installed with the kernel-doc package.
How to set module options for boot drivers
Module options are set in the file /etc/modprobe.conf, or (with versions of module-init-tools in F10+) on the kernel command line. Drivers that are needed to boot the system are put into an initrd, and their options are copied from modprobe.conf by the mkinitrd script that builds the initrd. To change module options for those drivers, you can change the /etc/modprobe.conf file and rebuild the initrd, or alternatively (in recent releases of Fedora) you can simply append on the kernel command line.
For example, to disable adma mode on an nVidia SATA controller, add these options to the kernel command line (format is <modulename>.<option>=value):
sata_nv.adma=0
Alternatively, add this line to /etc/modprobe.conf:
options sata_nv adma=0
To get options set in /etc/modprobe.conf into the initrd, run the mkinitrd program. Usually this is just the command mkinitrd /boot/initrd-$(uname -r).new.img $(uname -r) to build a new initrd for the currently-running kernel without overwriting the exisitng one. (See man mkinitrd for help on additional options.) To test the new initrd, reboot the system and use the command line editing facilities to change the name of the initrd. Or, create a new stanza in the /etc/grub.conf file something like this (see above for the original):
title Fedora Core [with new initrd] (2.6.29-0.215.rc7.fc11.i586) root (hd0,1) kernel /vmlinuz-2.6.29-0.215.rc7.fc11.i586 ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.29-0.215.rc7.fc11.i586.new.img
This will let you boot with either the new or the old initrd by pressing the up arrow / down arrow keys on the very first boot screen. Once everything is tested, remove the original initrd and rename the new one to the same name as the old one, then remove the "[with new initrd] " stanza from /etc/grub.conf.
Can't find root filesystem / error mounting /dev/root
- A lot of these bugs end up being a broken initrd due to bugs in mkinitrd. Get the user to attach their initrd for their kernel to the bz, and also their /etc/modprobe.conf, or have them examine the contents themselves if they are capable of that.
- Picking apart the initrd of a working and failing kernel and doing a diff of the init script can reveal clues.
To take apart an initrd, do the following ..
mkdir initrd cd initrd/ gzip -dc /boot/initrd-2.6.23-0.104.rc3.fc8.img | cpio -id
Another way to examine the initrd is with Midnight Commander. Add the extension .cpio.gz to the filename and then just place the cursor over the name and press Enter.
- Unsupported mount options like "relatime" in /etc/fstab can cause problems. Removing any references to the "relatime" option and rebuilding the initrd will fix this.
Crashes/Hangs
- Checking whether or not the CapsLock key (or NumLock or ScrollLock) causes the light on the keyboard to change state can be used as an indication of whether or not the kernel has hung completely, or if there is something else going on.
- For boot related issues we need as much info as possible, so removing
quietrhgbfrom the boot flags should be the first thing to ask for. - Slowing down the speed of text output with
boot_delay=1000(the number may need to be tweaked higher/lower to suit) may allow the user to take a digital camera photo of the last thing on screen. - Booting with
vga=791(or even just vga=1 if the video card won't support 791) will put the framebuffer into high resolution mode to get more lines of text on screen, allowing more context for bug analysis. -
initcall_debugwill allow to see the last thing the kernel tried to initialise before it hung. - There are numerous switches that change which at times have proven to be useful to diagnose failures by disabling various features.
-
acpi=offis a big hammer, and if that works, narrowing down by tryingpci=noacpiinstead may yield clues -
nolapicandnoapicare sometimes useful -
nolapic_timercan be useful on i386; on x86_64 this option is callednoapictimer - Given it's new and still seeing quite a few changes,
nohz=offand/orhighres=offmay be worth testing. (Though this is kernel 2.6.21 and above only) - If you get no output at all from the kernel, sometimes booting with
earlyprintk=vgacan sometimes yield something of interest. - If the kernel locks up with a 'soft lockup' report, booting with
nosoftlockupwill disable this check allowing booting to continue. - If the kernel locks up really early, booting with
edd=skipmbroredd=offmay help - The system can hang because the clock isn't running properly, see System clock runs too fast/slow
- Sometimes the system can hang because it is looking for nonexistent floppy drives. See Boot pauses probing floppy device
- Sometimes multiple options are needed, e.g.
clocksource=acpi_pm nohz=off highres=off - Try to narrow down the options needed to the absolute minimum. This helps the kernel maintainers find the underlying problem.
- If it hangs after "Freeing unused kernel memory: 280k freed" you might have glibc.i686 when your processor is not capable of i686. Replace it to glibc.i386 and be sure the "i686" and "nosegneg" directories are deleted.
Boot pauses probing floppy device
On some machines (mostly laptops with removable floppy drives), boot will pause while the (non-existant) floppy device is probed. A series of the following messages will appear:
end_request: I/O error, dev fd0, sector 0 end_request: I/O error, dev fd0, sector 0 Buffer I/O error on device fd0, logical block 0
This is caused by initrd's nash searching for filesystem labels on the floppy device. This problem can be avoided by adding floppy.allowed_drive_mask=0 to the kernel boot options.
Can't find installation CD/DVD or hard drives
- Try
pci=nomsi,nommconf. This disables PCI Message Signaled Interrupts and MMCONFIG, which are only needed by a few systems at this time.
- Try booting with
libata.dma=1[use DMA only for hard drives] orlibata.dma=0[do not use DMA at all] . This can at least get the system installed, then the drivers can be updated.
Install runs very slowly
If the system runs very slowly, it may have a BIOS bug that causes part of the system memory to be uncached. Playing with the mem= parameter can work around this problem. Trying for example, mem=1000M will limit the system to 1000 megabytes of memory and may make the install run much faster.
Can't install
Sometimes, even booting with acpi=off or various other boot command line options, the kernel refuses to boot on some subsets of hardware. If none of the above tricks helps, then..
- In rawhide bugs, if the report is something that would prevent someone from installing the next release (crashes during boot, doesn't find hard disks etc), mark the bug as blocking 'F9Blocker' (bug 235706).
- if it's against the previously released version of Fedora, then it's possible that the problem was caused in a kernel bug that has since been fixed upstream. As Fedora constantly rebases to newer upstream kernels, they'll get picked up by the respins done by the folks at http://fedoraunity.org Suggest that the user tries an updated ISO if one is available.
Diagnosing "My machine locked up"
This can be a tricky one to diagnose. Most users don't have serial console capability, so we're mostly guessing in the dark.
- For possible workarounds for this problem, see Crashes/Hangs
- If it's repeatable, hooking up a serial cable to a second box can be useful for capturing kernel messages that may get printed just before the lockup. Configure the machine being debugged to boot with
console=ttyS0,115200 console=tty0and run a terminal program such asminicomon the other end. Configure the remote end to talk at the same baud rate (115200). (In minicomctrl-a, p, i, enter. More info on setting up a serial terminal can be found at http://searchenterpriselinux.techtarget.com/tip/0,289483,sid39_gci1118136,00.html - Sometimes just getting lsmod output from users can yield enough clues if there are multiple reports and common modules between both. (It also allows to filter out reports from users of nvidia,vmware etc).
- Hooking up serial console / can sometimes get debug info out of the machine.
- If the hang happened whilst in X, the machine may still respond to ssh logins from other machines. Try this to get a dmesg.
- The magic sysrq key might work. See QA/Sysrq for details.
- booting with
nmi_watchdog=2may cause a backtrace to occur when the lockup happens.
Suspend/Resume to RAM failure
The most common failure mode is 'black screen on resuming'.
- Laptops using the nv driver should be considered hibernate-only capable as per https://www.redhat.com/archives/fedora-test-list/2007-September/msg00365.html
- Find out if the system is locked up completely by hitting the caps lock key.
- If the capslock light doesn't toggle, the system is completely dead. Try again, but this time before suspending, activate the pm_trace functionality with
echo 1 > /sys/power/pm_trace. This reprograms the real time clock to contain a few bytes of information which we can use to diagnose which driver failed to resume. After the hang, reboot, boot up again. Now use the commanddmesg | grep "hash matches"and you will get a list of matches like this:hash matches device 0000:05:06.1. The last device on the list is likely the one thats causing problems. To find out which driver is causing the problem you will have to look up the driver in/sys/bus/pci/drivers/. This can be done usingfind /sys/bus/pci/drivers/ -name "0000:05:06.1". It will return a path similar to this one:/sys/bus/pci/drivers/firewire_ohci/0000:05:06.1which means that the firewire_ohci driver is causing troubles. Unloading the module usingmodprobe -r firewire_ohcishould fix the suspend issues. Please also note that pm_trace uses the RTC for storing the data, which will result into a wrong system clock after boot. To fix it just use system-config-date to set the correct date. - If the capslock light does toggle, then the system did come back up, and it's possible that we just failed to reinitialise the video. http://people.freedesktop.org/~hughsient/quirk may contain further useful information to diagnose this problem. It may also be useful to initiate the suspend from a tty (
ctrl-alt-f1) and runpm-suspend ; dmesg > dmesg.out ; syncby hand. Upon resuming you'll now have some more debug info to sift through. Additionally, this way when it resumes, you already have a console logged in from which you can type commands 'blind'. Tryingvbetool postfor example may bring things back to life. - Proprietary 3d graphics driver users should test with respective open source drivers.
- Try
rmmod'ing various modules before doing the suspend. If this makes things work again, retry with a smaller set of modules unloaded. Keep retrying until you narrow down which module is to blame. - Another trick that sometimes works to force video to come back up is to enable the BIOS password. This makes the system resume in a VGA text mode that the kernel recovers from a lot easier. Not a real solution, but it can help to diagnose other problems.
System clock runs too fast/slow
- Try a different clock source, e.g. :
clocksource=acpi_pm
- Clock sources can be changed at runtime by writing the new clocksource name to the file
/sys/devices/system/clocksource/clocksource0/current_clocksource, but be aware that changing to an unstable/broken clock source can hang the system. Changingtscorjiffiestoacpi_pmshould be okay. (The list of available sources is in the fileavailable_clocksourcein the same directory.)
- The kernel's tickless mode is enabled by default in Fedora 7 and 8, but can sometimes cause incorrect timekeeping. Using
nohz=off highres=offwill disable it.
Sound card doesn't work
"High Definition Audio" devices
Many times the model can't be detected properly. Adding the correct model to the sound card driver's entry in /etc/modprobe.conf will force the driver to use that model, e.g. options sound-card-0 model=3stack. Options for this driver are documented in the file /usr/share/doc/kernel-doc-<version>/Documentation/sound/alsa/ALSA-Configuration.txt in the kernel-doc package.
System hangs on reboot
Changing the reboot method can work around this problem. To force a reboot method other than the default, use the reboot= kernel option:
-
reboot=bforces reboot through the system BIOS.
-
reboot=wforces a 'warm" reboot (no memory test.)
These can be combined: reboot=b,w forces a warm reboot using the system BIOS.
Booting is slow
The first thing to do is isolate which part of the boot process is slow to determine if the fault is the kernel, the initrd scripts, or other parts of the boot process. One way to do this is using the bootchart application. Install this with yum, and the next time you reboot, profiling will be done during boot which can be collected by running the command bootchart which will generate a .png file containing a graph showing where the time was spent. If the kernel appears to stall during boot, booting with the boot parameter printk.time=1 will insert timestamps before every message the kernel prints to its ringbuffer. Retrieve these messages with dmesg, and look for large deltas between two timestamps to isolate (for eg) drivers which may be spending a long time initialising.
Creation of slab failed
In Rawhide/devel kernels (and in -debug flavors of released kernels), Fedora uses the SLUB allocator with full slab debugging enabled by default. The debugging might cause problems in some rare cases: memory allocations can fail, causing the system to panic. Slab debugging can be disabled with the option slub_debug=- (a single minus sign.) Note, that this option will hide an actual bug that really should be reported and fixed rather than worked around.
USB devices don't work
This can be caused by USB autosuspend stopping and starting devices repeatedly. To disable autosuspend globally, use the kernel option usbcore.autosuspend=-1 .
Problems with PCMCIA / PC Card adapters
By default, the kernel only reserves a fairly small amount of memory and I/O space for PC Card adapters. Some adapters need more space, or will not work within the default range of addresses.
- The amount of memory allocated can be set using the
cbmemsizekernel option. Default is 64 megabytes, but it can be changed to e.g. 256 megabytes using the optioncbmemsize=256M. Going over 256M is not recommended.
- The default for Cardbus IO space is 256 bytes, but it can be changed using
cbiosize, e.g. to change the size to 4096 bytes, usecbiosize=4096. Setting this to a value larger than 4096 may cause problems.
nVidia SATA controllers don't recognize all connected drives
- Try the kernel parameter
pnpacpi=off
Differences between the Fedora 7 and Fedora 8 kernels
How to make Fedora 8 behave like Fedora 7 and vice versa
In Fedora 8, USB autosuspend is enabled while it is disabled in Fedora 7
- To make Fedora 8 behave like Fedora 7, add
usbcore.autosuspend=-1to the kernel options. - To make Fedora 7 behave like Fedora 8, add
usbcore.autosuspend=2to the kernel options.
In Fedora 8, libata ACPI is enabled while it is disabled in Fedora 7
- To make Fedora 8 behave like Fedora 7, add
noacpi=1to the libata module's options. - To make Fedora 7 behave like Fedora 8, add
noacpi=0to the libata module's options.
In Fedora 8, PCI message signaled interrupts (MSI) are enabled while they are disabled in Fedora 7
- To make Fedora 8 behave like Fedora 7, add
pci=nomsito the kernel options. - To make Fedora 7 behave like Fedora 8, add
pci=msito the kernel options.
