Kernel Debugging Tricks

This page describes some tricks and techniques to help debug the kernel.
1) Changing the ring buffer size
 The internal kernel console message buffer can sometimes be too small to capture all of the printk messages, especially when debug code generates a lot of printk messages. If the buffer fills up, it wraps around and one can lose valueable debug messages.
 To increase the internal buffer, use the kernel boot parameter: log_buf_len=N
             where N is the size of the buffer in bytes, and must be a power of 2.
 If one does not specify the log level then the default log level of KERN_WARNING is used.
For example, enable all levels of console message: echo 7 > /proc/sys/kernel/printk
To view console messages at boot, remove the quite and splash boot parameters from the kernel boot line in grub. This will disable the usplash splash screen and re-enable console messages
2) Serial Console
Serial console enables one to dump out console messages over a serial cable. Most modern PCs do not have legacy serial ports, so instead, one can use a USB serial dongle instead. A "null serial cable" or "universal file transfer cable" is needed to connect the target computer with the host.
 In addition, one needs to enable USB serial support as a kernel build configuration: CONFIG_USB_SERIAL_CONSOLE=y CONFIG_USB_SERIAL=y  and enable the appropriate driver, e.g.: CONFIG_USB_SERIAL_PL2303=y  and boot this kernel with console=tty console=ttyUSB0,9600 one may need to adjust the baud rate appropriately.

 Note: Generally, there is NO hardware or software flow control on serial console drivers, which means one may get dropped characters when running very high speed tty baud rates, such as 115200 baud.

3) Slowing down kernel messages on boot
 One may find a machine hangs during the kernel boot process and one would like to be able to see all the kernel messages but unfortunately they scroll off the console too quickly. One can slow down kernel console messages at boot time using by building the kernel with the following option enabled: CONFIG_BOOT_PRINTK_DELAY=y
and boot the machine with the following kernel boot parameter: boot_delay=N
   where N = msecs delay between each console message.

4)Kernel panic during suspend
 Debugging suspend/resume issues can be difficult if the kernel panics during suspend, especially late in the suspend because console messages are disabled. One can stop console messages from being suspended by using the kernel parameter no_console_suspend: no_console_suspend=1
This will force the console not to suspend. Boot with this option, chvt 1 (to console #1), and suspend using pm-suspend