Kernel Degbugging Tips

Detecting hung tasks in Linux

Linux kernels have an infrastructure to detect hung tasks. When this infrastructure is active it will periodically get activated to find out hung tasks and present a stack dump of those hung tasks (and maybe locks held).

The infrastructure is based on a single kernel thread named as "khungtaskd" So if you do a ps in your system and see that there is entry like [khungtaskd] you know it is there I have one in my system: 
136 root SW [khungtaskd] 

The loop of the khungtaskd daemon is a call to the scheduler for waking it up after ever 120 seconds (default value) The core algorithm is like this: 
1. Iterate over all the tasks in the system which are marked as TASK_UNINTERRUPTIBLE (additionally it does not consider UNINTERRUPTIBLE frozen tasks & UNINTERRUPTIBLE tasks that are newly created and never been scheduled out). 2. If a task has not been switched out by the scheduler atleast once in the last 120 seconds it is considered as a hung task and its stack dump is displayed. If CONFIG_LOCKDEP is defined then it will also show all the locks the hung task is holding. 

 Kernel Log (Ring) Buffer

Linux kernel generates log messages using printk(). These messages are stored in a "ring buffer".

The log levels decide the importance of the message being printed, kernel defines 8 log levels in the file printk.h 

#define KERN_EMERG "<0>" /* system is unusable*/
#define KERN_ALERT "<1>" /* action must be taken immediately*/
#define KERN_CRIT "<2>" /* critical conditions*/
#define KERN_ERR "<3>" /* error conditions*/
#define KERN_WARNING "<4>" /* warning conditions*/
#define KERN_NOTICE "<5>" /* normal but significant condition*/
#define KERN_INFO "<6>" /* informational*/
#define KERN_DEBUG "<7>" /* debug-level messages*/

The console loglevel can be found by looking into the file /proc/sys/kernel/printk 

$ cat /proc/sys/kernel/printk
4 4 1 7

The first number in the output is the console log level, the second is the default log level, third is the minimum log level and fourth is the maximum log level.

The console log level can be changed by writing into the proc entry 

$ echo "6" > /proc/sys/kernel/printk
$ cat /proc/sys/kernel/printk
6 4 1 7

 Stack Corruption

Stack corruption is a phenomenon in which some memory locations at stack are accessed unintentionally due to wrong coding leading to change in values at those memory locations. Since the data corruption happens on stack memory locations, hence the term Stack Corruption.

There can be quite a few ways in which stack corruption may occur : 

1. When due to some weirdly written code, all the stack memory gets eaten up
2. Accessing array out of bounds
3. An undefined/freed pointer pointing or storing a garbage stack address.
4. When due to some reason, the return address for a function call gets corrupted.

Back trace core dump file will understand the problem of Stack corruptions.

What are the CPU states found in "top" output?

Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

# us -> User CPU time: The time the CPU has spent running users’ processes that are not niced.
# sy -> System CPU time: The time the CPU has spent running the kernel and its processes.
# ni -> Nice CPU time: The time the CPU has spent running users’ process that have been niced.
# wa -> iowait: Amount of time the CPU has been waiting for I/O to complete.
# hi -> Hardware IRQ: The amount of time the CPU has been servicing hardware interrupts.
# si -> Software Interrupts.: The amount of time the CPU has been servicing software interrupts.