18 Debugging

Debugging the Linux Kernel

Prerequisites and Bug Characteristics

Successful kernel debugging requires a well-defined, reliably reproducible bug, identification of the kernel version where the bug originated, and comprehension of the associated code.

• Root Causes: Kernel bugs stem from incorrect code, synchronization failures (improper locking), or incorrect hardware management.
• Execution Cascades: Minor errors often trigger a chain of events culminating in a fatal error.
  • A race condition due to missing reference counts leads to a freed structure.
  • Subsequent access to the structure yields garbage data or a NULL pointer dereference.
  • The dereference triggers an unrecoverable system state, resulting in a system crash.

To identify the exact point of failure in this cascade, the primary tool available for introspection is kernel-level printing.

Debugging by Printing (printk)

The printk() function is the kernel equivalent of the C library printf() function, augmented with specialized robust execution capabilities and logging levels.

• Robustness and Constraints:
  • Callable from almost any context, including interrupt context and process context.
  • Callable simultaneously across multiple processors without requiring the caller to hold a lock.
  • Cannot be used prior to console initialization during early boot. The architecture-specific early_printk() function bypasses this limitation for early debugging.
• Loglevels:
  • Strings prepended to messages (e.g., <4>) determine whether the message is output to the console based on the current console_loglevel.
  • Levels range from KERN_EMERG (<0>, highest priority) to KERN_DEBUG (<7>, lowest priority).
  • Messages lacking a specified loglevel default to DEFAULT_MESSAGE_LOGLEVEL (typically KERN_WARNING).
• The Log Buffer:
  • Messages are written to a circular buffer of size LOG_BUF_LEN (configurable via CONFIG_LOG_BUF_SHIFT, defaulting to 16KB on uniprocessor systems).
  • Allows asynchronous, concurrent read/write access, making it safe for interrupt contexts.
  • Prevents memory exhaustion by overwriting the oldest messages when the buffer reaches maximum capacity.
• User-Space Integration:
  • The klogd daemon extracts messages from the buffer by reading /proc/kmsg or executing the syslog() system call.
  • Messages are passed to the syslogd daemon, which appends them to /var/log/messages.

When printk reveals a critical inconsistency, or the kernel encounters a fault it cannot log safely, execution halts and the system issues an oops.

The Kernel Oops

An oops is the mechanism by which the kernel signals a fatal error, handling the failure by printing an error message, dumping register contents, and providing a function back trace.

• Contextual Behavior:
  • If the oops occurs in process context, the kernel kills the offending process and attempts to continue executing.
  • If the oops occurs in interrupt context, the idle task (PID 0), or the init task (PID 1), the kernel cannot safely recover and initiates a panic(), halting the system.
• Decoding Back Traces:
  • ksymoops: An external user-space utility used historically to translate memory addresses into symbolic function names by mapping them against the System.map file.
  • kallsyms: An in-kernel feature enabled via CONFIG_KALLSYMS that natively stores symbolic names of function addresses in the kernel image, eliminating the need for external decoding tools.

Analyzing an oops requires exposing granular fault data, which is achieved by enabling specific kernel configuration options.

Configuration Options and Assertions

The kernel provides compile-time configuration options and runtime macros to enforce architectural rules and immediately flag deviations.

• Debug Configurations:
  • CONFIG_DEBUG_KERNEL: Enables the overarching kernel hacking menu.
  • CONFIG_DEBUG_SPINLOCK_SLEEP: Enables the central atomicity counter to detect operations that might sleep while holding a lock or operating with disabled kernel preemption.
• Runtime Assertions:
  • BUG() and BUG_ON(): Trigger an architecture-specific illegal instruction, forcing an oops and a stack trace. BUG_ON() wraps the assertion in an unlikely() compiler branch optimization.
  • BUILD_BUG_ON(): Evaluates expressions at compile time, aborting compilation if the condition evaluates to true.
  • panic(): Prints a fatal error message and permanently halts the kernel.
  • dump_stack(): Prints register contents and a back trace to the console without killing the process or halting the system.

When assertions fail and lock the system entirely, low-level hardware interrupts provide a fallback recovery mechanism.

Magic SysRq Key

The Magic SysRq key (CONFIG_MAGIC_SYSRQ) acts as a non-maskable hardware escape hatch to communicate with an otherwise unresponsive kernel.

• Activation: Enabled dynamically by writing to the sysctl interface: echo 1 > /proc/sys/kernel/sysrq.
• Execution: Invoked via standard key combinations (e.g., Alt+PrintScreen on x86).
• Core Commands:
  • SysRq-s: Synchronizes all dirty buffers to disk.
  • SysRq-u: Unmounts all mounted filesystems.
  • SysRq-b: Reboots the machine at the hardware level.

If the system remains unresponsive or deeper memory inspection is required, dedicated debuggers must be attached to the running kernel.

Kernel Debuggers

Specialized debuggers inspect the live kernel memory space or establish remote serial connections to manipulate data dynamically.

• gdb (GNU Debugger):
  • Executes against the uncompressed kernel image (vmlinux) and reads live memory from /proc/kcore.
  • Capable of printing variables and disassembling functions.
  • Strictly read-only; lacks support for memory modification, breakpoints, or single-stepping.
• kgdb:
  • A patch enabling remote debugging over a serial line connecting a test machine to a development machine.
  • Provides full debugger functionality: reading/writing variables, setting breakpoints, and single-stepping execution.

Because debuggers are heavy tools, isolated functional testing often relies on strategic code injections and system probes.

Poking and Probing Techniques

Developers implement localized scaffolding to isolate test code or monitor specific execution frequencies without halting the kernel.

• Execution Fencing:
  • UID Conditionals: Wraps experimental algorithms in a check against a specific User ID (e.g., if (current->uid != 7777)), preventing system-wide instability during process-context testing.
  • Condition Variables: Global integer flags acting as toggle switches, modified dynamically via debuggers to route execution paths.
• Telemetry and Throttling:
  • Statistics: Global variables (e.g., unsigned long foo_stat) incremented per event to track occurrence ratios.
  • Rate Limiting: Capping output frequencies based on elapsed jiffies compared against the HZ timer frequency. The printk_ratelimit() API automates this, defaulting to one message every 5 seconds after an initial burst of 10.
  • Occurrence Limiting: Maintaining a static counter to hard-cap execution of a debug block to an absolute number of iterations.

If targeted probing fails to locate the root cause in the current source, isolating the historical commit that introduced the fault becomes necessary.

Binary Searching to Find the Culprit Change

Binary searching systematically halves the commit history between a known-working kernel version and a known-broken kernel version to isolate the exact source modification that introduced the bug.

• Git Automation (git bisect):
  • git bisect start: Initializes the search algorithm.
  • git bisect bad <revision>: Flags the revision where the bug is present.
  • git bisect good <revision>: Flags the last known stable revision.
  • Git automatically checks out the median commit. The developer compiles and tests, recursively marking the state as good or bad until Git identifies the singular offending commit.
  • Directory scopes can be restricted to isolate the search (e.g., git bisect start -- arch/x86).

Exhausting all local debugging and bisection techniques necessitates escalation to the broader development community.

Community Escalation

When local analysis fails, bug reports and decoded oops data are escalated to the Linux Kernel Mailing List (LKML) and the designated subsystem maintainers. Submissions require precise reproduction steps, hardware specifications, and isolated patches formatted cleanly for peer review.