An “Oops” is what the kernel throws at us when it finds something faulty, or an exception, in the kernel code. It’s somewhat like the segfaults of user-space. An Oops dumps its message on the console; it contains the processor status and the CPU registers of when the fault occurred. The offending process that triggered this Oops gets killed without releasing locks or cleaning up structures. The system may not even resume its normal operations sometimes; this is called an unstable state. Once an Oops has occurred, the system cannot be trusted any further.

Let’s try to generate an Oops message with sample code, and try to understand the dump.

Setting up the machine to capture an Oops

The running kernel should be compiled with CONFIG_DEBUG_INFO, and syslogd should be running. To generate and understand an Oops message, Let’s write a  sample kernel module,oops.c:

The associated Makefile for this module is as follows:

Once executed, the module generates the following Oops:

Understanding the Oops dump

Let’s have a closer look at the above dump, to understand some of the important bits of information.

The first line indicates a pointer with a NULL value.

IP is the instruction pointer.

This is the error code value in hex. Each bit has a significance of its own:

• bit 0 == 0 means no page found, 1 means a protection fault
• bit 1 == 0 means read, 1 means write
• bit 2 == 0 means kernel, 1 means user-mode
• [#1] — this value is the number of times the Oops occurred. Multiple Oops can be triggered as a cascading effect of the first one.

This denotes on which CPU the error occurred.

The Tainted flag points to P here. Each flag has its own meaning. A few other flags, and their meanings, picked up from kernel/panic.c:

• P — Proprietary module has been loaded.
• F — Module has been forcibly loaded.
• S — SMP with a CPU not designed for SMP.
• R — User forced a module unload.
• M — System experienced a machine check exception.
• B — System has hit bad_page.
• U — Userspace-defined naughtiness.
• A — ACPI table overridden.
• W — Taint on warning.

RIP is the CPU register containing the address of the instruction that is getting executed. 0010comes from the code segment register. my_oops_init+0x12/0x21 is the <symbol> + the offset/length.

This is a dump of the contents of some of the CPU registers.

The above is the stack trace.

The above is the call trace — the list of functions being called just before the Oops occurred.

The Code is a hex-dump of the section of machine code that was being run at the time the Oops occurred.

Debugging an Oops dump

The first step is to load the offending module into the GDB debugger, as follows:

Next, add the symbol file to the debugger. The add-symbol-file command’s first argument isoops.o and the second argument is the address of the text section of the module. You can obtain this address from /sys/module/oops/sections/.init.text (where oops is the module name):

From the RIP instruction line, we can get the name of the offending function, and disassemble it.

Now, to pin point the actual line of offending code, we add the starting address and the offset. The offset is available in the same RIP instruction line. In our case, we are adding0x0000000000000038 + 0x012 =  0x000000000000004a. This points to the movl instruction.

This gives the code of the offending function.

References

The kerneloops.org website can be used to pick up a lot of Oops messages to debug. The Linux kernel documentation directory has information about Oops — kernel/Documentation/oops-tracing.txt. This, and numerous other online resources, were used while creating this article.