Embedded Linux. Debugging the kernel

Coming to embedded linux from the world of microcontrollers, such a familiar tool for debugging code as step-by-step debugging of code on a target piece of hardware using a hardware programmer was sorely lacking. The previous articles describe how we learned to debug the u-boot: 1, 2 bootloader. The kernel turned out to be more complicated. For example, it turned out that the Linux kernel cannot be compiled with optimization disabled (-O0). The article describes how we still managed to run the kernel on an ARM microprocessor in step-by-step debugging mode.

Source preparation

$ git clone https://github.com/wireless-road/imx6ull-openwrt.git
$ cd imx6ull-openwrt
$ ./compile.sh flexcan_ethernet

The kernel sources after the build is complete can be found here:


Building a kernel with the -Og flag

First of all, we tried to compile the kernel with optimization disabled and came across an unpleasant surprise:

$ cd ./build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-imx6ull_cortexa7/linux-4.14.199/
$ make menuconfig

Such an opportunity is not provided in the prinitsa! Only two optimization options are offered: -O2 and -Os… Debugging in GDB in both cases will not do anything good – instead of stepping through the code Program Counter will chaotically jump over entire pieces of code and function calls. When you try to subtract the values ​​of variables, you will come across the message every now and then: Optimized Out… If you crawl into the .config and Makefile and try to build a kernel with a flag -O0then the assembly will crash with a lot of error messages. How It revealed, this is in principle impossible, since the optimization when building the kernel is used for completely different purposes, for which optimization flags should not be used, namely, to disable unused code. A dirty hack, which, apparently, can not get rid of. Fortunately for us, some people before us were still puzzled by the problem of debugging the kernel and even wrote patchwhich allows you to build the kernel with the flag -Ogbut, for some reason, rejected by the community. As a result, we had to adapt it manually for our sources.

The essence of the patch is to add a third version of the debug-optimized kernel assembly:

+KBUILD_CFLAGS	+= $(call cc-disable-warning,maybe-uninitialized,)
 KBUILD_CFLAGS   += -O2 -fno-reorder-blocks -fno-tree-ch $(EXTRA_OPTIMIZATION)

and disable compile-time errors by defining:

+#if !defined(__CHECKER__) && !defined(CONFIG_CC_OPTIMIZE_FOR_DEBUGGING)
 # define __compiletime_warning(message) __attribute__((warning(message)))
 # define __compiletime_error(message) __attribute__((error(message)))
 #endif /* __CHECKER__ */

The full code of the patch and related changes can be found here: 1, 2

Device Tree and JTAG

Next, you need to make sure that the pins of the microprocessor, to which the JTAG interface is brought out, are not reused for any other purposes. To do this, open the dts file we are using and look for references to jtag pins. If you find something similar:

pinctrl_sai2: sai2grp {
	fsl,pins = <

then you’re out of luck. You can only get the ability to debug the kernel by disabling the interface that JTAG pins are reusing. They will have to disable… Those. in the worst case, if you need to debug functionality that uses the JTAG pins of the microprocessor, you will not be able to do this. Those. the need to use the JTAG interface must be laid down at the stage of developing the printed circuit board of the device. Or check the documentation for the purchased module.

The last thing to do is create a configuration assemblies, in which the default is optimized for debugging purposes:


Then you can rebuild the image:

./compile.sh flexcan_ethernet

or just the kernels separately:

make target/linux/compile

Eclipse graphical IDE

Its settings for kernel sources are practically no different from the settings for debugging the U-boot loader: the section “Installing the IDE and creating a project” of the previous article, with the only difference that when creating a project, you need to select the directory with the kernel sources instead of the bootloader sources.

At the moment, we have not yet been able to correctly debug the kernel from Eclipse, and therefore it is so far used only to navigate through the code. In the future, we plan to add a full-fledged graphical debugging of the code from the IDE.

Debugging the kernel in console mode

So, let’s proceed directly to debugging.

We start the piece of hardware and interrupt the kernel loading to stay in the U-boot console:

We start the GDB server:

$ JLinkGDBServer -device MCIMX6Y2 -if JTAG -speed 1000

We start the GDB session:

$ cd ./build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-imx6ull_cortexa7/
$ gdb-multiarch vmlinux.debug --nx

in the console gdb session:

(gdb) target remote localhost:2331
(gdb) restore flexcan_ethernet-uImage binary 0x82000000
(gdb) restore image-flexcan_ethernet.dtb binary 0x83000000
(gdb) b __hyp_stub_install
Breakpoint 1 at 0x80110b20: file arch/arm/kernel/hyp-stub.S, line 89.
(gdb) c

after that the bootloader console should come to life. Load the kernel in it:

=> bootm 82000000 - 83000000

After that switch back to the gdb session console, you should see something like this:

Breakpoint 1, __hyp_stub_install () at arch/arm/kernel/hyp-stub.S:89
89 store_primary_cpu_mode r4, r5, r6

You are at the point where the kernel starts! Try to walk through the code with commands s and n, and display the values ​​of variables / registers with the command p:

If you continue to step through the code with the s and n commands, you can get tired. To quickly get to the function you are interested in, set a breakpoint, for example, the function start_kernel:

(gdb) b start_kernel
(gdb) c

try to walk in it. There should not be any chaotic movement through the code, i.e. if you give the command n (function jump), then you should not unexpectedly end up in the body of any other function. The values ​​of most of the variables found in the code should also be readable without any messages “optimized out“For a check, you can use Eclipse to make sure everything is really going according to plan:

That’s all.

In further articles, we:

  • run u-boot and the kernel sequentially in the debugger;

  • Let’s figure out how the bootloader transfers control to the kernel;

  • build a kernel boot map by analogy with the U-boot map;

  • We will pack development tools in a docker container, which will minimize errors when deploying a development environment by new developers (including beginners), including launching a graphical IDE and connecting to a USB programmer from a docker container;

  • learn how to debug kernel modules and much more.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *