Thursday, February 9, 2012

Linux early boot debug

As a part of my project at NCSU (North Carolina State University), I am trying to setup the development environment for a FreeScale Board (i.MX53QSB). This includes the ability to build a custom kernel. However, when I tried to cross-compile the kernel and load it on the board, the console sadly hung at

Uncompressing Linux... done, booting the kernel

Armed with my previous experience at the u-boot/kernel interface, I made sure that the architecture number in uboot and kernel matched. At this stage the kernel is in its very nascent stages of booting. No drivers have been initialized, including the in-kernel UART drivers. This means no kgdb. printk and such debugging interfaces do not work at such early stages of booting. JTAG would probably work, but my exposure to JTAG is limited. Nonetheless, the kernel being as awesome as it is, there is an excellent way to debug at even such an early stage: The Kernel Low-Level debug facilities. 

In brief, this particular (and many other boards too) board comes with a debug UART. UART1 is set in debug mode by uboot. Enabling CONFIG_DEBUG_LL in the kernel will link in some functions that will enable early code to setup and use this debug UART. The logs can then be viewed easily on the host machine.

Following are the steps to enable low-level debug:
1) make ARCH=arm menuconfig
2) Go to "Kernel Hacking". Enable "Kernel Debugging". This will unhide lots of options, including "Kernel low-level Debugging functions". Enabling that will unhide "Early printk". Enable "Early printk" too.

After a full recompile, append "earlyprintk" to the bootargs in the uboot environment (Usually it is the ${bootargs} variable). Thats it!

Now, booting the kernel shows a step-by-step log of what the kernel is doing. In my case, I found out that there was an unhandled alignment trap that led to a kernel crash. The fix was to add "-mno-unaligned-access" to CFLAGS while kernel compilation. After solving a couple of issues, now I am stuck here. 

Uncompressing Linux... done, booting the kernel.
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 218112
Kernel command line: console=ttymxc0,115200 earlyprintk root=/dev/mmcblk0p1 rw rootwait
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 352MB 512MB = 864MB total
Memory: 871240k/871240k available, 13496k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xf4600000 - 0xffe00000   ( 184 MB)
    vmalloc : 0xe0800000 - 0xf2000000   ( 280 MB)
    lowmem  : 0x80000000 - 0xe0000000   (1536 MB)
    pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
    modules : 0x7f000000 - 0x7fe00000   (  14 MB)
      .init : 0x80008000 - 0x80030000   ( 160 kB)
      .text : 0x80030000 - 0x804ebc68   (4848 kB)
      .data : 0x804ec000 - 0x80539ca0   ( 312 kB)
SLUB: Genslabs=11, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
TrustZone Interrupt Controller (TZIC) initialized
MXC GPIO hardware
imx-uart.0: ttymxc0 at MMIO 0x53fbc000 (irq = 31) is a IMX

As soon the kernel UART drivers kick in, my console starts spitting out garbage. This should be an easy fix now.
/me goes back to poring over the UART driver