Thursday, February 9, 2012

Linux early boot debug

As a part of my project at NCSU (North Carolina State University), I am trying to setup the development environment for a FreeScale Board (i.MX53QSB). This includes the ability to build a custom kernel. However, when I tried to cross-compile the kernel and load it on the board, the console sadly hung at

Uncompressing Linux... done, booting the kernel

Armed with my previous experience at the u-boot/kernel interface, I made sure that the architecture number in uboot and kernel matched. At this stage the kernel is in its very nascent stages of booting. No drivers have been initialized, including the in-kernel UART drivers. This means no kgdb. printk and such debugging interfaces do not work at such early stages of booting. JTAG would probably work, but my exposure to JTAG is limited. Nonetheless, the kernel being as awesome as it is, there is an excellent way to debug at even such an early stage: The Kernel Low-Level debug facilities. 

In brief, this particular (and many other boards too) board comes with a debug UART. UART1 is set in debug mode by uboot. Enabling CONFIG_DEBUG_LL in the kernel will link in some functions that will enable early code to setup and use this debug UART. The logs can then be viewed easily on the host machine.

Following are the steps to enable low-level debug:
1) make ARCH=arm menuconfig
2) Go to "Kernel Hacking". Enable "Kernel Debugging". This will unhide lots of options, including "Kernel low-level Debugging functions". Enabling that will unhide "Early printk". Enable "Early printk" too.

After a full recompile, append "earlyprintk" to the bootargs in the uboot environment (Usually it is the ${bootargs} variable). Thats it!

Now, booting the kernel shows a step-by-step log of what the kernel is doing. In my case, I found out that there was an unhandled alignment trap that led to a kernel crash. The fix was to add "-mno-unaligned-access" to CFLAGS while kernel compilation. After solving a couple of issues, now I am stuck here. 

Uncompressing Linux... done, booting the kernel.
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 218112
Kernel command line: console=ttymxc0,115200 earlyprintk root=/dev/mmcblk0p1 rw rootwait
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 352MB 512MB = 864MB total
Memory: 871240k/871240k available, 13496k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xf4600000 - 0xffe00000   ( 184 MB)
    vmalloc : 0xe0800000 - 0xf2000000   ( 280 MB)
    lowmem  : 0x80000000 - 0xe0000000   (1536 MB)
    pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
    modules : 0x7f000000 - 0x7fe00000   (  14 MB)
      .init : 0x80008000 - 0x80030000   ( 160 kB)
      .text : 0x80030000 - 0x804ebc68   (4848 kB)
      .data : 0x804ec000 - 0x80539ca0   ( 312 kB)
SLUB: Genslabs=11, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
TrustZone Interrupt Controller (TZIC) initialized
MXC GPIO hardware
imx-uart.0: ttymxc0 at MMIO 0x53fbc000 (irq = 31) is a IMX

As soon the kernel UART drivers kick in, my console starts spitting out garbage. This should be an easy fix now.
/me goes back to poring over the UART driver 


lalitanand dandge said...

Hi Jitesh,
I am trying to work with the early prints from the linux kernel. I am using OMAP4 based Pandaboard for my evaluation.

I would like to understand the flow or the interaction of the different drivers[serial, console, uart etc] relating the early prints.

What I am trying to achieve is! using usb serial gadget driver to get the early prints on usb.

I am trying the patch given at

Can you please provide me some pointers on uart, serial and console drivers.

Thanks and regards,

Happy said...
This comment has been removed by the author.
Happy said...

Hi Jitesh

Imx35 getting stuck with "Uncompressing Linux ................."

This question is Not Answered.(Mark as assumed answered)
santhosh sl Level 1
santhosh sl Jan 23, 2013 4:53 AM

Following steps I followed to load the Demo Linux image provided in to IMX35

Booting from MMC:

1.Erase complete MMC

2.$ sudo dd if=./Desktop/mx35_3stack_redboot_mmc.bin of=/dev/sdd bs=512 skip=2 seek=2 -- To copy Redboot image on to MMC


Loading Image:

with Redboot booted up

1. load -r -b 0x100000 -m xmodem

Redboot now is ready to receive data and printing out character “C” continuously. To send a file, click on “Transfer -

> Send File -> Xmodem (under Protocol) -> Browse”, choose the file to download and then click on “Send”

2. fis init -- To initialize flash

3. fis create -f 0x200000 kernel -- Section on flash to copy our Image

4. fis load -d kernel

5. fis list --

here we can see following sections being created

... Read from 0x07ee0000-0x07eff000 at 0x00060000: .

Name FLASH addr Mem addr Length Entry point

RedBoot 0x00000000 0x00000000 0x00040000 0x00000000

FIS directory 0x00060000 0x00060000 0x0001F000 0x00000000

RedBoot config 0x0007F000 0x0007F000 0x00001000 0x00000000

kernel 0x00200000 0x00100000 0x00200000 0x00100000

6. Then when i give Run, it is getting stuck at following error

load entry_address=0x100000



Uncompressing Linux.............................................................

Can you help me on this.