Linux early boot debug

As a part of my project at NCSU (North Carolina State University), I am trying to setup the development environment for a FreeScale Board (i.MX53QSB). This includes the ability to build a custom kernel. However, when I tried to cross-compile the kernel and load it on the board, the console sadly hung at

Uncompressing Linux... done, booting the kernel

Armed with my previous experience at the u-boot/kernel interface, I made sure that the architecture number in uboot and kernel matched. At this stage the kernel is in its very nascent stages of booting. No drivers have been initialized, including the in-kernel UART drivers. This means no kgdb. printk and such debugging interfaces do not work at such early stages of booting. JTAG would probably work, but my exposure to JTAG is limited. Nonetheless, the kernel being as awesome as it is, there is an excellent way to debug at even such an early stage: The Kernel Low-Level debug facilities. 

In brief, this particular (and many other boards too) board comes with a debug UART. UART1 is set in debug mode by uboot. Enabling CONFIG_DEBUG_LL in the kernel will link in some functions that will enable early code to setup and use this debug UART. The logs can then be viewed easily on the host machine.

Following are the steps to enable low-level debug:

1) make ARCH=arm menuconfig

2) Go to "Kernel Hacking". Enable "Kernel Debugging". This will unhide lots of options, including "Kernel low-level Debugging functions". Enabling that will unhide "Early printk". Enable "Early printk" too.

After a full recompile, append "earlyprintk" to the bootargs in the uboot environment (Usually it is the ${bootargs} variable). Thats it!

Now, booting the kernel shows a step-by-step log of what the kernel is doing. In my case, I found out that there was an unhandled alignment trap that led to a kernel crash. The fix was to add "-mno-unaligned-access" to CFLAGS while kernel compilation. After solving a couple of issues, now I am stuck here. 

Uncompressing Linux... done, booting the kernel.
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 218112
Kernel command line: console=ttymxc0,115200 earlyprintk root=/dev/mmcblk0p1 rw rootwait
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 352MB 512MB = 864MB total
Memory: 871240k/871240k available, 13496k reserved, 0K highmem
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    DMA     : 0xf4600000 - 0xffe00000   ( 184 MB)
    vmalloc : 0xe0800000 - 0xf2000000   ( 280 MB)
    lowmem  : 0x80000000 - 0xe0000000   (1536 MB)
    pkmap   : 0x7fe00000 - 0x80000000   (   2 MB)
    modules : 0x7f000000 - 0x7fe00000   (  14 MB)
      .init : 0x80008000 - 0x80030000   ( 160 kB)
      .text : 0x80030000 - 0x804ebc68   (4848 kB)
      .data : 0x804ec000 - 0x80539ca0   ( 312 kB)
SLUB: Genslabs=11, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:848
TrustZone Interrupt Controller (TZIC) initialized
MXC GPIO hardware
imx-uart.0: ttymxc0 at MMIO 0x53fbc000 (irq = 31) is a IMX
�枞������`>�&���&�8�&���8����>枞����8��&

As soon the kernel UART drivers kick in, my console starts spitting out garbage. This should be an easy fix now.

/me goes back to poring over the UART driver 

Plotting a dependency graph

While working with building loads of packages, it is often necessary to visualise dependencies between them. Two years ago I suffered a lot trying to do just that and at best I came up with a python script that gives a textual list of deps. Today, I found an awesome tool to plot dependency graphs. Although useless for most of the people, it is quite a nifty trick for the few who work with thousands of packages.

Two RPMs need to be installed before anything else

# yum install rpmorphan graphviz

Now, assume you need all the deps for rebuilding "rpm" (The actual RPM rpm)
First, generate a textual dep graph using:

# rpmdep -dot rpm.dot rpm

The "dot" is a file-format which will be used by the "dot" program to plot a graph. To plot the graph

# dot -Tsvg rpm.dot -o rpm.svg

The command takes rpm.dot and plots the graph in an svg format.

Simple!!

A sample image I generated is here:

vim: How to setup awesome autocomplete

Recently I moved to a company which has a huge codebase and loves long names! All vim/emacs users will sympathize with such a situation. Eclipse would autocomplete names in a jiffy. I was sorely missing good autocomplete on my favourite editor: vim.

Typically by setting the "dict" variable appropriately, one can autocomplete english words, but what about C/C++ code?
One fine morning, I had an epiphany. Why don't people use the ctags/etags database to autocomplete. The database already has lots of semantic information about the code and it would have all the project specific keywords too. Now, I am not a guy who has too many epiphanies, so I was sure someone had implemented it even before I could spell "vim". With this in mind, I googled the specific terms "vim autocomplete using ctags". The first stackoverflow result had a brilliant suggestion: omniautocomplete.

Before I go into what omniautocomplete (a plug-in for vim) does, let me first list down my requirements from an autocomplete tool
1) Autocomplete sensibly: Use the knowledge that I am editing "c++" code and give sensible suggestions.
2) It should *not* be enabled by default when I run vim i.e. do not blindly putt crao in my .vim. The reason is that I work on multiple projects at the same time and I don't want suggestions from one project to be mixed with other.
3) No lag in autocompleting.
4) Provide a quick way to update the autocomplete "database".

Some vim scripting + omnicomplete provides all that and more. What more, you might ask. Well, it provides an eclipse-like pop-up when asked to autocomplete. It can also fire when I enter a "." or "->". It also has an option to show "function prototypes" alongwith autocomplete suggestions. This is awesome!!!

To get you excited, here is a screenshot. It looks exactly the same.

Setting up omnicomplete is super-easy. Installation instructions are outlined here. All it involves is unzipping a zip file to ~/.vim

The next part is more interesting:
Since ctags database is used, ctags needs to be compiled with some specific options. The commandline I picked up from their documentation is this:

$ ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .

Now, the "tags" variable in vim needs to point to the tags of your project. Setting this up each time you open vim can be irritating. Very irritating. So, I wrote the following vim function to alleviate the issue:

function! SetProj(proj)
        if a:proj == "mythread"
                map <C-F12> :!ctags -R --c++-kinds=+p --fields=+iaS --extra=+q -f /home/jitesh/repos/mythread_lib/tags /home/jitesh/repos/mythread_lib<CR>
                set tags+=/home/jitesh/repos/mythread_lib/tags
                let OmniCpp_ShowPrototypeInAbbr = 1
                let OmniCpp_MayCompleteDot = 1
        endif
endfunction

Now, when I open vim, all I have to do is:

:call SetProj("mythread")

Note - Re-generating the ctags database is also pretty easy, Just press Ctrl+F12. Notice the keymapping in the vim function above.
So, when I define a new function in my code, I just do a Ctrl+F12 and the tags are updated and ready to be auto-completed.

Pressing ctrl+P or ctrl+N invokes the autocompletion.

When I  start vim, none of this stuff is loaded, so there is no load-time penalty/bloat + I can work on multiple projects just by calling "SetProj" with the appropriate parameter.

I sign-off a happy user! vim FTW

Block processing and shell-scripts

Feature 1:
Shell-scripts can easily process files/records line-by-line.
A typical construct to do so is:

ps aux |
while read line do;
        echo $line
done


But, this kind of a thing isn't very easy to process outputs of
commands like ifconfig. ifconfig has output in blocks instead of
lines. Recently I had to do it and I was really surprised that I had
no idea of how to parse blocks of data in shell-script. After some
time, I figured it out myself. The pipe command pipes the output to a
shell. So what if that shell contains another shell-script instead of
a single command?
Here's how to parse ifconfig command. The script prints out a list of
interfaces and corresponding IP address (NIL if no IP): (Note:
Brackets fork a new shell)


ifconfig -a |
(
#Read the first line
while read line; do
    INTERFACE_ADDR=`echo $line | awk '{print $1}'`


    # Read second line in the block and store the IP address, if any
    read line;
    IP=`echo $line | grep "inet addr" | awk '{print $2}' | cut -d':' -f2`


    while true; do
        # Skip all other lines from the block
        read line;


        # End of block is a new line. Start parsing new block then
        if [ -z "$line" ]; then
            break;
        fi
    done
    echo $INTERFACE_ADDR $IP
done
)


There. This is albeit a simple example. You can construct truly
awesome block parsing constructs using the sub-shell (i.e. the
brackets '()' )

Feature 2:
Wonder why a shell-script doesn't have a block commenting feature?
Wonder no more.
Here is a cool trick using here-documents to comment out a block of shell script


: <<'END1'
"The commented block goes here"
END1


Notes:
1) ":" means "true". /dev/null equivalent for a shell script.
2) Make sure that the string "END1" is the same at both the places,
else, this thing is not going to work.
Happy scripting :-)

Jajuk

Writing a music player for the lazy is a overwhelmingly tough job to do. The lazy people expect all kinds of automatic stuff. Play by Genre, ambience, mood, ratings. Play with untagged songs nicely instead of dumping them under one "Unknown" Label. Automatic tagging of songs whenever possible. Automatically fetch Lyrics, cover, wiki information. Ability to detect which songs the user doesn't like and play them lesser and lesser. File-tree view, album-tree view, artist-tree view. Automatic purging of duplicates. Music discovery via similar artists or people-who-liked-this-also-liked-this. Internet Radio for the ones who are bored of their collection. Some ego-boosting features for music junkies like all kinds of graphs and statistics of your collection and the Genre you like the most.

Overall, I have wayy too many expectations out of a music player. It is more like a music manager for me. Till date I hadn't gotten over my first love Amarok-1.4 completely. I didn't like the direction Amarok-1.9 and 2.0 took, so I dumpled Amarok long ago. iTunes and all its Linux clones are disappointing altogether too, so do NOT mention them. Finally today I came across this Java-based player called "Jajuk". I have to say it the most I liked since Amarok-1.4. It has brilliant features, amazingly intuitive interface which somehow hides all the complexity behind the complex features it provides. There is just an endless list of features and they actually work! It is my new default music player.

There are some catches though ->
1) Being Java-based it is heavy. Once I got used to it though, it is not a problem.
2) Global hotkeys are not supported. Now this is a *must* have. I have filed a feature request. Lets see whether it gets footage :-D.

Give Jajuk a try. It is good.
Find it here: http://jajuk.info/index.php/Main_Page

Traffic Shaping

So today was a rather weird day and I had a searing pain in my head. So I decided on a special masochistic thing to learn : traffic shaping. Since I am working on a project that needs this exclusively, I reckoned it will be helpful too.

I am sure most of you must have come across this term. In case you haven't, the wiki page here provides a good introduction. Traffic shaping is mostly used by ISPs and people who manage routers. It is a way to give priority to certain users or certain types of data. The term "Giving priority" hugely underestimates the complexity of the task. eg. Real time data streaming may value real-time data delivery whereas others might tolerate some delay. In general, when traffic shaping goes near QoS, it becomes a difficult problem to solve.

However, here I am going to rant about the ISP aspect of traffic shaping (i.e. allocating bandwidth per user, per service, etc). "tc" is a very powerful command in Linux to implement traffic shaping. I won't be giving sample commands or such because tc is quite complex. Instead (if anyone ever ends up using this post, which I strongly suspect is going to be my future self) I will give out links to tutorials I used for my setup.

Some introduction to tc is in line. With tc you can create queues per device. Multiple queues. You can choose which queueing discipline you are going to use. Queueing discipline says how the queue is managed. You can have a standard FIFO queue or a Token-Bucket filter (neatly manages bandwidth difference between two lines) or Stochastic Fair Queueing (ensures fairness). Queues can be defined as hierarchy and they can inherit each other's properties. Eg. I might define a queue for www traffic which takes up 40% of the bandwidth and 60% of everything else. Now, I can define another queue for ssh which is a child of the "other" queue (it basically means that the ssh queue can borrow bandwidth from its parent and go beyond its limit to occupy upto 60% of the bandwidth. This way you guarantee minimum service). On routers, each queue can represent a customer or a chunk of customers. High paying customers get more share of the bandwidth. After we are done defining this classes of traffic, next step is to define rules for saying which traffic belongs to which class. tc provides a wide variety of filters for the purpose. You can look at almost any of the TCP or IP headers, which interface the packet comes from, etc. For all purposes, this set of filters proves to be sufficient.

Now that we know how tc works, lets see what all tc can do:
1) Provide only a defined amount of bandwidth to a particular user (write a filter based on IP address)
2) Provide a defined amount of bandwidth to a particular service (write a filter based on port number)
3) Enable flexibility. i.e. if there is a burst of traffic in one class accommodate the burst instead of providing hard boundaries. (burst and cburst parameters control that)
4) Rates can be specified in percentage of total as well as actual absolute values like 8kbits, etc.

An addition to tc can do more wonderful stuff. It is called "netem" which stands for network emulator. netem has tunable knobs for any parameter you can conceive which can help you simulate a WAN. Here is what we tried today:
1) Simulate delays. If you are simulating a WAN at home, the major issue is that delays don't get simulated. With tc combined with netem, you can introduce fake delays. You can even vary delays about a point. eg. You can say probabilistically change delay at 100ms +- 10ms. You can even change the probability distribution by which it randomises the delay.
2) Simulate losses. When in WAN loses are inevitable. Either due to network congestion or corruption. netem with tc can simulate both. You can simulate percentage losses, losses in bursts, losses following a specific probability distribution, losses following a particular pattern. Anything under the sun.
3) Packet duplication, packet corruption, packet reordering.
4) Introducing a latency is only a defined type of traffic (to test QoS)

Anyone who knows a little about networks realises how difficult each one of this is to implement. Tuning all these knobs appropriately can give you an awesome simulation of the internet. I am working on a project which needs me to simulate network congestion at home. With a line-speed router and a speed of 100mbps, it is really tough to simulate congestion. We used tc command exclusively (with a lot of actual studies which has derived the loss and corruption values) to successfully simulate the internet at home. We now a get a very good TCP congestion window graph. The things that tc can do are pretty amazing. Unfortunately tc is a complex command and needs a lot of knowledge first to start using it. Fret not. Here are links to some tutorials that can get you going:

The classic TLDP tutorial
HTB tutorial : Contains some excellent explanations and practical examples with commands.
netem Introduction

Learning tc is a process and the end result is pretty satisfying. \m/ to the author of tc.

gdb-heap

The newer gdb (gdb 7 and above) is supporting memory debugging.
Quoting from http://fedoraproject.org/wiki/Features/MemoryDebuggingTools

"The new "gdb-heap" package adds a new "heap" command to /usr/bin/gdb.

The command allows you to get a breakdown of how that process is using
dynamic memory.

It allows for unplanned memory usage debugging: if a process
unexpectedly starts using large amounts of memory you can attach to it
with gdb, and use the heap command to figure out where the memory is
going. You should also be able to use it on core dumps.

We believe this approach is entirely new, and is unique to Fedora 14. "

Sounds promising!