Nothing technical about it

Linux early boot debug

2012-02-09T14:56:00.001-08:00

As a part of my project at NCSU (North Carolina State University), I am trying to setup the development environment for a FreeScale Board (i.MX53QSB). This includes the ability to build a custom kernel. However, when I tried to cross-compile the kernel and load it on the board, the console sadly hung at

Uncompressing Linux... done, booting the kernel

Armed with my previous experience at the u-boot/kernel interface, I made sure that the architecture number in uboot and kernel matched. At this stage the kernel is in its very nascent stages of booting. No drivers have been initialized, including the in-kernel UART drivers. This means no kgdb. printk and such debugging interfaces do not work at such early stages of booting. JTAG would probably work, but my exposure to JTAG is limited. Nonetheless, the kernel being as awesome as it is, there is an excellent way to debug at even such an early stage: The Kernel Low-Level debug facilities.

In brief, this particular (and many other boards too) board comes with a debug UART. UART1 is set in debug mode by uboot. Enabling CONFIG_DEBUG_LL in the kernel will link in some functions that will enable early code to setup and use this debug UART. The logs can then be viewed easily on the host machine.

Following are the steps to enable low-level debug:

1) make ARCH=arm menuconfig

2) Go to "Kernel Hacking". Enable "Kernel Debugging". This will unhide lots of options, including "Kernel low-level Debugging functions". Enabling that will unhide "Early printk". Enable "Early printk" too.

After a full recompile, append "earlyprintk" to the bootargs in the uboot environment (Usually it is the ${bootargs} variable). Thats it!

Now, booting the kernel shows a step-by-step log of what the kernel is doing. In my case, I found out that there was an unhandled alignment trap that led to a kernel crash. The fix was to add "-mno-unaligned-access" to CFLAGS while kernel compilation. After solving a couple of issues, now I am stuck here.

Uncompressing Linux... done, booting the kernel.
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 218112
Kernel command line: console=ttymxc0,115200 earlyprintk root=/dev/mmcblk0p1 rw rootwait
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 352MB 512MB = 864MB total
Memory: 871240k/871240k available, 13496k reserved, 0K highmem
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
DMA : 0xf4600000 - 0xffe00000 ( 184 MB)
vmalloc : 0xe0800000 - 0xf2000000 ( 280 MB)
lowmem : 0x80000000 - 0xe0000000 (1536 MB)
pkmap : 0x7fe00000 - 0x80000000 ( 2 MB)
modules : 0x7f000000 - 0x7fe00000 ( 14 MB)
.init : 0x80008000 - 0x80030000 ( 160 kB)
.text : 0x80030000 - 0x804ebc68 (4848 kB)
.data : 0x804ec000 - 0x80539ca0 ( 312 kB)
SLUB: Genslabs=11, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:848
TrustZone Interrupt Controller (TZIC) initialized
MXC GPIO hardware
imx-uart.0: ttymxc0 at MMIO 0x53fbc000 (irq = 31) is a IMX
�枞��`>�&��&�8�&��8��>枞��8��&

As soon the kernel UART drivers kick in, my console starts spitting out garbage. This should be an easy fix now.

/me goes back to poring over the UART driver

Plotting a dependency graph

2011-09-23T10:25:00.001-07:00

While working with building loads of packages, it is often necessary to visualise dependencies between them. Two years ago I suffered a lot trying to do just that and at best I came up with a python script that gives a textual list of deps. Today, I found an awesome tool to plot dependency graphs. Although useless for most of the people, it is quite a nifty trick for the few who work with thousands of packages.

Two RPMs need to be installed before anything else

# yum install rpmorphan graphviz

Now, assume you need all the deps for rebuilding "rpm" (The actual RPM rpm)
First, generate a textual dep graph using:

# rpmdep -dot rpm.dot rpm

The "dot" is a file-format which will be used by the "dot" program to plot a graph. To plot the graph

# dot -Tsvg rpm.dot -o rpm.svg

The command takes rpm.dot and plots the graph in an svg format.

Simple!!

A sample image I generated is here:

vim: How to setup awesome autocomplete

2011-08-09T20:45:00.001-07:00

Recently I moved to a company which has a huge codebase and loves long names! All vim/emacs users will sympathize with such a situation. Eclipse would autocomplete names in a jiffy. I was sorely missing good autocomplete on my favourite editor: vim.

Typically by setting the "dict" variable appropriately, one can autocomplete english words, but what about C/C++ code?
One fine morning, I had an epiphany. Why don't people use the ctags/etags database to autocomplete. The database already has lots of semantic information about the code and it would have all the project specific keywords too. Now, I am not a guy who has too many epiphanies, so I was sure someone had implemented it even before I could spell "vim". With this in mind, I googled the specific terms "vim autocomplete using ctags". The first stackoverflow result had a brilliant suggestion: omniautocomplete.

Before I go into what omniautocomplete (a plug-in for vim) does, let me first list down my requirements from an autocomplete tool
1) Autocomplete sensibly: Use the knowledge that I am editing "c++" code and give sensible suggestions.
2) It should *not* be enabled by default when I run vim i.e. do not blindly putt crao in my .vim. The reason is that I work on multiple projects at the same time and I don't want suggestions from one project to be mixed with other.
3) No lag in autocompleting.
4) Provide a quick way to update the autocomplete "database".

Some vim scripting + omnicomplete provides all that and more. What more, you might ask. Well, it provides an eclipse-like pop-up when asked to autocomplete. It can also fire when I enter a "." or "->". It also has an option to show "function prototypes" alongwith autocomplete suggestions. This is awesome!!!

To get you excited, here is a screenshot. It looks exactly the same.

Setting up omnicomplete is super-easy. Installation instructions are outlined here. All it involves is unzipping a zip file to ~/.vim

The next part is more interesting:
Since ctags database is used, ctags needs to be compiled with some specific options. The commandline I picked up from their documentation is this:

$ ctags -R --c++-kinds=+p --fields=+iaS --extra=+q .

Now, the "tags" variable in vim needs to point to the tags of your project. Setting this up each time you open vim can be irritating. Very irritating. So, I wrote the following vim function to alleviate the issue:

function! SetProj(proj)
        if a:proj == "mythread"
                map <C-F12> :!ctags -R --c++-kinds=+p --fields=+iaS --extra=+q -f /home/jitesh/repos/mythread_lib/tags /home/jitesh/repos/mythread_lib<CR>
                set tags+=/home/jitesh/repos/mythread_lib/tags
                let OmniCpp_ShowPrototypeInAbbr = 1
                let OmniCpp_MayCompleteDot = 1
        endif
endfunction

Now, when I open vim, all I have to do is:

:call SetProj("mythread")

Note - Re-generating the ctags database is also pretty easy, Just press Ctrl+F12. Notice the keymapping in the vim function above.
So, when I define a new function in my code, I just do a Ctrl+F12 and the tags are updated and ready to be auto-completed.

Pressing ctrl+P or ctrl+N invokes the autocompletion.

When I start vim, none of this stuff is loaded, so there is no load-time penalty/bloat + I can work on multiple projects just by calling "SetProj" with the appropriate parameter.

I sign-off a happy user! vim FTW

Block processing and shell-scripts

2011-04-22T13:26:00.001-07:00

Feature 1:
Shell-scripts can easily process files/records line-by-line.
A typical construct to do so is:

ps aux |
while read line do;
        echo $line
done

But, this kind of a thing isn't very easy to process outputs of
commands like ifconfig. ifconfig has output in blocks instead of
lines. Recently I had to do it and I was really surprised that I had
no idea of how to parse blocks of data in shell-script. After some
time, I figured it out myself. The pipe command pipes the output to a
shell. So what if that shell contains another shell-script instead of
a single command?
Here's how to parse ifconfig command. The script prints out a list of
interfaces and corresponding IP address (NIL if no IP): (Note:
Brackets fork a new shell)


ifconfig -a |
(
#Read the first line
while read line; do
    INTERFACE_ADDR=`echo $line | awk '{print $1}'`

    # Read second line in the block and store the IP address, if any
    read line;
    IP=`echo $line | grep "inet addr" | awk '{print $2}' | cut -d':' -f2`

    while true; do
        # Skip all other lines from the block
        read line;

        # End of block is a new line. Start parsing new block then
        if [ -z "$line" ]; then
            break;
        fi
    done
    echo $INTERFACE_ADDR $IP
done
)

There. This is albeit a simple example. You can construct truly
awesome block parsing constructs using the sub-shell (i.e. the
brackets '()' )

Feature 2:
Wonder why a shell-script doesn't have a block commenting feature?
Wonder no more.
Here is a cool trick using here-documents to comment out a block of shell script

: <<'END1'
"The commented block goes here"
END1

Notes:
1) ":" means "true". /dev/null equivalent for a shell script.
2) Make sure that the string "END1" is the same at both the places,
else, this thing is not going to work.
Happy scripting :-)

Jajuk

2010-11-30T12:14:00.001-08:00

Writing a music player for the lazy is a overwhelmingly tough job to do. The lazy people expect all kinds of automatic stuff. Play by Genre, ambience, mood, ratings. Play with untagged songs nicely instead of dumping them under one "Unknown" Label. Automatic tagging of songs whenever possible. Automatically fetch Lyrics, cover, wiki information. Ability to detect which songs the user doesn't like and play them lesser and lesser. File-tree view, album-tree view, artist-tree view. Automatic purging of duplicates. Music discovery via similar artists or people-who-liked-this-also-liked-this. Internet Radio for the ones who are bored of their collection. Some ego-boosting features for music junkies like all kinds of graphs and statistics of your collection and the Genre you like the most.

Overall, I have wayy too many expectations out of a music player. It is more like a music manager for me. Till date I hadn't gotten over my first love Amarok-1.4 completely. I didn't like the direction Amarok-1.9 and 2.0 took, so I dumpled Amarok long ago. iTunes and all its Linux clones are disappointing altogether too, so do NOT mention them. Finally today I came across this Java-based player called "Jajuk". I have to say it the most I liked since Amarok-1.4. It has brilliant features, amazingly intuitive interface which somehow hides all the complexity behind the complex features it provides. There is just an endless list of features and they actually work! It is my new default music player.

There are some catches though ->
1) Being Java-based it is heavy. Once I got used to it though, it is not a problem.
2) Global hotkeys are not supported. Now this is a *must* have. I have filed a feature request. Lets see whether it gets footage :-D.

Give Jajuk a try. It is good.
Find it here: http://jajuk.info/index.php/Main_Page

Traffic Shaping

2010-11-22T22:23:00.001-08:00

So today was a rather weird day and I had a searing pain in my head. So I decided on a special masochistic thing to learn : traffic shaping. Since I am working on a project that needs this exclusively, I reckoned it will be helpful too.

I am sure most of you must have come across this term. In case you haven't, the wiki page here provides a good introduction. Traffic shaping is mostly used by ISPs and people who manage routers. It is a way to give priority to certain users or certain types of data. The term "Giving priority" hugely underestimates the complexity of the task. eg. Real time data streaming may value real-time data delivery whereas others might tolerate some delay. In general, when traffic shaping goes near QoS, it becomes a difficult problem to solve.

However, here I am going to rant about the ISP aspect of traffic shaping (i.e. allocating bandwidth per user, per service, etc). "tc" is a very powerful command in Linux to implement traffic shaping. I won't be giving sample commands or such because tc is quite complex. Instead (if anyone ever ends up using this post, which I strongly suspect is going to be my future self) I will give out links to tutorials I used for my setup.

Some introduction to tc is in line. With tc you can create queues per device. Multiple queues. You can choose which queueing discipline you are going to use. Queueing discipline says how the queue is managed. You can have a standard FIFO queue or a Token-Bucket filter (neatly manages bandwidth difference between two lines) or Stochastic Fair Queueing (ensures fairness). Queues can be defined as hierarchy and they can inherit each other's properties. Eg. I might define a queue for www traffic which takes up 40% of the bandwidth and 60% of everything else. Now, I can define another queue for ssh which is a child of the "other" queue (it basically means that the ssh queue can borrow bandwidth from its parent and go beyond its limit to occupy upto 60% of the bandwidth. This way you guarantee minimum service). On routers, each queue can represent a customer or a chunk of customers. High paying customers get more share of the bandwidth. After we are done defining this classes of traffic, next step is to define rules for saying which traffic belongs to which class. tc provides a wide variety of filters for the purpose. You can look at almost any of the TCP or IP headers, which interface the packet comes from, etc. For all purposes, this set of filters proves to be sufficient.

Now that we know how tc works, lets see what all tc can do:
1) Provide only a defined amount of bandwidth to a particular user (write a filter based on IP address)
2) Provide a defined amount of bandwidth to a particular service (write a filter based on port number)
3) Enable flexibility. i.e. if there is a burst of traffic in one class accommodate the burst instead of providing hard boundaries. (burst and cburst parameters control that)
4) Rates can be specified in percentage of total as well as actual absolute values like 8kbits, etc.

An addition to tc can do more wonderful stuff. It is called "netem" which stands for network emulator. netem has tunable knobs for any parameter you can conceive which can help you simulate a WAN. Here is what we tried today:
1) Simulate delays. If you are simulating a WAN at home, the major issue is that delays don't get simulated. With tc combined with netem, you can introduce fake delays. You can even vary delays about a point. eg. You can say probabilistically change delay at 100ms +- 10ms. You can even change the probability distribution by which it randomises the delay.
2) Simulate losses. When in WAN loses are inevitable. Either due to network congestion or corruption. netem with tc can simulate both. You can simulate percentage losses, losses in bursts, losses following a specific probability distribution, losses following a particular pattern. Anything under the sun.
3) Packet duplication, packet corruption, packet reordering.
4) Introducing a latency is only a defined type of traffic (to test QoS)

Anyone who knows a little about networks realises how difficult each one of this is to implement. Tuning all these knobs appropriately can give you an awesome simulation of the internet. I am working on a project which needs me to simulate network congestion at home. With a line-speed router and a speed of 100mbps, it is really tough to simulate congestion. We used tc command exclusively (with a lot of actual studies which has derived the loss and corruption values) to successfully simulate the internet at home. We now a get a very good TCP congestion window graph. The things that tc can do are pretty amazing. Unfortunately tc is a complex command and needs a lot of knowledge first to start using it. Fret not. Here are links to some tutorials that can get you going:

The classic TLDP tutorial
HTB tutorial : Contains some excellent explanations and practical examples with commands.
netem Introduction

Learning tc is a process and the end result is pretty satisfying. \m/ to the author of tc.

gdb-heap

2010-10-26T18:31:00.001-07:00

The newer gdb (gdb 7 and above) is supporting memory debugging.
Quoting from http://fedoraproject.org/wiki/Features/MemoryDebuggingTools

"The new "gdb-heap" package adds a new "heap" command to /usr/bin/gdb.

The command allows you to get a breakdown of how that process is using
dynamic memory.

It allows for unplanned memory usage debugging: if a process
unexpectedly starts using large amounts of memory you can attach to it
with gdb, and use the heap command to figure out where the memory is
going. You should also be able to use it on core dumps.

We believe this approach is entirely new, and is unique to Fedora 14. "

Sounds promising!

Writing cache efficient programs

2010-10-02T01:31:00.000-07:00

Just recently I discovered how much effect cache misses can have on the running time of your program! Its truly amazing how much savings you can make, yet I had ignored it for more than 4 years now.

Consider the matrix multiplication program again


for (i = 0; i < MAX_NUM; i++) {
     for (j = 0; j < MAX_NUM; j++) {
          for (k = 0; k < MAX_NUM; k++)
               Y[i][j] += A[i][k] * B[k][j];
      }
}

Note that when you have a cache miss (depending on your cache properties), you'll fetch in 32 bytes of consecutive data from main memory. Now see the B[][] array in the program. We are accessing it column-wise. Now, once we access B[0][0], we fetch in B[0][0] tp B[0][7]. But, we only end-up using B[0][0]. Only 12.5% cache usage. REALLY bad!

When MAX_NUM = 1000, this code takes ~6.778ms to run.

Now, how can we optimise this for good cache usage. One method is to transpose B[][] matrix and use the transpose in the multiplication. (Transpose = Interchange rows and columns). This way, we'll access B[][] by its rows and improve cache usage.

New code will look something like this (For the moment, ignore the time taken to transpose the matrix. We'll come to it later):


for (i = 0; i < MAX_NUM; i++) {
     for (j = 0; j < MAX_NUM; j++) {
          for (k = 0; k < MAX_NUM; k++)
                Y[i][j] += A[i][k] * B[j][k];
      }
}

Notice how we access B[][] row-wise now. Our cache-efficient version takes 5.629ms for a 1000 dimension array.
So how much did we save? We saved 16.9% of execution time!! Isn't that cool?
17% savings is too much to ignore!!

Coming back to transpose of the matrix. There are multiple solutions to this problem:
1) Store it as the transpose right from the start. Thus, you don't have to do that operation before a multiplication. Neat solution.
2) There are cache-efficient ways to transpose a matrix which take less than 0.15ms for a 1000-element array. You still end-up with ~12% savings! I'll try to write a post on that someday too.

Ofcourse this is a very crude solution to use cache efficiently and can be optimised further to get even lower execution times. There is one called "Blocking factor" which I am still learning. I never thought so much about cache hits/misses when writing a program. For 17% savings, I certainly should!

Combined with my earlier post on parallelizing a program, you could effectively have double the cache size (assuming dual core) and hence more the savings!

Geek Saturday (Rants)

2010-09-25T14:19:00.001-07:00

Fixed three irritating problems on my Fedora-13 laptop.

1) When fading, mouse/keyboard input doesn't stop fading and you have to go enter the password to unlock the screen.
https://bugzilla.redhat.com/show_bug.cgi?id=612620
The latest xorg-x11-server update fixed the problem. Simple enough!

2) Inserting headphones in the audio jack doesn't mute the speakers.
https://bugzilla.redhat.com/show_bug.cgi?id=623632
After some grepping in /var/log/messages and google, found out that my kernel couldn't detect the model of my audio card. Simple fix was to add
<code>
options snd_hda_intel model=quanta
</code>
to /etc/modprobe.conf/dist-alsa.conf to force snd_hda_intel module to use the given model.

3) Flash videos don't work in fullscreen.
https://bugzilla.redhat.com/show_bug.cgi?id=608957
Simple fix was this:
<code>
mkdir -p /etc/adobe
echo "OverrideGPUValidation=1" > /etc/adobe/mms.cfg
</code>

There is just one irritating problem now. When I shutdown Fedora with windows open on various desktops, Fedora should remember the desktop number the window was open on. However, when I start back, all windows come back to desktop 1. hmm Couldn't figure out a solution to that yet :-(. But it was low priority anyway. All the major problems are now resolved :-). Peace!

On paranoia

2010-09-24T17:52:00.001-07:00

This graduation thing is making me a paranoid. First I semantically obfuscated my email addresses everywhere and now I just turned off cookies on my browser after reading this: http://www.cookiecentral.com/. gmail and most sites which need some sort of a user identification ofcourse use cookies so you have to go selectively allow cookies for certain sites. But after reading how doubleclick.net uses cookies, I really don't mind the extra effort.

After my MS, I will probably be such a security paranoid as to even block ssh connections to my machine as Linus does :-(. Ignorance is such a bliss!

Writing parallel programs with C and gcc

2010-09-15T17:18:00.000-07:00

I just found a cool gcc feature. gcc implements the OpenMP specification. OpenMP specification details how to write shared memory parallel programs in various languages. This got me interested and I took the best parallelisable example : The matrix multiplication. Matrix multiplication is an O(n^3) algorithm. The loop structure of a matrix multiplication program look like this:


for (i = 0; i < MAX_NUM; i++) {
     for (j = 0; j < MAX_NUM; j++) {
          for (k = 0; k < MAX_NUM; k++) 
               Y[i][j] += A[i][k] * B[k][j];
      }
}

The i and j loops are parallel. This means that given a unique (i, j) combination, the computations in the loop body do not depend on any previously computed values for multiplication i.e. each matrix element can be computed independently.

Lets get out hand dirty and make the i loop parallel. I have chosen size of matrix to be 1000X1000 so that we can see the speed-up clearly.

Now, all we have to do is, add one #pragma line before the i loop. Here is what the parallel code will look like


#pragma omp parallel for default(shared) private(i, j, k)
for (i = 0; i < MAX_NUM; i++) {
     for (j = 0; j < MAX_NUM; j++) {
          for (k = 0; k < MAX_NUM; k++) 
               Y[i][j] += A[i][k] * B[k][j];
      }
}

Don't worry about the #pragma statement. These are actually quite simple to figure out and are a 5 minute job to learn from here: http://www.openmp.org/mp-documents/cspec20.pdf

Note that we haven't specified how many parallel tasks to fork. This is specified via an environment variable OMP_NUM_THREADS. This allows great flexibility in specifying how many threads to use without recompiling the program.

To enable OpenMP spec, use the "-fopenmp" switch with gcc:


$ gcc -o parallel -fopenmp parallel.c

Here are some test-runs which clearly show the speed-up:
Only one thread (serial program)


$ export OMP_NUM_THREADS=1
$ time ./parallel

real    0m13.633s
user    0m13.558s
sys    0m0.028s

Two threads


$ export OMP_NUM_THREADS=2
$ time ./parallel

real    0m8.144s
user    0m13.963s
sys    0m0.035s

Four threads


$ export OMP_NUM_THREADS=4
$ time ./parallel

real    0m7.960s
user    0m13.815s
sys    0m0.037s

Notice the speed-up?
With one thread, we have a execution time of 13.6s. With two threads it becomes 8.144s which is almost half. And with four threads it is 7.9s. Now what happened in the case of 4 threads? Didn't we expect 1/4th time? Well, mine is only a dual core PC, so the max speed-up for a CPU intensive task is when number of parallel tasks = number of cores/threads = 2.

You can see the running parallel threads with


$ ps -Haux | grep parallel
jitesh    7240 26.0  0.0  44536  1164 pts/0    Rl+  20:10   0:00 ./parallel
jitesh    7240 25.5  0.0  44536  1164 pts/0    Rl+  20:10   0:00 ./parallel
jitesh    7240 27.5  0.0  44536  1164 pts/0    Rl+  20:10   0:00 ./parallel
jitesh    7240 27.5  0.0  44536  1164 pts/0    Rl+  20:10   0:00 ./parallel

The "-H" switch displays the threads.

Cool, isn't it?
Thus, a parallel program with minimum effort and LOTS of time savings. We reduced the execution time of matrix multiplication to half with proper parallelism. On a 16-core machine with native support for threads, imagine how parallel programs will perform!

gcc and OpenMP FTW!

Note: If you are interested in learning OpenMP, see the spec links I have posted in the body of the blog.

A note on email address obfuscation

2010-09-06T08:22:00.001-07:00

Email address obfuscation:
When you put your email address on the internet (eg.
jitesh@example.com), there is a chance that your address is mined for
spamming. Spam-bots crawl the internet day-and-night to mine email
addresses for spamming. Now, you can do something to prevent a spam
bot reading your email address. The key factor here is that an
automated BOT is going to scan the pages, so it is easy to fool it. Or
is it?

A very common form of obfuscation is to spell out the special
characters. eg. jitesh AT example DOT com. Note that this is infact a
very weak form of obfuscation. Because, you are simply changing the
syntax of writing your email address and hence, the bot-writer has to
just add one more grammar to his list of rules and he is done.
Ofcourse, bot-writers are not so stupid as to miss this simple change.

What you need is a semantic change which is impossible (or extremely
hard to be practical) for the bots to infer. But, humans can do it
easily. eg. I might obfuscate my address as : my-first-name AT example
DOT com
Note that: "my-first-name" here is a semantic change. Only a human can
infer that this is to be replaced by "jitesh". This is REAL
obfuscation.

So guys, if you have a non-gmail account with a sucky spam filter,
make your obfuscation stronger and do NOT underestimate spammers :-)

Loving LaTeX

2010-09-01T23:01:00.000-07:00

I've been learning LaTeX for a while now and it is *awesome* (For the uninitiated LaTeX is a language used for typesetting. You can print all sorts of mathematical and scientific symbols using LaTeX). Here is the result of experiments. Pretty neat eh?

\m/

2010-09-01T19:05:00.001-07:00

From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroups: comp.os.minix
Subject: Free minix-like kernel sources for 386-AT
Message-ID: <1991Oct5.054106.4647@klaava.Helsinki.FI>
Date: 5 Oct 91 05:41:06 GMT
Organization: University of Helsinki

Do  you pine for the nice days of minix-1.1, when men were men  and wrote  their own device drivers? Are you without a nice project and  just dying  to cut your teeth on a OS you can try to modify for your  needs? Are  you finding it frustrating when everything works on minix? No  more  all-nighters to get a nifty program working? Then this post might  be  just for you

As  I mentioned a month(?) ago, I'm working on a free version of  a  minix-lookalike for AT-386 computers. It has finally reached the  stage  where it's even usable (though may not be depending on what you  want),  and I am willing to put out the sources for wider distribution.

It  is just version 0.02 (+1 (very small) patch already), but I've   successfully run bash/gcc/gnu-make/gnu-sed/compress etc under it.

Completing the Text Processing Arsenal

2010-05-08T10:18:00.001-07:00

For months and months, I have been procrastinating to learn the last tool in my text processing arsenal: awk. After a particularly uneventful afternoon, I decided today was the day. Sleeves rolled-up, chips by the table, all sources of interference disposed off, I sat down and finally put the last piece of the jigsaw in place. Today I rant about my text processing arsenal.

Here is a list of text processing commands/scripts available at your disposal in Linux and how you can combine them to serve your purpose.

1) The very basics: echo, cat, less (Useful to look through large logs or files that don't fit into one screen), more (less is really better), head (Prints first few lines of a file. Can be a very good help in scripts), tail (with -f switch, can be useful in watching logs real-time as they are generated)

2) Some more basics: grep. grep is oxygen. I use it all the time! I suggest you really go through that linked tutorial and thoroughly learn grep. It has some nice tools in its toolbox (like umm, the -v switch which inverts the regular expression, etc). I have also written a tutorial on grep earlier.

3) sort, uniq: I find both of them terribly useful for handling lists and looping on the same action for a list.
eg.
<code>
# cat package_list | grep "^java-*" | sort | uniq | xargs -n 1 build_package.sh
</code>
This command takes a list of packages, searches for all java packages, eliminated duplicates and schedules a build for each,

4) All the above commands mainly encompass reading or searching through already available data. What about removing/selecting only parts of sentences or translating one character to another? (like small-case characters to upper case, etc). Not to worry. cut, tr to the rescue. Cut can remove parts of a line (by defining delimiters and selecting the part) and I use tr mostly to convert lower to upper case or squeeze whitespace. More on that here.

5) Getting your hands really dirty. All of the above commands really enhance your ability to mine data from a file. grep-ping interesting lines using grep, selecting only part of them using cut, making it presentable using tr, etc etc. What about actual editing in shell-scripts? Something like find and replace? Well, *lightning and thunder* sed is here! sed (which stands for stream editor) has saved me a lot of time. The very motif of sed is search and replace. And if offers VERY advanced searching patterns. You can specify regular expressions, search only a part of a file, replace first or all occurrences and a lot more. A VERY awesome tutorial is here. After grep, sed is the most useful tool I know.
eg. Converting fs_get to xfs_get in the entire source was never easier than this:
<code>
# find . -name "*.[cChH]" | xargs sed -i 's/fs_get/xfs_get/g`
</code>
Done!

6) All this is fine. But what about the advanced data mining and report generation tasks? awk is here. awk can process tables and columns of data. I majorly use awk to easily select and reformat text.
eg. If you observe your /var/log/messages file, the 5th field is always the process name. Suppose I wanted to find out how many times ntpd synced to the latest time. I can run
<code>
# cat /var/log/messages | awk '$5 ~ /^ntpd/' | grep "kernel time sync" | wc -l
</code>
Don't be intimidated by the awk syntax, its simple really. Just invest 10 mins is reading this tutorial here.

7) Lastly, the most insignificant command that I have never used in my life: join. Does exactly what a database join does.

So, this is my text processing arsenal. With echo, cat, grep, sed and awk I could also write a mini-database which will have the worst performance ever, but will work just fine nevertheless. Insert, Delete, Edit, Query, Join can all be implemented using just these commands. Among them, they cover almost all the text processing requirements.

Finally, I would like to add some more commands that I rarely use or are in my wishlist or can be useful for my readers.

8) dos2unix, unix2dos: DOS likes to write newline as "\r\n". UNIX prefers "\n". This can lead to really interesting issues in Makefiles, etc when files are written on Windows and executed on Linux. Above 2 utilities are used to convert from one format to another. iconv is a generic converter from hundreds to hundreds of formats.

9) gettext: This is in my wishlist, but I don't see myself learning to use gettext anywhere in the near future. Gettext is used to localize a program to your own language or a foreign language.

Thats all for now folks. Comments/Feedback appreciated.

MP3 tagger

2010-02-19T23:48:00.001-08:00

After trying a host of automatic tagger tools : EasyTag, Kid3, pytagger, id3v2, iTunes itself, etc etc, I've found the perfect auto-tagging tool.

Picard by MusicBrainz. And the awesomeness is that it is GPL :-)

There is MusicBrainz integration available with Amarok, iTunes, rhythmbox, banshee, etc.. but none of the tools has as awesome and simple interface as Picard! The other tools, namely, easytag, kid3, pytagger and id3v2 use the CDDB database blindly. CDDB doesn't allow queries by track name. Only album name and artist. That is an inconvenience. Also, you cannot cluster multiple mp3 files together into one and query the database. Thus, each file has to be queried manually. More inconvenience.
Picard to the rescue! It has all those niceties + the awesome (open and free to download) MusicBrainz database to back it up and a very very intuitive and simple UI!

Hail Picard!

UBI / UBIFS on NANDSIM simulator

2010-01-19T21:43:00.000-08:00

NAND simulator (NANDSIM) is an extremely useful debugging and development tool which simulates NAND flashes in RAM or a file.NANDSIM can emulate various errors and report wear statistics, which is extremely useful when testing how flash software handles errors.

Steps involved in the working of NANDSIM :

1 ) We work as a super user.
aruna@narsil:~$ sudo su
[sudo] password for aruna:

2) Now we mount the NANDSIM module.That is,we create a virtual raw flash device.
The parameters are as follows :
first_id_byte : The first byte returned by NAND Flash 'read ID' command (manufacturer ID)

second_id_byte : The second byte returned by NAND Flash 'read ID' command (chip ID).The entire table of chip ids is given in the file nand_ids.c in the source code given here.

third_id_byte and fourth_id_byte are optional parameters which are initialized to 0xff (empty) by the system if not specified by the user.They are the third and fourth ID returned by the READ ID command.

By default NANDSIM uses RAM but if you do not have enough RAM, you can make it emulate the flash on top of a file using the cache_file nandsim module parameter.

We create a 256MiB emulated NAND flash with 2KiB NAND page size :

root@narsil:/home/aruna# modprobe nandsim first_id_byte=0x20 second_id_byte=0xaa third_id_byte=0x00 fourth_id_byte=0x15

root@narsil:/home/aruna# cat /proc/mtd
dev: size erasesize name
mtd0: 10000000 00020000 "NAND simulator partition 0"

3)MTD is not LDM-enabled and udev does not create device MTD device nodes automatically, so create /dev/mtd0

root@narsil:/home/aruna# mknod /dev/mtd0 c 90 0

4) Now we attach UBI to our MTD device. (This can also be done by the ubiattach utility)

root@narsil:/home/aruna# modprobe ubi mtd=0

5)We create a volume on our newly mounted UBI.

root@narsil:/home/aruna# ubimkvol /dev/ubi0 -N myvolume -s 200MiB

Here we have ,
Volume ID 0, size 1626 LEBs (209793024 bytes, 200.1 MiB), LEB size 129024 bytes (126.0 KiB), dynamic, name "myvolume", alignment 1

root@narsil:/home/aruna# cat /proc/mtd
dev: size erasesize name
mtd0: 10000000 00020000 "NAND simulator partition 0"
mtd1: 0c813000 0001f800 "myvolume"
Here we note that the erasesize is the size of the physical eraseblock on mtd0 (raw flash) which is 128Kb but the size on mtd1 is the size of the logical eraseblock.For each PEB,space is required to store 2 headers - one to keep track of the erase counters and the other for the logical-physical mapping.This space is not included in the LEB.Hence the size of LEB is 126Kb.

6)We mount ubifs on our UBI volume.And copy a file to it.

root@narsil:/home/aruna# mkdir /mnt/ubifs
root@narsil:/home/aruna# mount -t ubifs ubi0:myvolume /mnt/ubifs
root@narsil:/home/aruna# cp /home/aruna/5.l /mnt/ubifs

7) Now we wish to create an image of our data so that we need not have to mount an empty device the next time we use our flash.

This can be done in two steps : Firstly we use a tool called mkfs.ubifs which creates a ubifs image of our data.We can stop here and then for next use we again create an mtd device , mount ubi on it,create a ubi volume and then use this image to recover our data.

The second step is to use the ubinize tool on the image created by the mkfs.ubifs tool.This creates ubi image (which includes data about ubi volumes ).In this case,on the next mount ,we create an mtd device and then use the image created
by ubinize.After this we mount ubi and ubifs. (This is a better method as we do not have to mount the empty ubi every time.)

Both,mkfs.ubifs and ubinize are mtd-utils.Hence we have to sort out the dependencies before using them.

8) In Fedora install zlib-devel, lzo-devel, and e2fsprogs-devel packages and in Debian install zlib1g-dev, liblzo2-dev and uuid-dev packages.

9) The git repository of mtd-utils is available at git://git.infradead.org/mtd-utils.git
We use :
root@narsil:/home/aruna#git clone git://git.infradead.org/mtd-utils.git

10) The clone command creates a new directory named mtd-utils.We cd into this directory.Here we find another directory called ubi-utils.In this directory,

root@narsil:/home/aruna/mtd-utils/ubi-utils# make install

This is to be done before step 11 else lubi library is not available for mkfs.ubifs

11)Now back in mtd-utils,

root@narsil:/home/aruna/mtd-utils#make install

12) Now that our utilities are properly installed , we create the ubifs image first.In the folder mkfs.ubifs :

root@narsil:/home/aruna/mtd-utils/mkfs.ubifs# ./mkfs.ubifs -r /mnt/ubifs -m 2048 -e 129024 -c 2047 -o ubifs.img

where,
/mnt/ubifs is data whose image is to be created

-m 2048 indicates that the minimum input/output unit size of the flash this UBIFS image is created for is 2048 bytes (NAND page in this case)
-e 129024 is the logical eraseblock size of the UBI volume this image is created for
-c 2047 specifies maximum file-system size in logical eraseblocks; this means that it will be possible to use the resulting file-system on volumes up to this size (less or equivalent) so in this particular case, the resulting FS may be put on volumes up to about 251MiB (129024 multiplied by 2047)

13) Copy the created file ubifs.img to the ubi-utils folder.Then create another file called ubinize.cfg

root@narsil:/home/aruna/mtd-utils/ubi-utils# cat ubinize.cfg
[ubifs]
mode=ubi
image=ubifs.img
vol_id=0
vol_size=200MiB
vol_type=dynamic
vol_name=myvolume
vol_flags=autoresize

Now we create the ubi image :

root@narsil:/home/aruna/mtd-utils/ubi-utils# ubinize -o ubi.img -m 2048 -p 128KiB -s 512 ubinize.cfg

where
-p 128KiB is the physical eraseblock size of the flash chip the UBI image is created for is 128KiB (128 * 1024 bytes)
-s 512 indicates that the flash supports sub-pages and sub-page size is 512 bytes

Now we have the ubi.img image saved on persistent storage.

14) We can unmount the file system and our virtual device.

15) The next time we need to use the device ,we follow only these steps :

a)Create the device

aruna@narsil:~$ sudo su
[sudo] password for aruna:
root@narsil:/home/aruna# modprobe nandsim first_id_byte=0x20 second_id_byte=0xaa third_id_byte=0x00 fourth_id_byte=0x15
root@narsil:/home/aruna# mknod /dev/mtd0 c 90 0

b) Load image to it :

root@narsil:/home/aruna/mtd-utils/ubi-utils# dd if=ubi.img of=/dev/mtd0 bs=20481024+0 records in
1024+0 records out
2097152 bytes (2.1 MB) copied, 0.0220489 s, 95.1 MB/s

c)Mount UBI to it :

root@narsil:/home/aruna/mtd-utils/ubi-utils# modprobe ubi mtd=0
root@narsil:/home/aruna/mtd-utils/ubi-utils# cat /proc/mtd
dev: size erasesize name
mtd0: 10000000 00020000 "NAND simulator partition 0"
mtd1: 0f90c000 0001f800 "myvolume"

d)Mount UBIFS to our volume :

root@narsil:/home/aruna/mtd-utils/ubi-utils# mount -t ubifs ubi0:myvolume /mnt/ubifs

We can check the file we pasted here before creating the image by:

root@narsil:/home/aruna/mtd-utils/ubi-utils# ls /mnt/ubifs
5.l

Happy reading ! :)

git-perforce interface

2010-01-19T05:28:00.001-08:00

The best thing I found out and learned today was the git interface for perforce repositories. There are many things about perforce that made me cringe.
1) No support for local commits. Limited support is provided by distinct changelists, but it only works if a non-intersecting set of files are being modified in both commits. It is rarely the case. This usually means you cannot commit small logical chunks of code and sometimes have to submit hugeeeee code-changes (which is not recommended)
2) No rollbacks or editing commits.
3) Limited branching support (Like I don't have permission to branch on the repository. But, with git I can have infinite local branches)
4) Send out patches that others can apply. (Perforce patches cannot be applied)
5) Manually check-out each file I am going to edit. This is a major pain.

So, I set out looking for some kind of git-perforce inter-conversion software and found that git itself provides the required scripts. My internet is screwed, so I cannot provide a full link as of now. but, you'd find the git's git repository at git-scm.com. Goto contrib/fast-import. Under that where you'll find both the documentation and the script. Unfortunately, fedora decided not to package it.

With that in place, I was able to use my perforce repository as a git repository. Commit local changes. Edit commits. Send out patches for each commit separately and all the cool stuff git has to offer. (Note that perforce repository can be seen as remote branch p4/master) git-p4 assigns a changelist id to each commit, thus mapping perforce commits to git commits. At last, just call "git-p4 rebase" (to accept latest changes) and then "git-p4 submit" to submit all changes to the perforce repository.

Another hassle I got rid of was to manually check-out every file I want to edit in perforce. With git, its not necessary.

I sign-off as a happy user!

Update:
Here is the link I was talking about.
http://repo.or.cz/w/git.git/tree/ff6d26a0e1d8fad775010fa3b689c0c027da8bb0:/contrib/fast-import

The FOSS.in experience

2009-12-05T23:11:00.000-08:00

As some of you know (possibly because I have been raving about it so much) this year's FOSS.in was held at Bangalore from 1st to 5th of December. I attended the conference on 2nd and 3rd of December. My plan was to attend all the 5 days, but some inevitabilities at work made it impossible.

The trip started on a very good note as I boarded the Kingfisher flight from Lohagaon on 2nd of December. For the entire duration of the flight, I was kept entertained by the very nice show on the smallish personal TV : Kingfisher calendar girls. The flight was half an hour late which messed up my already tight schedule. I took a shuttle to Jayanagar from the airport (which takes approximately one and a half hour), went to the hotel room and was out within 90 seconds and reached Nimhans Convention Centre just in time to catch Rahul Sundaram's talk on the Fedora buildsystem. Before lunch, I also managed to catch some bits of the talk by James Morris on seLinux sandboxing. Kedar had a fun time mocking my formal attire. (In my defence, I didn't know people come casually to the conference. I thought everyone will be in tip-top formals)

After a surprisingly great lunch, I went to attend Kedar's talk on Fedora-ARM (the reason I was at the conference). The attendees included many Fedora guys, some interested students and some others that I cannot place. Kedar talked on the status of Fedora-ARM, how to contribute, ARM or secondary arch specific issues and other stuff. Read here for more info. Rahul inaugurated the dist-f13 tag with his gnote build. After that was Rahul's talk on how to create a customized Fedora remix. I was looking forward to this particular talk since the idea of making a remix for sheevaplug has been playing into my mind since a while now. It was insightful. I moved on to attend Holger Frether's "How to make WebKit faster". It wasn't what I was expecting, but Holger gave a nice demonstration of how to use a number of well-known and less-known tools to measure performance. His focus was on beagle-board. The best talk of the day was the Keynote by Harald Welte. He talked about bringing open-source to less explored platforms like GSM, RFID, etc. Very Very inspiring!

As the day wrapped up, we decided to head to the Forum Mall for some sight-seeing (ahem). After some window-shopping, trying out fresh baked cookies (which turned out to be horrible) and exploring the Landmark bookstore, we went to have a brilliant lunch at the Anand Adiyaar Bhavan. Later we stopped by at the Corner House for desserts. I dared to order "Death by Chocolate" not thinking once why it was named so and ignoring Kedar's warnings. The result was disastrous. I never thought chocolate would ever nauseate me so much! A tip for the readers: if you ever order that at Corner House, share one between 5 people.

Attended Lennart Pottering's "Pulseaudio Internals" on the second day. He focused on the do's and dont's on System programming. Very insightful! He is a very good speaker. I wanted to talk to him (like REALLY wanted to talk to him), but a couple of guys just wouldn't let him go! They crushed my dream :-(. After another fabulous lunch, attended Rahul's talk on PackageKit and Harald Welte's workout on GSM. Went to the Nokia Maemo's booth and got a peek into how they are using Linux on their ARM core. Maemo is one neat piece of hardware. I would certainly recommend it to anyone looking for an open-source phone.

I really wanted to attend the Keynote by Milosch Mariac, since rumour had it that they are using Fedora-ARM in their product. Kedar later confirmed that the rumour was indeed true. But, sadly I had to fly back to Pune :-(.

Some of the things that really impressed me were:

Lots and lots of students attending the conference. They get a nice exposure through all the workouts and talks.
Wireless connectivity on the entire campus. Really made me realise how important my laptop is! I looked up so many terms that I heard/overheard. Very educational.
There was nothing formal about it and people were so passionate about technology.
Very Very educational.

It was my very first conference, so I wasn't as proactive as I should have been, but I was better at it on the 2nd day than the first. I look forward to FOSS.in 2010 :-).

PS: Kedar called me later just to tell me that he talked to Lennart and Milosch. grrrr. Jealous!

I want to shout this out a thousand times!

2009-11-01T23:28:00.001-08:00

A recent discussion on a fedora mailing list:

> I'd suggest that anyone who sets up a system without any user accounts
> _and_ somehow needs a GUI to configure the system _and_ can't manage
> to figure out the settings to change so they can login as root should
> probably not be pretending to be a competent administrator.
>
> Are there not enough examples from Windows of why it's a terrible idea
> to run with full administrator privileges -- especially software like
> web browsers?
>
>
Look guys, I didn't ask for a Lecture on how to do things your way, I
just ask where is Konqueror in Root.

How very true! Many a times, I know what risk I run by using GUI as root user, but I still want to. I have my calculations of risk. They have no right to disable running stuff as root. Make it hard. Okay. Spew out warnings. I don't care. But, just let the thing work! Disabling it is just imposing your viewpoints on everyone and it is NOT alright!

Lots of Love,
Me

Atomic operations from userspace

2009-10-29T06:16:00.001-07:00

Lennart Pottering, the pulseaudio developer, has made some interesting observations on emulating atomic operations, using ARMv5 as a running example. Obviously, the post is about architectures who do NOT have hardware support and lack or have minimal kernel support. (However, some Andrew Hayley has noted an obscure kernel API for the purpose). A good read overall.

I don't fully understand the post yet, but I intend to dig deep into this once I get free time.

FYI

yum spookiness

2009-10-27T02:37:00.001-07:00

Recently, I started getting tracebacks on a simple "yum update"

# yum update -y
Loaded plugins: fastestmirror, presto, refresh-packagekit
Setting up Update Process
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 309, in user_main
    errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 178, in main
    result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 352, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 368, in doCommand
    return base.erasePkgs(extcmds)
File "/usr/share/yum-cli/cli.py", line 641, in erasePkgs
    if not self.remove(pattern=arg):
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 3023, in remove
    (e,m,u) = self.rpmdb.matchPackageNames([kwargs['pattern']])
File "/usr/lib/python2.6/site-packages/yum/packageSack.py", line 224, in matchPackageNames
    exactmatch.append(self.searchPkgTuple(pkgtup)[0])
IndexError: list index out of range

Initially I thought it was due to corrupted yum metadata. So, I ran a "yum clean all" to no avail. But then, rpm also started acting spooky. "rpm -q kdelibs" returned nothing although I was pretty sure I had installed kdelibs (since amarok depends on it). Looked like a bombed rpm database.
So, the simple solution was to run "rpm --rebuilddb" which reconstructs the rpm database. Everything worked like a charm after this!

PS: This is more a note to myself than a post. If it helps anyone, well and good. :)

Using the IRC

2009-10-21T06:42:00.001-07:00

(Following are some ways I know to ensure privacy and avoid someone spoofing your identity.)

First, I'll recommend registering yourself with a nickname so that no one can steal your identity.
Use this nice tutorial to register your nick on freenode. Other services will have their how-tos.

Second you might want to disable FINGER, TIME and VERSION requests.
FINGER enables getting personal information like Email. TIME gives out your location information and VERSION gives out the OS and client names and versions you are using. Knowing the versions, an attacker might try a specific exploit.
Here is a nice tutorial to do it in xchat. Look at the client specific documentation if you use a different client.

Also, you might want to hide yourself from the global who/whois list (The way to do it on freenode is:)
/umode +i
(+i = invisible mode)

Leave a comment if you know a good addition to this list.

Automake supports cscope and ctags!

2009-10-13T23:34:00.001-07:00

Finally!
Automake includes support for automatic generation of cscope and ctags targets. Now, theoretically, you can take any open-source project which uses GNU build system (and manyyy of them do) and type "make cscopelist" or "make ctags" or "make tags" and it will automatically generate the tags! Not that generating tags was a very difficult process earlier, but with support in automake, we can tinker around with the tags a lot.

Eg. If there is an arch/ subdirectory in your source-code and you don't want your cscope tags to be polluted by the IA64 code that you don't really care about, you can do it very easily by modifying the Makefile.am file in the arch directory.

I.e. you can easily control how your tags are generated!

This is the commit that went in,
http://git.savannah.gnu.org/cgit/automake.git/commit/?id=ffad481680a4e6a8f220c70f07b105c9f3f2dfdf

Hail automake!

Masochism ...

2009-10-06T13:33:00.000-07:00

Objective : Whenever volumes are mounted, I see volume icons on my Desktop. I want to hide the icons because they mess with my wallpaper.

Method 1 : Find a GUI option.
Result : None present

So, after half an hour of grep-ing in ~ and lsof-ing, I found out that the option I need to set is probably hidden deep down into one of the huge gconf schemas.

So, I started to learn the gconftool-2 tool. Its one nasty command.


# gconftool-2 --dump / > /tmp/gconf-dump
# wc -l /tmp/gconf-dump 
96086 /tmp/gconf-dump

Crap! that huge!
So, I grep-ed for various keywords for another 15 min. Thats when I got a match.


# grep volumes /tmp/gconf-dump 
<key>/schemas/apps/nautilus/desktop/volumes_visible</key>
(hiding other results for the purpose of sanity)

So, thats the key I have to flip. Pored over the man-page of gconftool-2 and here is the magic command:


# gconftool-2 --type bool --set /schemas/apps/nautilus/desktop/volumes_visible false

Similarly, you can also turn-off, trash_icon_visible, network_icon_visible (hides samba mounts), home_icon_visible and computer_icon_visible. and voila! A clean desktop!

You'll have to logout and login back again to see the changes. Although, I am sure there's better way. (Like to ask gconfd to reload the config). But, its 2:00am already and I need some sleep. So, I'll update this post tomorrow!

Whew!

RPM Hell

2009-10-02T00:46:00.001-07:00

End-users these days are rarely exposed to "rpm". They'll probably be using yum or PackgeKit (or a GUI-based software installer). RPM, FYI, is the backend package management tool used by Fedora, SuSe, Mandriva, etc (together known as the RPM-based distributions). And it sucks bigtime!

Although RPM has come a long way from what it was in the 1990s and early 2000s, it stands on poor foundations and is unlikely to get any better soon. Having worked on Fedora-ARM for over half a year now, I can see the glaring issues that it faces. Fedora has done a hell of a nice job of mitigating those, but when you start to dig deeper, you realise how filthy the RPM world is.

Here is a link to a VERY old pre-yum days paper, "RPM Hell". Its a google cache link since the site which hosted the original paper was down. Here is the original paper.
Now the interesting thing is that, most of the issues mentioned in this paper STILL EXIST and are very real!

Following issues stand out:

1) Package management is pathetic. If I want to install a terminal-based editor, in all probability, I might end-up installing a whole load of GTK and X crap that I am never going to use. The main reason for this is because RPM packages are monolithic. Large packages like emacs typically install all GUI files needed to support it in X even when it is a headless server. So, we have a (needless) dependency, emacs -> gtk. Now, gtk pulls in LOTS of its own dependencies which are really not required and I end up with a bloated installation. Now, this problem can be solved by splitting emacs into 2 packages. emacs-common and emacs-gtk and I just need to install emacs-common. But, we'll soon realise that there are many capabilities a software can provide and we can't keep making a separate package for each. We'll end up with an unwieldy distribution. So, RPM should have a way of internally defining capabilities and installing only those which are really required.

While we were working on the fedora-ARM project and making a basic rootfs for F-11, we had to build almost half of the GNOME packages to satisfy dependencies. WTH!! Gnome shouldn't even come in the picture yet! But thats what happens when you are working with monolithic packages!

2) There's no way to install multiple versions of a software. Now this is not really a RPM problem, it is an inherent Linux issue. Rules like, all executables go to /usr/bin make it difficult to install multiple versions, since there'll be file conflicts.
So, if "A" needs B-1.0.0 but, "C" needs B-2.0.0, you have to choose between A and C. Fedora has greatly alleviated this problem by making sure every package that requires "B" would require the same version of "B". There are a lot of rebuilds, etc for the purpose. And they are partly justified (If we don't force packages to move to newer versions, they never will and we'll end-up with 10 versions of each package installed on our system). But sometimes, you just need a way around! and there's no way you can do it.
eg. There was a time when we had a working package built aganist python 2.4 and the new distributions had python 2.6. Now there were some difficulties because of which we couldn't rebuild the package aganist python 2.6. Only if there could be multiple versions installed! But sigh! We had to devise some dirty workarounds to get stuff working.

There are many other issues that yum intelligently hides. Most of them are highlighted in the "RPM hell" link above. Guess its time to rethink RPM?

Adding a swap device

2009-08-05T05:55:00.001-07:00

Lets assume you upgrade you RAM from 1 GB to 2 GB. Since, the swap device is usually statically created during install-time and you'll probably be stuck with less swap space than the recommended figure. Then, you have 3 options
1) Who needs extra swap?
2) Make a swap file (simpler, but slower)
3) Extend the swap device.

For Approach 2, you can use the "dd" command to created a file of required size and then use it as a loop device.

Lets see the Third approach in detail.

Run "$sudo parted --list" to get a list of block devices attached to the system.
Use "parted /dev/sda resize <partition-number> <start> <end>"

"/dev/sda" is the HDD on my machine. Replace appropriately from the "parted --list"'s output.
Take the "partition-number" from the "parted --list" command's output. Use the swap device's partition number. Adjust the start end as you wish (can be specified in MBs.. eg. parted /dev/sda resize 5 5000M 7000M"

You don't need to worry about keeping the space free or defragmenting the drives to avoid data loss. Parted (claims) to take care of it.

If you never had a swap space (and created a partition just now) or you are using a swap file,
Make a swapfs on the partition

"$mkswap <device>"
(eg. $mkswap /dev/sda5)

Then, tell the system to swap on that device
"$swapon <device>"
(eg. $swapon /dev/sda5)

Add an entry in /etc/fstab for persisting the settings
Sample entry:
"/dev/sda5 swap swap defaults 0 0"

Thats it!

Check whether everything went fine by,
"$cat /proc/swaps"

Recommended swap-space = twice of total physical memory.

enable "sudo" in Fedora..

2009-08-03T06:35:00.001-07:00

One fine day I became tired of "su -" and entering the password every other minute. (Also, had to write a script which needed to sudo). So, I decided to enable sudo access. Its pretty simple.
There's a sudo'ers file : /etc/sudoers

Run "man sudoers" to find out exactly what format the file has to be in. It is a bit complex, but don't distress, you won't need its complexity unless you are a funky sysadmin. The sudoers file usually comes with the basic infrastructure and most-frequently-used categories defined. You can define your own categories too. (refer to "man sudoers")

One thing to remember is that the sudoers file should be modified *only* with the "visudo" command. visudo checks the syntax while exiting and optionally, replaces the old file if the syntax is found to be invalid.

Just append the following entry to your sudoers file
jitesh ALL=(ALL) NOPASSWD: ALL

where, "jitesh" is the username on my machine.
"NOPASSWD: ALL" means that sudo won't ask for password. Use with care.

Avada Kedavra.

2009-07-31T22:58:00.001-07:00

Saving yourself from the Avada Kedavra of Linux : rm -rf ./

http://www.linuxjournal.com/article/10428
(Link via cheezo)

Inline Assembly in C

2009-07-15T10:25:00.000-07:00

I found this very simple and insightful tutorial. Its very good
http://www.ibiblio.org/gferg/ldp/GCC-Inline-Assembly-HOWTO.html

Fedora: A walk down the memory lane

2009-07-02T10:06:00.001-07:00

(For the un-initiated, Fedora is a Linux Distro which was forked off from Red Hat in the early 2000s.)
A nice read:
http://www.raiden.net/articles/Fedora_A_Hat_with_a_History/

Talks about the evolution of Fedora, Red Hat, RPM and also some of the design decisions that has made Red Hat what it is today.

Treat Warnings as Errors

2009-07-01T00:27:00.001-07:00

If you are reading this blog and have even a iotic sense of humour, you must have heard (and liked) this joke:

------

A man is smoking a cigarette and blowing smoke rings into the air. His girlfriend becomes irritated with the smoke and says, "Can't you see the warning on the cigarette pack? Smoking is hazardous to your health!"

To which the man replies, "I am a programmer. We don't worry about warnings; we only worry about errors."

-------

Ofcourse, I laughed my ass off when I first heard the joke. Awesome it was! But then, the last month totally changed my perception. (Probably because I faced the side-effects of that).

So, ARM architecture is very specific about alignment. It hates mis-aligned access (Its the side-effect of being an arch for embedded systems). Here is an example of unaligned access:

1 short short_arr[4];
2 long *long_ptr = short_arr;
3 blah = *long_ptr;

since the data type of short_arr is "short", gcc forces only a 16-bit alignment on the array (on 32-bit systems). but then, long_ptr is probably 32-bit and that screws up alignment for line 3. Although, i386 would work perfectly fine (but require one extra memory cycle), ARM will just give garbage data (if misaligned).

And many many many software packages are filled with such errors. Just to give you an example:
# cat /proc/cpu/alignment
User: 1265555

(this file is present only an ARM architecture because of ARMs peculiar alignment requirements)

See the number of alignment errors?
Packages like mhash (which calculate hash) build fine but fail run-time, this makes such errors difficult to track down.

So, always make it a point to compile your C programs with "-Werror" option. It treat warnings as errors. It is *very* important for your programs to work on just about any platform. Please pay attention to warnings, because warnings are there FOR A REASON. gcc developers aren't stupid!
(Although sometimes, you might be sure that even if gcc warns about misalignment, the access is aligned. eg, when you typecast an IP struct into a MAC struct, etc. These are the *only* times you can let go off the -Werror option)

But, until this point sinks into the mind of every developer, there's no option but to live in this mess. :(

Emacs splash-screen

2009-06-28T22:26:00.001-07:00

So one fine day, I grew tired of the emacs splash-screen. You try to edit a file and emacs shows its splash-screen instead of the file contents and then you have to "Press any key to continue". So one day it got on my nerves and I decided that if I don't find a way to disable it, I am going to switch to vi. But, how can an editor as customisable as emacs NOT have an option to disable the splashscreen? Ofcourse it does! and it was the first result on my google search! (probably there are just too many people in this world irritated by the splashscreen, why doesn't emacs just do away with it?)

Simply add this line to your .emacs file:

(setq inhibit-splash-screen t)

yum plug-in : Presto

2009-06-28T00:13:00.001-07:00

I have ranted about Presto in my earlier blogpost. Yum's fedora maintainers implemented this really cool idea of maintaining difference between binary packages instead of the binary packages themselves. It would help a lot to push out updates. I was sceptical at first about the amount of savings it would result in (because obviously binary difference is something very complicated). I expected a 20-30% saving. Although people kept reporting numbers of upto 90%.

Today, I decided it was time to "yum update" my machine. (crashing pidgin, crashing rhythmbox, bad font rendering). So, I ran yum update. It showed me a total download size of 876MB (i.e. the total size without presto.. since, presto is new and under beta.. its integration with yum is only perfunctory). So, I dozed off. Today I check my internet usage and guess what, the total usage yesterday night was less than 100MB. Awesome! (Usually, presto shows the difference. It prints out a line saying "This is actual download size and this was download size with presto". But, as I dozed off, the line went off the terminal limit).
Anyway, it is infact 90% savings! that is so awesome! Now I can "yum update" even during the day :D. No need to wait for the night free time.

Goooo fedora!!!!!

Linux sound level too low?

2009-06-27T23:31:00.001-07:00

After having (finally) upgraded Fedora-11, I found it pretty annoying that the sound level was wayy too low. I mean, even with speaker volume set to full, system volume set to full and application volume set to full (not to forget pulseaudio volume .. which is another layer between the app and the system), the sound levels were only just enough for me to watch a movie without missing out a dialog. WTH! I have awesome speakers and they should ideally crack my windows. But here, even full volume was just barely enough. It hurt my ego (:P).

On Windows, the full volume is a real window-cracker. What is the problem with Linux then? (I could replicate the same issue on Ubuntu 9.04 too.. that should explain why I went for generic "Linux" and NOT "fedora"). Then I found this bug filed in the fedora bugzilla which made things clearer (actually, they didn't). Don't bother to read, it went swooosh over my head. The only thing that stuck was - How to solve the problem.

Here it is:
Make sure you have the package "alsa-utils" installed.

For fedora:
$ sudo yum install alsa-utils

For Ubuntu:
$ sudo apt-get install alsa-utils

Then, run
$ alsamixer -c 0
and increase the master volume beyond green area to the red area. (I understand that the red area implies increased noise, but I didn't really notice much of a difference with a slightly less than maximum setting. Even with max master volume, there is only a slightly noticeable noise)

I don't understand why this control is hidden away in such a manner. What is the whole point? Why not export a GUI for it? Weird. Probably they'll fix it in fedora 12. There's been a lot of work in the sound area in the last 6 months. There are going to be some bugs afterall. Anyway, this healed my ego :P. Linux sound is at-par with any other sound system! It makes me happy!

Emacs: linux-c-mode and Line numbers

2009-06-25T03:25:00.001-07:00

Tired of the slow ssh access to one of the servers located outside India, I finally figured it was time to start playing with emacs to reduce the number of keystrokes needed.

First on, I wanted the linux-c-mode enabled *by default* whenever I open a file whose extension is ".c" (Yeah Yeah, I know extensions are meaningless for Linux.. but emacs is intelligent :) ). For all the other type of files, linux-c-mode should be OFF, because it will mess-up their indentation.
Second, line numbers should be displayed on the left-hand-side panel of each file I open. Then to go a particular line I just "M-g g". Simple.
Third, All this should be accessible to my user account as well as root user.

So, I edit the ~/.emacs file (create if not already present) like this (for the first goal):
(defun linux-c-mode ()
"C mode with adjusted defaults for use with the Linux kernel."
(interactive)
(c-mode)
(c-set-style "K&R")
(setq tab-width 8)
(setq indent-tabs-mode t)
(setq c-basic-offset 8))

(add-to-list 'auto-mode-alist '("\\.c$" . linux-c-mode))

The first chunk defines the linux-c-mode (from Documentation/CodingStyle) and the last line enables it by default ONLY for *.c files. Simple!

For the second part, I got a script linum.el from google (Vedang says it has been integrated into emacs)
You can find the script here: http://jitesh.1337.googlepages.com/linum.el
Drop this script at /usr/share/emacs/site-lisp/

There's a problem with this script. It doesn't print a space after the line number. Sad!
Here is a patch I wrote to fix it: http://jitesh.1337.googlepages.com/linum.el.patch

Now I want to display line numbers for EACH file I open in emacs. So, I add the following lines in the .emacs file:
(require 'linum)
(global-linum-mode 1)

(First line is needed to autoload linum.el. The second line enables it by default)
To toggle the linum mode, use "M-x linum-mode"
Simple!

Finally, I wanted to give access to root user too.
A simple solution would be to make a symlink/hardlink from /root/.emacs to /home/jitesh/.emacs.
However, I wasn't too concerned about the other users on the system :) .So, I simply copied to .emacs file to /usr/share/emacs/site-lisp/site-start.d/ and renamed it as myinit.pl. So, now, these things get enabled by default for *all* users.

Now that everything is setup (and steps documented via this blog), let me go back to coding peacefully in emacs!

Upgrading to Fedora-11 (Leonidas)

2009-06-11T09:00:00.000-07:00

Finally I got my hands on the brand new Fedora release. Fedora-11 (Leonidas). As I had a sneek-peak into the development process this time around (work reasons), I was pretty excited to upgrade to this new version. There are a couple of nice features in Fedora-11 that makes it awesome.
1) Presto: Delta-RPMs. If you are updating your system, yum now fetches only the *change* between two versions instead of pulling in the new version entirely. Savings of 60-70% of bandwidth have been reported.

2) PackageKit: Makes installation of new packages and plug-ins very easy and user-friendly.

Time for installation.
So, I booted into F10 and set the system up for the normal upgrade method that I use. Which is:
1) Insert the DVD.
2) Make a yum repo which points to the DVD.
3) Issue a "yum update"
4) Watch a movie while your system upgrades.

However, this time I decide to be a good boy and read the fedora docs first to see whether it had to say anything about upgrades. Turns out it was a good decision. Apparently, there is a bug which makes such an upgrade impossible and leads to a possibly "unusable system". Whew! close shave!

So, why the hell did they release F-11 when a method of upgrade was broken? Well, there were vibrant discussions in the fedora community over this bug and the final decision was: This upgrade method is not "recommended" and "supported" and hence, it isn't a F-11 blocker bug :(. Sigh!

So, guess I'll have to follow the "recommended and supported" method. Boot from DVD and choose "upgrade". I'll have to stare at the screen for an hour while the system upgrades :)
Looks like I'll wait for a respin which contains a fix.
(/me adds himself to the CC list for the bug)

Another caveat is that the rpm format has been changed in Fedora-11 to make it more secure. This makes upgrade impossible because the installed rpm simply cannot read these new rpms. The solution is: Fedora has issued an update to F-10 "rpm" package which is aware of this new format. So, you'll have to run a "yum update" on F-10 first (older Fedora's will have to be moved to fedora-10 and then updated via "yum update"). Once, this new "rpm" package is installed, then the upgrade should go smoothly. (Note: If you don't update "rpm", yum will fail with "MD5 sum mis-match" type of errors)

The delta between Fedora-10 and Fedora-11 is huge as compared to the previous deltas. There has been some heavy development. Lets see if Leonidas has the same majesty as the King ...

Debugging a segmentation fault using gdb

2009-05-21T23:53:00.000-07:00

I am not a big proponent of gdb. If you *really* know what you are doing, gdb shouldn't be required. But, every now and then, you come across code that has used function pointers exclusively and then, hand-debugging becomes painful. gdb to the rescue.

You'll need the following pre-requisites to use gdb to debug a segmentation fault:
1) make sure you have compiled the executable WITH debugging symbols. i.e. the "-g" flag. eg
gcc -g -o hello hello.c
Without debugging symbols, gdb won't be able to do much.

2) Linux should core-dump on segmentation fault. Set:
ulimit -c unlimited
(man ulimit for more info)

Now just run that the excutable that is segfaulting. As soon as it segfaults, you should get an output something like "Segmentation fault (core dumped)". ls in your working directory and you will find a new core file has been created (probably with the name core.{pid})

Now, we just have to tell gdb to analyze this core. Here's how
gdb {executable} {dump file}

eg. gdb hello core.1324

Check out the output spit out by gdb and make sure that all debugging symbols have been loaded.
Now, on the gdb prompt:

(gdb) bt
(bt = backtrace .. prints stack strace)
with this backtrace you'll now know *exactly* where the program segfaulted. The code file, line number and the call which was the culprit.

You can even analyze variable values on any frame. Just change to that frame:
(gdb) frame {num}
eg. (gdb) frame 2

and use:
(gdb) info locals
(gdb) info args
to query the values of local variables and passed arguments, respectively.

With all this info, you can pin down the exact reason for the segfault pretty easily. Saves a lot of time!

Packages: Naming scheme

2009-01-28T06:32:00.000-08:00

Its been a while since I wrote a nice article and today I just felt that inner need to write one. Today's topic is an introduction to the naming scheme used to manage packages. But, before that it'll be useful to distinguish between some similar terms.

Package: This is the common name that we generally use. Eg. "Amarok" is a package name
Build: A build is a particular release of a package. eg "Amarok-1.4.12-1.fc10" is a build. Note that a build specification is architecture independent.
RPM: An RPM is the architecture specific binary of a build. Eg. "Amarok-1.4.12-1.fc10.i386.fc10.rpm" is an rpm.
SRPM: Source RPM is architecture independent and contains the sources of the build packaged as an rpm. (Now, why would that be useful, you may ask. Why not a tarball? That'd take up another post to clarify. Queuing the topic)

Naming scheme:
Generally, a build is named in the following manner.
Name-Version-Release (popularly known as the NVR of a package)
Name: might be a simple name such as "Amarok". It can contain the hyphen character.
Version: Version is usually in x.y.z format. "z" being the least significant and "x" being the most significant. Usually, if a respun rpm contains only bug-fixes, only the "z" part is incremented. If there are some major changes or feature additions, "y" is incremented. An architectural change warrants an "x" increment.
But, some packages (like tzdata) use a different versioning scheme. (Since, tzdata package deals with timezone stuff, they find it convenient to include the year in the version. Eg. 2007k, etc.) But, an overwhelming majority use the x.y.z scheme.

Release: It is generally only a decimal number. It can be viewed as the minor version number. If the change is very small, so that incrementing even "z" is unjustified, release number is incremented. It might contain small bugfixes or other minor changes.

When, someone distributes a version of the rpm that deviates from upstream, (he has applied his own patches to the source) he should change the release number to reflect the same. A string is to be appended to the release number.
eg. If someone patches amarok to add his own cool new feature. He has to change the release number from "1" (say), to "1.fa1" or something similar. Just append a string at the end. So, when you see a non-standard package, you can instantly make out from the name.

RPMs also contain the architecture name along with the NVR.

Now, suppose you have 2 packages "blah-1.2.3-1.fc10" and "blah-1.2.3-1.fc11". And, there's a bug-fix which only applies to the fc10 version, what do you do? Certainly, you cannot change the Version number or Release number of the fc10 rpm because it was a "fc10" specific fix and hence, not a property of the build. In such a case, the build is spun as "blah-1.2.3-1.fc10.1". But this is a fairly rare case.

Well, This was a fairly basic post. But, this one merely does the job of heralding a series of posts on messing with RPMs and SRPMs. So, stay tuned!

Writing a resume ...

2009-01-23T22:10:00.000-08:00

Resume writing has always been a mystery to me. A rather pointless exercise of listing what you've done in the last 20 years. I mean come on, what does the HR care whether I have secured 40% or 99.99% in the 10th standard? Unless physics and chemistry combine to form a software engineer much like blue and yellow combine to give green. Another thing that has endlessly puzzled me in a resume is "Hobbies". When I was briefly working with the TnP cell (2 hours, before I resigned), I've seen people having such amusing hobbies and the courage to pen them down on the resume. How can "Watching TV" "playing video games" or even "playing pranks" be included in the hobbies section on a resume? Why would the HR give a damn even if you have a hobby of mooning every stranger you see!! Utterly pointless...

But, then my brother gave me this amazing link. (much like the ads where you are fat and the girls are disgusted by you and they always prefer the slim ones and then your friend gives you a pill to make you super-slim within a day) It is titled "Tips for a slightly less awful resume". The author has had a good experience in resume screening. The points he has made clears many of the myths we have about resume writing and to say the least, they are ground-shaking. Whatever I thought were the best practices in resume writing are listed as a strict no-no with a very good line of reasoning. It is a must read for everyone who wants to make that resume of yours slightly less awful.

The post is a bit long and I am, myself very wary of long posts. I generally skip them, if they are not entertaining enough. With this one I was hooked till the last line. The man has got an amazing sense of humour and some really sharp sarcasm. The post isn't a sermon on the do's and don'ts of resume writing but a slightly less awful waste of time on how _not_ to screw-up your resume :)

Please do make it a point to read the post atleast once before you make your "awful" resume! :)

Adding/Removing stuff from start-up

2008-12-19T03:12:00.000-08:00

Its a wonder how frequently we need to add stuff to the start-up and yet we prefer the hard-ass manual way. You can lose hours if you forget to do one small thing (eg. one of my builds, which takes 2 hours on an average, failed merely because I forgot to start httpd.) Thats when I considered this start-up thing seriously.

1) Adding/Removing services:
In very (very) loose terms, services comprise of everything under /etc/init.d directory. (Files under this directory are scripts that manage starting, stopping, restarting and status reporting of the services). The 'chkconfig' command is used to Add/remove these services from the start-up. These can be done per-runlevel too. (Just pass a suitable '--level' option)
eg. To start httpd on startup. Just type
>> chkconfig httpd on
(To remove httpd from startup, replace 'on' with 'off')
(Syntax ==>> chkconfig {service-name} on/off)

(To add a script to be managed by chkconfig, just copy the script to /etc/init.d directory and run
>> chkconfig --add {name}
This creates links in /etc/rc[0-6].d directories. Scripts in these directories are run each time the run-level is switched (provided that the service isn't already running))

Type in 'chkconfig --list' to get a nice summary of the status of services at each runlevel.

2) Initializing environment variables at start-up:
Just add an "export name=value" command to ~/bash_profile file. (Create the file if it isn't already there). Note that this would be a per-user initialization.
To initialize an environment variable globally, add an entry to the /etc/profile file.

3) Initializing aliases:
Add a "alias name=value" to ~/bashrc file. Create the file if it isn't already present.
/etc/bashrc file if you need global changes.

4) Random stuff in start-up:
What if you want to run a random command at start-up? (eg. Flushing ip-tables). In fedora you can add the commands to /etc/rc.d/rc.local file. But, the file is distro dependant. Most probably, you'll find the appropriate file in /etc/rc.d directory.

Hope it helps!

Patching....

2008-12-06T20:39:00.000-08:00

Before we delve into applying patches, let us have a look at a sample patch file.


--- old_test    2008-12-07 15:43:27.000000000 +0530
+++ new_test    2008-12-07 15:43:39.000000000 +0530
@@ -1,5 +1,6 @@
This
is
+also
a
test
file

The first line gives details about the original file and the second line about the modified file.
Then "@@ -1,5 +1,6 @@" marks the beginning of the hunk. This line tells the patch command where, in the file-to-be-patched should it find the specified change.

The line prefixed by "+" is the line that was added in the modified version. Other lines constitute the context.

Now why is context important?
Here's why. Consider this scenario. You work on this amazing new idea on a file called blah.c. At the same time, another of your friend modifies the same file to implement his crazy new idea. So, would your patch not apply to your friend's version of blah.c?
Chances are, it will!.. because patch tries to find the same "context" in the neighbourhood of the original line numbers. If it does, patch applies successfully!... Now this is what makes patches a good idea!.. DISTRIBUTED DEVELOPMENT!

So, coming back to the patch command,
>> patch -pNUM < patch-file

Note that patch reads the patch file from the standard input. So, you have to redirect the standard input to the patch file.

the -p option is used to specify number of components to strip.
Here's a line from a patch file. Lets take it up as an example to understand the -p option
diff --git a/wlan/driver/linux/some-name.c b/wlan/driver/linux/some-name.c

the -p option works on the path "a/wlan/driver/linux/some-name.c".
with -p1 : it strips one element from the LHS of the path. Thus, it will try to find "wlan" directory in the current directory.

with, -p2 : patch will strip the first 2 components. i.e. "a" and "wlan" and it will try to find "driver" directory in the current directory.

and so on.
If you don't give a -p option, -p1 is used by default.

Unapplying patches:
It is possible that you applied a wrong patch to your code. Or you find out after applying the patch that the damn thing is not going to work. Or there is a regression. Whatever. The point is, you want to revert the patch.

Just pass a "-R" option to patch and it will try to unapply the patch. Although, do read the man page carefully before using this option.

Random notes on the patch command:
1) Use the strip-level option (-p) properly. If patch command doesn't find the file to be patched, chances are that you have specified an improper strip level option. (In that case, patch becomes interactive and asks you the path of the file to be patched. Most of the times, it means that you need to correct the strip-level option. so, exit with ctrl+c and give the correct strip level)

2) In case, one of the hunk fails (Which is the case when, the original file is modified by someone else prior to applying the patch and there is a conflict), the hunk that failed is saved in the "filename.rej" file. You need to apply that change manually. Just open the .rej file, see which hunk failed and make the changes manually. Also, the original version of the file is saved in filename.orig, just in-case you changed your mind or you are just too lazy to do the hard-work.

3) A patch is a defacto standard to give out changes like bug-fixes, minor feature additions, security patches etc. Patches are extensively used when a team is working on the same workspace (which is almost always the case). Revision control softwares like git, SVN, CVS, perforce, etc all support patches extensively in all their operations.
Patches are something you really want to know!

Hope it helped!

Diff and Patch

2008-12-03T07:43:00.000-08:00

Patches show the difference between two versions of the same file.
A patch shows which lines were newly added and which lines were removed. What makes a patch very useful is that a patch can be applied to the old version of the file to obtain a new version.
i.e. (old version) X (Patch) = (new version)

This fact makes it easy to manage a community software where a software is open-source and anyone can submit changes. Trust me on this, without patches we would be nowhere.
For eg, Andrew Morton (Linux kernel maintainer) gets hundreds of patches everyday. In the absence of patches, he would have to integrate each and every change manually which would be a VERY time consuming and an inaccurate process. Patches automise this job.

"Diff" is used to create patches and "patch" is used to apply it.

Diff:
syntax
>> diff [options] [original version] [modified version]
Generally, I used the diff command with "-du" options.
eg. diff -du old_test.c new_test.c > test.patch

Note that diff produces output on the stdout and you'll have to redirect it to a file.
Options:
-d : Use an algorithm which will produce smaller set of differences
-u : Use unified output format
(-u causes newly added lines to be prefixed by "+" and removed lines prefixed by "-". Otherwise, other prefixes are used which people generally dislike :)

-r : recursive.
Now this is a beauty. You can find differences between all files under a directory. So, if you have changed multiple files, use this option to generate only ONE patch and _not_ a patch for each file separately.

-N : Include new files
With -r, it is possible that you might have added a new file in the newer version of the software which wasn't there in the old version. Diff by default ignores such files. Use this option to change this default behaviour.

Random notes on patches:
1) In patch files, you will find something called hunks. Hunks are a set of changes alongwith some context. By default, 5 lines above the change and 5 lines below the change constitute the context. View a patch file in emacs/vim and you'll know what I am talking about. Context is a very important part of the patch file. I'll come back to it when I talk about applying patches.

2) The beauty of patch files is that you can actually MODIFY them. Well, it won't seem to be a big deal if you havn't used patch much. But, it is a quite handy trick. You can remove entire hunks and the patch is still valid!
So, play around with it a bit.

3) Many people and inherently afraid of patches. Trust me they'll have a very hard time when they work in a group on a single project. Patches are a great way to review changes that have been made or pass them on to others. Without patches, I cannot concieve a way I'll pass on my changes to others!
Patches are very easy to use. Just try and get comfortable with them. Ask around if you think you are stuck.

Now that we know how to generate patches, it is time to move ahead and learn how to apply these patches.
But, I think this post is long enough. So, I'll write about it in the next post.

(Do leave a comment and let me know if I need to add/remove/modify anything to make the post lucid and easily understandable yet comprehensive)

How to browse source-code?

2008-11-12T01:32:00.000-08:00

99% of your life is wasted is studying code written by someone else. And studying this code can be an awfully irritating task. (Although, studying codes written by TE/BE students can be a really pleasing experience. You can just lambaste them on every possible line! It is fun. Trust me. Simply log-on to the lab-server, access someone else's code. Don't worry about the permissions. Seems that PICT's lab assistants believe in open source. So, get someone else's code and vent your anger through comments. Don't forget to save the file :D)

Anyway, getting back to the point. There are few tools available which can make the task a bit simpler.
1) Etags
2) Cscope
3) Doxygen

My good friend Vedang has written enough about them. Here are the links to Etags and Cscope posts.

Personally, I think Cscope is just amazing. You can find functions which call this particular function and there is a regular expression search too. Etags is quick but primitive. I generally use Etags for simple browsing and Cscope only when Etags cannot help!

Tools aside, you also need to have a strategy for browsing source code. You cannot simply dig into the source code to understand how linux works. (Although, it may work if you are sufficiently genius.)
So, first try to get a doc explaining what the piece of code does. Having an insight into the working certainly makes browsing the code a lot simpler and faster. Then, use whatever-piece-of-software you have so that you know the inputs and outputs. Start with the source code only when you think you have a good understanding of the product.

Don't worry if you cannot seem to understand or remember anything the first time. Be persistent and you'll get the knack of it.

Browsing the source, btw, is a very good way to understand how people think. So, if you have a crush, go get her source code! :P

grep-ing your way to glory!

2008-10-30T06:41:00.000-07:00

Grep is oxygen. Also, Grep is a very very powerful content search tool found on Unix-based systems.

Basic Syntax:
>> grep [options] [pattern] [files to be searched]
eg. >> grep -r Hello .
(Grep the current directory for the string "Hello". Make the search recursive i.e. search within sub-directories too)

grep searches in all files. Even binary files. This can sometimes be irritating, since results from binary files are seldom useful. Pass a "-I" ignore binary files. (grep examines the first few of the file to determine whether the file is binary or ASCII. extensions are NOT used)

Also, you may want to ignore case. Use "-i"

By default, the format of printing the output is
"[path-to-file]: [line containing the matching pattern]"
Sometimes it might be useful to alter this pattern.
Use
1) "-n" to display the line number in the file where a match was found
2) "-o" to show only the matching text and NOT the entire line.
3) "-C [num of lines]" to show the context (before and after)
4) "--color" to show the matching pattern in color.

Also, sometimes you might be interested in only the number of matching lines. use "-c".

Another VERY useful option is "invert-match" (i.e. select non-matching lines)
use "-v".
eg. >> grep -v -e "^#" my.c
(i.e. select all lines that do NOT start with a #)

grep can take only basic regular expressions. Use "-e" to pass an expression which uses the extended regex format.

These are pretty much the frequently used options. man grep for more options.

Some Grep usage scenarios:
grep alongwith piping makes a great combo!
you can say something like
>> find . -name "*.c" | xargs grep -in Hello
to search only C files within a directory.
Or you can say
>> find . -name "*.c" | grep hello | grep -v not_welcome
to find a filename containing the string "hello" in it but NOT "not_welcome" !!

NOTES:
1) I purposely havn't talked about regular expressions. They are worthwhile to learn since they are required everywhere, not only for grep. With a good knowledge of regular expressions, grep is awesomely powerful!
2) grep is slow. Understandable. since it is content search!
3) Are more examples required? Let me know!

PS:
So, grep is used for search. But, what about search and replace?
Well, use "sed" for that. sed is good! (sed has saved me loads and loads of time. It is good to know atleast basic sed)
A nice tutorial can be found here.

EOF

So what is so great about find?

2008-10-23T10:56:00.000-07:00

Why do people not like find? It is a nice command to have. I gather that there is no "find" in BSD and that find is "impure" (I am not sure about both) Well, frankly, it makes no difference to me, as a user, whether a command is pure or impure. Find is a simple slick and helpful command and that is what I care about. As for BSD, get the sources and compile!

LOCATE:
Anyway, there is an alternative to find (albeit lame). It is "locate".
Basic syntax :

>> locate [pattern]

(Don't mind the square brackets. '<' brackets makes google think that I am writing some xml doc. So, a quick fix) Locate merely searches the database of names that was created by "updatedb" command. updatedb is periodically scheduled by crond. So, your database is automatically updated. You don't need to worry about that. One advantage of this approach is that the search is lightning fast. If you compare the performance of locate and find, locate would win by light years! Also, the syntax of locate is deceptively simple. Merely, "locate {pattern}"
But, as every girl you like is always taken, locate has its own drawbacks.

1) locate command doesn't check whether the file actually exists on your machine. It simply consults the database and gives back the results. (Thus, after an updatedb, you delete a file, you still get it in the locate results until crond schedules another updatedb). You can pass "-e" to locate to avoid this condition. -e checks whether the file actually exists.

2) If you create a new file, it won't appear in the locate results for a while. (Until crond schedules an updatedb)

3) There is no way you can ask locate to search only within a particular directory. You have to search the entire directory tree, always. This, in my opinion, is the worst drawback of locate.

Some other options that could be passed to locate:
-i : ignore case
-b : match basename only. (i.e. the final component of the whole name)
-e : check if file exists
-E : print only files that do not exist, but were there when database was made.

FIND:
Find is good.
With find you can search a part of the direcotry tree. The basic syntax is

>> find [directory to search in] [options]

eg. >> find . -name "Linux"
the beauty of the command is that, you can have many many search parameters. Like:
1) -name [pattern]
Search name of the file
2) -user [username]
Search for files owned by this particular user
3) -perm -{octal permission bits}
Search for files with this permissions.
4) -type [c]
Search for files only of type . (eg. b for block. p for pipe etc etc)
and many more. Check out the man page. You can combine the above options.
eg. >> find . -name "Linux" -user httpd

Also, with find the expression part can be "expression" "!expr" "expr or expr" "expr and expr" etc etc. Quite handy!

What more? You can format your output as per you wish. check the -print option in the find man page. Havn't found the need to use this one, but such options are good for writing scripts.

When you search in "/", find unecessarily searches the mounted windows paritions too? No prob, pass "-mount" and it won't !! It'll skip all mounts.

With find, you can have more control over the search. But the downside is that find is terribly and horribly slow.

As a third lame option. You can define an alias. Something like

>> alias myfind='updatedb; locate'

So, "myfind" becomes a new command which expands to 'updatedb; locate'. Neat, eh?
although, the overhead of updatedb might not always be acceptable. In those cases, you can simply use locate. updatedb, btw, is NOT very slow. So, it is a better option than find, if you don't want too tight a control on your search.

Enjoy searching!!

(PS: Lookout for the next post on grep)

Emacs v/s vi !

2008-10-14T09:26:00.000-07:00

While looking for a comparison among the editors I stumbled across these. Great!!
(Both emacs and vi are great and masochistic. Both of them have a steep learning curve but equally great are their uses. You can find a good comparison here.)

Emacs and vi have been involved in a never-ending holy war since ages. All of the verbal battles involve critisizing the people using the editor. I love such wars! :D. Yes. What with all the insulting and swearing :D. Here is a sample....

LOL ----->>

How to ask questions?

2008-10-14T08:50:00.000-07:00

n00bs often get frustrated when they start using Linux. Partly because it is very different than Windows and partly because they lack patience. Patience to figure out what is not working and why it is not working. Patience to learn. They want quick answers and so they turn to forums and mailing lists. We have seen hundreds of stupid posts/mails which go something like this - "I can't get xyz working. Help me. NOW!".

One thing we all should understand is the people who made the software are not your servants. You should be thankful they opted to share their effort for free. Isn't it your responsibility to atleast try somethings out before wasting their precious time?? You get to learn a lot in the process too. Its a win-win situation!!

So, before posting any question to a mailing list/forum please make sure thatyou have done your best but couldn't find the solution. (Google. Ask around. Have a look at the debugging output) Let others know what you have tried. Write it in your mail/post. Be respectful and humble.

There is a nice document which documents this particular aspect. You can find it here.

Happy getting-things-to-work!!! :D

Nothing technical about it !!

2008-10-13T05:22:00.000-07:00

So, we are supposed to mail our status reports each week to our manager. One of the guys (some people have a good good idea who he is !!) wrote this in his status report:

Tasks this week:
* Worked on "one build to rule them all" infrastructure And bang came the reply from the manager:

Three Builds for the Linux-kings in the MSI,
Seven for the SQA-lords in their shield-halls of stone,
Nine for MIPL Mortal Men doomed to code,
One for the Dark Lord Kedar on his Bamboo throne

In the Land of Linux where the RPMs lie.
One Build to rule them all, One Build to find them,
One Build to bring them all and in the flash-ness bind them
In the Land of Linux where the RPMs lie.Need I say more?

Coding Style...

2008-10-11T08:03:00.001-07:00

No technical blog can start without commenting on the coding style. 90% of the times the code written by you would have to be understood by a third person who will enhance or maintain it. (The other 10% being college assignments.. even there the code has to be understood by MANY third people :P). So, wouldn't it be nice if your code was easily understandable and readable?

You will understand the importance of coding style once you start studying code written by others. Some get so frustrated that they re-indent the entire code base (:P) which leads to worse problems!! :P. So, better write understandable code. Coding is an art and an art is nothing without presentation!

A very good guide to coding style can be found in the linux kernel documentation.
Just check out Documentation/CodingStyle in the Linux kernel sources. Remember that this coding style is recommended by people who have written/maintained millions of LOC. It has to be the best. And it is!

The file also provides a configuration to be integrated with emacs so that emacs does some of the things automatically for you. Emacs is great!

Welcome to the world where we like to speak quality C !! :)

The traditional post....

2008-10-09T08:59:00.001-07:00

As is the tradition, the first post of any blog always explains what the blog is going to be about. And as is the tradition, the blog is never about it !!

We'll stick to the tradition.
So here goes.
The blog will contain posts that I don't think are fit for my other blogs. My experiences and Humour. Basically everything that people don't think is fun. I won't say that I agree with them, but I can understand. Some things are better left to geniuses :P:D.
Anyway, expect technical or atleast no non-technical posts from this blog!

And have fun!
Amen!