UBI / UBIFS on NANDSIM simulator

NAND simulator (NANDSIM) is an extremely useful debugging and development tool which simulates NAND flashes in RAM or a file.NANDSIM can emulate various errors and report wear statistics, which is extremely useful when testing how flash software handles errors.

Steps involved in the working of NANDSIM :

1 ) We work as a super user.
aruna@narsil:~$ sudo su
[sudo] password for aruna:

2) Now we mount the NANDSIM module.That is,we create a virtual raw flash device.
The parameters are as follows :
first_id_byte : The first byte returned by NAND Flash 'read ID' command (manufacturer ID)

second_id_byte : The second byte returned by NAND Flash 'read ID' command (chip ID).The entire table of chip ids is given in the file nand_ids.c in the source code given here.

third_id_byte and fourth_id_byte are optional parameters which are initialized to 0xff (empty) by the system if not specified by the user.They are the third and fourth ID returned by the READ ID command.

By default NANDSIM uses RAM but if you do not have enough RAM, you can make it emulate the flash on top of a file using the cache_file nandsim module parameter.

We create a 256MiB emulated NAND flash with 2KiB NAND page size :

root@narsil:/home/aruna# modprobe nandsim first_id_byte=0x20 second_id_byte=0xaa third_id_byte=0x00 fourth_id_byte=0x15

root@narsil:/home/aruna# cat /proc/mtd
dev: size erasesize name
mtd0: 10000000 00020000 "NAND simulator partition 0"

3)MTD is not LDM-enabled and udev does not create device MTD device nodes automatically, so create /dev/mtd0

root@narsil:/home/aruna# mknod /dev/mtd0 c 90 0

4) Now we attach UBI to our MTD device. (This can also be done by the ubiattach utility)

root@narsil:/home/aruna# modprobe ubi mtd=0

5)We create a volume on our newly mounted UBI.

root@narsil:/home/aruna# ubimkvol /dev/ubi0 -N myvolume -s 200MiB

Here we have ,
Volume ID 0, size 1626 LEBs (209793024 bytes, 200.1 MiB), LEB size 129024 bytes (126.0 KiB), dynamic, name "myvolume", alignment 1

root@narsil:/home/aruna# cat /proc/mtd
dev: size erasesize name
mtd0: 10000000 00020000 "NAND simulator partition 0"
mtd1: 0c813000 0001f800 "myvolume"
Here we note that the erasesize is the size of the physical eraseblock on mtd0 (raw flash) which is 128Kb but the size on mtd1 is the size of the logical eraseblock.For each PEB,space is required to store 2 headers - one to keep track of the erase counters and the other for the logical-physical mapping.This space is not included in the LEB.Hence the size of LEB is 126Kb.

6)We mount ubifs on our UBI volume.And copy a file to it.

root@narsil:/home/aruna# mkdir /mnt/ubifs
root@narsil:/home/aruna# mount -t ubifs ubi0:myvolume /mnt/ubifs
root@narsil:/home/aruna# cp /home/aruna/5.l /mnt/ubifs

7) Now we wish to create an image of our data so that we need not have to mount an empty device the next time we use our flash.

This can be done in two steps : Firstly we use a tool called mkfs.ubifs which creates a ubifs image of our data.We can stop here and then for next use we again create an mtd device , mount ubi on it,create a ubi volume and then use this image to recover our data.

The second step is to use the ubinize tool on the image created by the mkfs.ubifs tool.This creates ubi image (which includes data about ubi volumes ).In this case,on the next mount ,we create an mtd device and then use the image created
by ubinize.After this we mount ubi and ubifs. (This is a better method as we do not have to mount the empty ubi every time.)

Both,mkfs.ubifs and ubinize are mtd-utils.Hence we have to sort out the dependencies before using them.

8) In Fedora install zlib-devel, lzo-devel, and e2fsprogs-devel packages and in Debian install zlib1g-dev, liblzo2-dev and uuid-dev packages.

9) The git repository of mtd-utils is available at git://git.infradead.org/mtd-utils.git
We use :
root@narsil:/home/aruna#git clone git://git.infradead.org/mtd-utils.git

10) The clone command creates a new directory named mtd-utils.We cd into this directory.Here we find another directory called ubi-utils.In this directory,

root@narsil:/home/aruna/mtd-utils/ubi-utils# make install

This is to be done before step 11 else lubi library is not available for mkfs.ubifs

11)Now back in mtd-utils,

root@narsil:/home/aruna/mtd-utils#make install

12) Now that our utilities are properly installed , we create the ubifs image first.In the folder mkfs.ubifs :

root@narsil:/home/aruna/mtd-utils/mkfs.ubifs# ./mkfs.ubifs -r /mnt/ubifs -m 2048 -e 129024 -c 2047 -o ubifs.img

where,
/mnt/ubifs is data whose image is to be created
-m 2048 indicates that the minimum input/output unit size of the flash this UBIFS image is created for is 2048 bytes (NAND page in this case)
-e 129024 is the logical eraseblock size of the UBI volume this image is created for
-c 2047 specifies maximum file-system size in logical eraseblocks; this means that it will be possible to use the resulting file-system on volumes up to this size (less or equivalent) so in this particular case, the resulting FS may be put on volumes up to about 251MiB (129024 multiplied by 2047)

13) Copy the created file ubifs.img to the ubi-utils folder.Then create another file called ubinize.cfg

root@narsil:/home/aruna/mtd-utils/ubi-utils# cat ubinize.cfg
[ubifs]
mode=ubi
image=ubifs.img
vol_id=0
vol_size=200MiB
vol_type=dynamic
vol_name=myvolume
vol_flags=autoresize

Now we create the ubi image :

root@narsil:/home/aruna/mtd-utils/ubi-utils# ubinize -o ubi.img -m 2048 -p 128KiB -s 512 ubinize.cfg

where
-p 128KiB is the physical eraseblock size of the flash chip the UBI image is created for is 128KiB (128 * 1024 bytes)
-s 512 indicates that the flash supports sub-pages and sub-page size is 512 bytes

Now we have the ubi.img image saved on persistent storage.

14) We can unmount the file system and our virtual device.

15) The next time we need to use the device ,we follow only these steps :

a)Create the device

aruna@narsil:~$ sudo su
[sudo] password for aruna:
root@narsil:/home/aruna# modprobe nandsim first_id_byte=0x20 second_id_byte=0xaa third_id_byte=0x00 fourth_id_byte=0x15
root@narsil:/home/aruna# mknod /dev/mtd0 c 90 0

b) Load image to it :

root@narsil:/home/aruna/mtd-utils/ubi-utils# dd if=ubi.img of=/dev/mtd0 bs=20481024+0 records in
1024+0 records out
2097152 bytes (2.1 MB) copied, 0.0220489 s, 95.1 MB/s

c)Mount UBI to it :

root@narsil:/home/aruna/mtd-utils/ubi-utils# modprobe ubi mtd=0
root@narsil:/home/aruna/mtd-utils/ubi-utils# cat /proc/mtd
dev: size erasesize name
mtd0: 10000000 00020000 "NAND simulator partition 0"
mtd1: 0f90c000 0001f800 "myvolume"

d)Mount UBIFS to our volume :

root@narsil:/home/aruna/mtd-utils/ubi-utils# mount -t ubifs ubi0:myvolume /mnt/ubifs

We can check the file we pasted here before creating the image by:

root@narsil:/home/aruna/mtd-utils/ubi-utils# ls /mnt/ubifs
5.l


Happy reading ! :)

git-perforce interface

The best thing I found out and learned today was the git interface for perforce repositories. There are many things about perforce that made me cringe.
1) No support for local commits. Limited support is provided by distinct changelists, but it only works if a non-intersecting set of files are being modified in both commits. It is rarely the case. This usually means you cannot commit small logical chunks of code and sometimes have to submit hugeeeee code-changes (which is not recommended)
2) No rollbacks or editing commits.
3) Limited branching support (Like I don't have permission to branch on the repository. But, with git I can have infinite local branches)
4) Send out patches that others can apply. (Perforce patches cannot be applied)
5) Manually check-out each file I am going to edit. This is a major pain.

So, I set out looking for some kind of git-perforce inter-conversion software and found that git itself provides the required scripts. My internet is screwed, so I cannot provide a full link as of now. but, you'd find the git's git repository at git-scm.com. Goto contrib/fast-import. Under that where you'll find both the documentation and the script. Unfortunately, fedora decided not to package it.

With that in place, I was able to use my perforce repository as a git repository. Commit local changes. Edit commits. Send out patches for each commit separately and all the cool stuff git has to offer. (Note that perforce repository can be seen as remote branch p4/master) git-p4 assigns a changelist id to each commit, thus mapping perforce commits to git commits. At last, just call "git-p4 rebase" (to accept latest changes) and then "git-p4 submit" to submit all changes to the perforce repository.

Another hassle I got rid of was to manually check-out every file I want to edit in perforce. With git, its not necessary.

I sign-off as a happy user!

Update:
Here is the link I was talking about.
http://repo.or.cz/w/git.git/tree/ff6d26a0e1d8fad775010fa3b689c0c027da8bb0:/contrib/fast-import