Creating OpenWrt x86 Live CD with USB Support

Update 09/04/2011: If your PC’s BIOS doesn’t have USB boot support, see Booting From USB Without BIOS Support followed by Easy Live USB for x86 OpenWRT.

Update 09/01/2011:  If your PC can boot from USB, perhaps a live USB setup will be more useful.  See Easy Live USB for x86 OpenWRT.

I decided to run OpenWrt on an old x86 PC as my broadband router, but the precompiled binaries do not work on this PC.  It does not have a hard drive or bootable USB.  I had to create a binary that boots from CD but saves configuration settings to a USB drive.  Here I will describe the changes I made to create this binary.

Source URLs

core svn://
packages svn://

These URLs point to OpenWrt Backfire 10.03, release on April 7, 2010.

OpenWrt File Systems

When running on its “native” platforms (routers), OpenWrt uses three file systems

  • A read-only SquashFS volume with core files for booting.
  • A writable JFFS2 volume for storage.
  • A overlay file system (mini_fo) that overlays the writable storage volume on top of the read-only boot volume, creating a writable root file system.

For the CD/USB-drive combination to work, three things are needed:

  • A replacement for the SquashFS volume.
  • Replacing the JFFS2 volume with a USB flash drive.
  • Modifying the initialization scripts to accommodate replacements.

Replacing SquashFS Volume

When OpenWrt boots from a router’s flash memory, Linux kernel mounts the flash device with SquashFS image at “/” and executes “/etc/preinit” to start initialization.

When booting from a CD, the kernel does not mount a file system at “/”.  Instead, it extracts files from an initramfs archive to populate the special file system rootfs.  After extraction “/init” is run to initialize the system.  “/init” in turn runs “/etc/preinit”.

To create an environment similar to SquashFS-booting, I modified “/init” to create a device to be mounted at “/”:

  1. /init” first creates “/dev/loop0” and “/dev/root”, both point to the loopback device #0.
  2. It creates an image file in rootfs, associates it with “/dev/root”, and format it as an ext2 file system.
  3. It copies files from rootfs to this image,
  4. If the image isn’t large enough, “/init” repeats steps 2 and 3 with a larger image.
  5. After files are copied, “/init” runs “switch_root” to move the ext2 image to “/” and to start “/etc/preinit”.
  6. If for some reason these steps can not be completed, the original OpenWrt “/init” script is run.

After these changes, “/etc/preinit” sees no differences between booting from CD or SquashFS, with the exception that “/dev/root” point to “/dev/loop0” when booting from CD, and “/dev/mtdblock0” when booting from SquashFS.

Patch for “/init”:

Index: target/linux/generic-2.6/base-files/init
--- target/linux/generic-2.6/base-files/init	(revision 20728)
+++ target/linux/generic-2.6/base-files/init	(working copy)
@@ -1,6 +1,57 @@
 # Copyright (C) 2006

+# The system is booted up from initramfs.  If
+# /dev/loop0 is available, we'll switch to that
+# instead.  Using /dev/loop0 allows us to use
+# extroot overlay
+try_loop0() {
+  # initialize /dev/root
+  which losetup > /dev/null 2>&1 || return 1
+  insmod loop > /dev/null 2>&1 || return 1
+  mknod /dev/loop0 b 7 0 || return 1
+  mknod /dev/root b 7 0 || return 1
+  # make a ${root_size} /dev/root, mount it,
+  # and copy files to it.  If it is too small,
+  # try a larger size until there is some
+  # space left
+  root_done=0
+  root_size=4000
+  cd /
+  while [ $root_done -ne 2 ]; do
+    # clean up if /new_root already exists
+    if [ -e /new_root ]; then
+      { umount /root ; losetup -d /dev/root; } >/dev/null 2>&1
+      rm -rf /new_root || return 1
+    fi
+    # make ${root_size} ext2 volume
+    echo "Trying root image @ ${root_size}K"
+    dd if=/dev/zero of=/new_root bs=1024 count=$root_size > /dev/null 2>&1
+    losetup /dev/root /new_root
+    mkfs.ext2 /dev/root > /dev/null 2>&1
+    mount -t ext2 /dev/root /root
+    # copy files
+    root_size=$((root_size+500)) # prep for next loop
+    { find . -xdev -type d | { cd /root ; xargs mkdir -p; }; } || continue
+    mkdir -p /root/storage || continue
+    { tar cf - `find . -xdev \( \! -type d -a \! -name new_root \)` |
+      { cd /root ; tar xf -; }; } > /dev/null 2>&1 || continue
+    root_done=$(($root_done+1))
+  done
+  exec switch_root -c /dev/console /root /etc/preinit
+# Try loop0 first.  If that doesn't work, continue
+# with initramfs

 . /etc/preinit

Replacing JFFS2 Volume

OpenWrt already has the block-extroot package that allows systems to use external devices for storage.  The only task for me was to create a setting that works with as many hardware configurations as possible.  For that I modified the sample mount entry in the default fstab configuration:

  • Change mount target from “/home” to “/storage”.
  • Use the label “rootfs_data” to locate the file system.  This ensures block-extroot can still find the USB drive when other storage devices are added.
  • Remove file system type for additional flexibility.
  • Enable fsck.
  • Set “is_rootfs” option as required by block-extroot.
  • Enable the modified entry.

Patch for “fstab.config”:

Index: package/block-mount/files/fstab.config
--- package/block-mount/files/fstab.config	(revision 20728)
+++ package/block-mount/files/fstab.config	(working copy)
@@ -7,12 +7,12 @@
 	option anon_swap 0

 config mount
-	option target	/home
-	option device	/dev/sda1
-	option fstype	ext3
+	option target	/storage
+	option label	rootfs_data
 	option options	rw,sync
-	option enabled	0
-	option enabled_fsck 0
+	option enabled	1
+	option enabled_fsck 1
+	option is_rootfs 1

 config swap
 	option device	/dev/sda2

Modifying Initialization Scripts

In Backfire 10.03, block-extroot is written to work on systems that have MTD devices and JFFS2 volumes.  If either does not exist, the initialization scripts do not create the overlay file system.  This is a problem for my x86 system since it has neither MTD nor JFFS2.

The solution is to run the block-extroot scripts before checking for MTD or JFFS2:

  • A new hook “do_mount_extroot” is created in the script file “75_mount_extroot”.  The name “75_mount_extroot” ensures “do_mount_extroot” is run before “do_mount_root” from “80_mount_root”, which checks for MTD and JFFS2.
  • Two block-extroot functions, “determine_external_root” and “external_root_pivot”, found in “50_determine_usb_root” and “60_pivot_usb_root” respectively, are moved from “do_mount_root” to the new hook “do_mount_extroot”.
  • block-extroot Makefile is modified to install “75_mount_extroot”.

Patches for block-extroot:

Index: package/block-extroot/files/75_mount_extroot
--- package/block-extroot/files/75_mount_extroot	(revision 0)
+++ package/block-extroot/files/75_mount_extroot	(revision 0)
@@ -0,0 +1,10 @@
+# Copyright (C) 2011 William H Liao
+do_mount_extroot() {
+    boot_run_hook preinit_mount_extroot
+boot_hook_add preinit_main do_mount_extroot
Index: package/block-extroot/files/60_pivot_usb_root
--- package/block-extroot/files/60_pivot_usb_root	(revision 20728)
+++ package/block-extroot/files/60_pivot_usb_root	(working copy)
@@ -16,5 +16,5 @@

-boot_hook_add preinit_mount_root external_root_pivot
+boot_hook_add preinit_mount_extroot external_root_pivot

Index: package/block-extroot/files/50_determine_usb_root
--- package/block-extroot/files/50_determine_usb_root	(revision 20728)
+++ package/block-extroot/files/50_determine_usb_root	(working copy)
@@ -46,5 +46,5 @@

-boot_hook_add preinit_mount_root determine_external_root
+boot_hook_add preinit_mount_extroot determine_external_root

Index: package/block-extroot/Makefile
--- package/block-extroot/Makefile	(revision 20728)
+++ package/block-extroot/Makefile	(working copy)
@@ -55,6 +55,7 @@
 	$(INSTALL_DIR) $(1)/lib/preinit
 	$(INSTALL_DATA) ./files/50_determine_usb_root $(1)/lib/preinit/
 	$(INSTALL_DATA) ./files/60_pivot_usb_root $(1)/lib/preinit/
+	$(INSTALL_DATA) ./files/75_mount_extroot $(1)/lib/preinit/
 	$(INSTALL_DIR) $(1)/lib/preinit
 	echo "extroot_settle_time=\"$(CONFIG_EXTROOT_SETTLETIME)\"" >$(1)/lib/preinit/00_extroot.conf
 	$(INSTALL_DIR) $(1)/overlay

Other Enhancements

Since fsck is enabled for “rootfs_data”, block-extroot will run fsck although it does not know the file system type.  To work around this  I modified “” to try all known types.

Patch for “”:

Index: package/block-mount/files/
--- package/block-mount/files/	(revision 20728)
+++ package/block-mount/files/	(working copy)
@@ -12,22 +12,25 @@
 	local found_fsck=0

-	[ -n "$fsck_type" ] && [ "$fsck_type" != "swap" ] && {
 		grep -q "$device" /proc/swaps || grep -q "$device" /proc/mounts || {
 			[ -e "$device" ] && [ "$fsck_enabled" -eq 1 ] && {
 				for known_type in $libmount_known_fsck; do
-					if [ "$known_type" = "$fsck_fstype" ]; then
-						fsck_${known_type} "$device"
-						found_fsck=1
-						break
-					fi
+					{ [ "$known_type" = "$fsck_fstype" ] || [ -z "$fsck_fstype" ] || [ "$fsck_fstype" = "auto" ]; } && {
+						[ "$known_type" = "$fsck_fstype" ] ||
+							echo "Trying fsck_$known_type on $device..."
+						fsck_${known_type} "$device" && {
+							found_fsck=1
+							break
+						}
+					}
 				if [ "$found_fsck" -ne 1 ]; then
+					{ [ -z "$fsck_fstype" ] || [ "$fsck_fstype" = "auto" ]; } &&
+						echo "Giving up fsck on $device"
 					logger -t 'fstab' "Unable to check/repair $device; no known fsck for filesystem type $fstype"
-	}


This change unfortunately runs into bug in the e2fsprogs package:  The function “fsck_e2fsck” in “” does not correctly return a status code.  I have an ugly fix.

Patch for “”:

Index: package/e2fsprogs/files/
--- package/e2fsprogs/files/	(revision 20728)
+++ package/e2fsprogs/files/	(working copy)
@@ -5,17 +5,21 @@

 fsck_e2fsck() {
-	e2fsck -p "$device" 2>&1 | logger -t "fstab: e2fsck ($device)"
+	# long and complicated pipe because ash does not
+	# support pipefail like bash does
+	{ e2fsck -p "$device" 2>&1; echo "#### status $?"; } |
+	awk "
+		/^#### status/ { exit \$NF }
+		{ print \$0 | \"logger -t \\\"fstab: e2fsck ($device)\\\"\" }
+	"
 	local status="$?"
 	case "$status" in
 		0|1) ;; #success
 		2) reboot;;
-		4) echo "e2fsck ($device): Warning! Uncorrected errors."| logger -t fstab
-			return 1
-			;;
+		4) echo "e2fsck ($device): Warning! Uncorrected errors."| logger -t fstab ;;
 		*) echo "e2fsck ($device): Error $status. Check not complete."| logger -t fstab;;
-	return 0
+	return $status

 fsck_ext2() {

Finally, a bug in a kernel module Makefile prevents block-extroot using an ext4 volume as the “rootfs_data” volume.  ext4 depends on crc16, but the Makefile does not mark crc16 as required by block-extroot.

Patch for “”:

Index: package/kernel/modules/
--- package/kernel/modules/	(revision 20728)
+++ package/kernel/modules/	(working copy)
@@ -64,7 +64,7 @@
   TITLE:=CRC16 support
-  AUTOLOAD:=$(call AutoLoad,20,crc16)
+  AUTOLOAD:=$(call AutoLoad,20,crc16,1)

 define KernelPackage/crc16/description

Configuring OpenWrt

Configuration for storing data on ext2 formatted USB drives:

·         Target system

o    X86

·         Target image

o    ramdisk:  Y

o    iso:  Y

·         Base system:

o    block-hotplug:  Y

o    busybox

§  Configuration

·         Linux System Utilities

o    losetup:  Y

·         Kernel Modules

o    Block Devices

§  kmod-loop:  Y

o    Filesystems

§  kmod-fs-ext2:  Y

o    USB Support

§  kmod-usb-core:  Y

§  kmod-usb-storage:  Y

§  kmod-usb-uhci:  Y

§  kmod-usb2:  Y

·         Utilities

o    Filesystem

§  e2fsprogs:  Y

o    disc

§  block-extroot:  Y

Creating ISO Image and Labelling USB Drive

After applying patches and configuring, the normal “make” command will compile OpenWrt and create the ISO image “openwrt-x86-iso.fs” in “bin/x86”.

Use the -L option of tune2fs to label a USB drive.

Future Plan

  • Backfire 10.03.1
  • Adding command line arguments to override “rootfs_data

Other Thoughts

Why a new file system is needed to replace initramfs/rootfs:

  • Can’t use “switch_root” with the overlay file system.  “switch_root” will delete rootfs content before switching, and this will make the read-only portion of the overlay file system empty.
  • pivot_root” doesn’t work with rootfs, and block-extroot scripts use “pivot_root”.   rootfs is special.  It belongs to the kernel and not mounted.  “pivot_root” can only swap mounted file systems.

Why the ext2 image on rootfs isn’t deleted by “switch_root”:

  • When “switch_root” unlinks that image (it does do that), the link to the image is removed.  However, since the image is still associated with a lookback device, it still has an open file descriptor.  This file descriptor prevents the space occupied by the image from being freed.

Why is the “” patch so ugly?

  • Because BusyBox’s ash does not have a “pipefail” feature as Bash does, a pipe always returns the status of its last command.  But for this script we want the exit status of the fsck command, which is not last.  So the fsck exit status is sent through the pipe to be extracted later.

How does OpenWrt shell scripts access UCI configuration files?

  • OpenWrt has a whole set of shell functions for that purpose.  Many (“config”, “config_load”, “config_get”, “config_set”, “config_foreach”) are in “/etc/”.  They create shell variables to store configuration values.  The names of these variables match UCI option names.

Regarding “rdinit=” and “init=” command line arguments:

  • “rdinit=” is for rootfs only.  “init=” is for the mounted root file system only.  “rdinit=” defaults to “/init”.  “init=” has no default.
  • The function “kernel_init” in “init/main.c” first checks for “rdinit=” in rootfs.  If it is not found, “rdinit=” is set to NULL, and “prepare_namespace” in “init/do_mounts.c” is called to mount the root file system.
  • kernel_init” then calls “init_post” (still “init/main.c”) to run the init process.  “init_post” doesn’t know whether it is dealing with kernel’s rootfs or a mounted file system.  It just tries “rdinit=” first, then “init=”, and finally some hard-coded names.

How to pass arguments to a module?

  • It seems adding “module_name.options” to kernel command line works.  See how OpenWrt uses block2mtd when booting x86 from ext2.

Change History

2011-09-04: Add note about Plop Boot Manager.
2011-09-01:  Add note about new live USB method.

Downloading Sources for Building OpenWRT

OpenWRT is a complex system with many components that depend on one another.  To successfully build a working system, a good starting point is to to download a set of files tested by the OpenWRT development team.

Core and Feeds

I divide OpenWRT files into four categories.  In the first category are OpenWRT core files.  These files are for building essential components, such as the Linux kernel or BusyBox.

The other three categories are all OpenWRT “feeds”.  A feed provides additional packages that expand OpenWRT’s features.

In the second category are LuCI files, and in the third are xWrt files.  The development teams of LuCI and xWrt work closely with the OpenWRT team.

In the last category are files for building all other packages.  For example, software RAID and multimedia packages are in this category.

Downloading Source Files

It takes five steps to download a set of tested source files:

  1. Find revision numbers
  2. Find core repository path
  3. Download core files
  4. Update feed configuration file
  5. Update feed files
Finding Revision Numbers

With each release the OpenWRT team marks this milestone on the project’s roadmap (  Starting from 10.03-rc1, revision numbers of files tested in a release are included as well.  For example, at the end of Backfire 10.03 announcement, one will find


Each revision number is actually a hyperlink to a project tree in a repository. This is convenient because we need know the core’s location in OpenWRT repository.

Finding Core Repository Path

To find this information, follow the link embedded in core’s revision number (20728 in this example). This opens up a source browser, displaying OpenWRT core project.  Near the top of the browser, under OpenWRT logo, is the repository path to the core project tree.  For Backfire 10.03, this path is “branches/backfire” (circled in red):


Downloading Core Files

To download a project from a repository, use Subversion “checkout” command:


where “URL” is a project’s URL, “REV” is the revision to check out, and “WORK_DIR” is the directory for checked out files.

For OpenWRT core, the project URL is a combination of OpenWRT repository URL and core’s repo path:

  • Repo URL:  svn://
  • Core repo path:  branches/backfire

Putting them together and adding the revision number, the command to check out Backfire 10.03 is:

svn co svn:// my_openwrt
Updating Feed Config

In the checked-out core project tree, there is a default configuration file for feeds.  For Backfire 10.03, this file correctly specifies repo URLs of packages, LuCI, and xWrt feeds, but not revision numbers:

src-svn packages svn://
src-svn xwrt
src-svn luci
#src-svn phone svn://
#src-svn efl svn://
#src-svn desktop svn://
#src-svn xfce svn://
#src-link custom /usr/src/openwrt/custom-feed

This omission is easily fixed:

$ cd my_openwrt
$ cp feeds.conf.default feeds.conf
$ vi feeds.conf

Now append the feeds’ revision numbers of a milestone (Backfire 10.03 in this example) to their URLs and save these changes:

src-svn packages svn://
src-svn xwrt
src-svn luci
#src-svn phone svn://
#src-svn efl svn://
#src-svn desktop svn://
#src-svn xfce svn://
#src-link custom /usr/src/openwrt/custom-feed
Updating Feed Files

After adding feed revision numbers, an OpenWRT script is used to download or update feed files:

./scripts/feeds update

After the download or update is finished, the working directory holds a set of tested core and feed files.  To build OpenWRT, just install packages, select the correct configuration, and run “make”.

Symptoms of Incompatible Feeds

Occasionally downloaded feed files are incompatible with the core files.  A symptom is that errors are reported while feeds are updated or downloaded:

ERROR: please fix feeds/packages/sound/mpdas/Makefile
ERROR: please fix feeds/packages/sound/mpd/Makefile
ERROR: please fix feeds/packages/sound/pulseaudio/Makefile
ERROR: please fix feeds/packages/admin/syslog-ng3/Makefile
ERROR: please fix feeds/packages/Xorg/wm/matchbox-window-manager/Makefile

Ensuring the correct revisions are specified in the feed configuration file often fixes this problem.