Alpine boot process on the Raspberry Pi

Today we will have a look at the Alpine Linux boot process on a Raspberry Pi in some detail.

clip_image001

The picture shows the contents of a “virgin” SD card with the Alpine image, which has not been booted yet.

clip_image003

This picture shows the contents of the boot folder.

Initial boot Stages on the Raspberry Pi

Refer to this link for an in-depth guide at different boot media:

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/bootflow.md

The Raspberry Pi does not have a classical “BIOS” as in x86 based IBM PC compatible computers. It’s CPU should be thought of as an add-on to the VideoCore IV for boot purposes, rather than the other way around.

Thus, initially the VideoCore IV, the GPU, has control. It starts off from it’s own ROM (first stage), and checks the boot sources for a file called bootcode.bin. For our purposes we will assume that this file is loaded from the SD card.

bootcode.bin (51 KB) is loaded into the local 128K L2 cache of the GPU. It enables RAM and loads the third stage: start.elf (2759 KB)

start.elf will read config.txt, and set up the appropriate settings for the VideoCore IV. config.txt is a parameter file for the Raspberry Pi where you can set e.g. the video mode, overscan, audio options, MPEG codecs, overclocking settings, etc. – it is “like” the BIOS settings part of the traditional computer. start.elf itself is the firmware for the GPU, which is it’s own operating system called “VideoCore OS”.

NB: There are different versions of start.elf, including start_x.elf which includes camera drivers, and start_cd.elf – which is a cut down version of start.elf, to be used when the GPU memory is reduced to the maximum: gpu_mem=16. This latter version might be interesting for embedded systems. Refer to this site for more information: https://www.raspberrypi.org/documentation/configuration/boot_folder.md

config.txt is optional. If you use the same SD card (or image) for several Pi’s, you can set conditional filters to apply to certain scenarios; you can even apply a condition based on the serial of the Raspberry Pi, or on the state of a GPIO. There is also the possibility to include additional files into the config.txt. Taken together, you get a quite powerful system for the basic configuration of the Raspberry Pi.

Of course, there are also boot options in config.txt, which control how the further boot proceeds.

start.elf then loads the kernel (as specified by config.txt / defaults), and launches it with cmdline.txt as kernel command line .

config.txt boot options

have a look at:

https://www.raspberrypi.org/documentation/configuration/config-txt/boot.md

to read more about the available boot options.

Alpine’s default config.txt

Let’s have a look at the config.txt which ships with Alpine’s 3.8.1 ARMHF release:

  • disable_splash=1
    • this will disable the rainbow screen on start (purely cosmetic)
  • boot_delay=0
    • do not wait in start.elf before loading the kernel. (a boot delay might be necessary for some SD cards to “get ready”)
  • gpu_mem=256
    • sets split of memory between GPU and CPU. The Pi’s memory is shared. Here we set 256 MB for the GPU, you might want to reduce this on embedded systems which do not drive displays.
  • gpu_mem_256=64
    • sets the split of memory for Pi’s with only 256 MB of total memory (only some old generation Pi 1B’s)
  • [pi0]
    • conditional statement for Pi Zero; this covers Pi Zero and Pi Zero W
    • all the statemens until the next conditional are going to be applied only if we run on a Pi Zero / Zero W
    • the Pi Zero / Zero W still use an older ARMv6 ISA CPU (ARM11 @ 1 GHz)
    • we will use a kernel compiled for ARMv6
  • kernel=boot/vmlinuz-rpi
    • this specifies the kernel to load
    • this must be an uncompressed kernel image file
    • 32 bit kernels are loaded to address 0x8000 by default
  • initramfs boot/initramfs-rpi
    • “the initramfs command specifies both the ramfs filename and the memory address to which to load it”
    • initramfs = initial RAM filesystem
    • this file can be extracted with 7Zip, for instance (extract twice)
    • NOTE: this configuration parameter is specified without the “=”
  • [pi1]
    • this will apply the following configuration lines to all Raspberry Pi 1 – will not match on Pi 2, etc.
  • kernel=boot/vmlinuz-rpi
  • initramfs boot/initramfs-rpi
  •  [pi2]
  • kernel=boot/vmlinuz-rpi2
    • notice how we use a different kernel, since here we have a CPU which can run ARMv7 code
  • initramfs boot/initramfs-rpi2
    • and a different initramfs …
  • [pi3]
    • kernel=boot/vmlinuz-rpi2
    • initramfs boot/initramfs-rpi2
  • [pi3+]
    • kernel=boot/vmlinuz-rpi2
    • initramfs boot/initramfs-rpi2
  • [all]
    • the following line(s) will again apply to all Pi’s
  • include usercfg.txt
    • will include a file usercfg.txt into the configuration, which does not yet exist in the default Alpine image.
    • here you could put additional custom configuration which you want, e.g. HDMI resolution, etc.
    • alternatively you could just paste it below in config.txt

Note: the configuration used in Alpine does not specify a second parameter for the initramfs (address at which to mount the initramfs); thus it is not advisable to copy this 1:1. That being said, it seems to work, probably defaulting to a safe default!

Read more about config.txt:

The Kernel and the Initramfs

The kernel

Depending on the config.txt settings, the correct kernel will be selected and loaded into RAM. Once it is loaded, the ARM processor core(s) (are/is) released from reset so (it/they) can boot the kernel.

It will be passed the contents of cmdline.txt as kernel command line. start.elf will also pass additional parameters of it’s own, for instance setting DMA channels, the MAC address of the SMSC LAN chip, etc.

Get the cmdline.txt:

cat /proc/cmdline

dmesg | grep “Command line”

cmdline.txt

let’s have a look at Alpine’s default cmdline.txt:

modules=loop,squashfs,sd-mod,usb-storage quiet dwc_otg.lpm_enable=0 console=tty1

This loads the modules

  • loop
    • the loop device is used to mount a file as a file system
  • squashfs
  • sd-mod
    • SCSI disk driver / block driver
    • apparently copied from Alpine default configuration, e.g. here: https://wiki.alpinelinux.org/wiki/Create_a_Bootable_Compact_Flash
  • usb-storage
    • Linux USB mass storage driver

NB: module names may contain both – and _, these symbols can apparently be interchanged.

quiet

  • sets the default kernel log level to KERN_WARNING, which suppresses all but very serious log messages during boot (source: Raspberry Pi documentation)

dwc_otg.lpm_enable=0

console=tty1

  • defines the serial console

Further reading:

initramfs

” The initramfs command specifies both the ramfs filename and the memory address to which to load it. It performs the actions of both ramfsfile and ramfsaddr in one parameter. The address can also be followkernel (or 0) to place it in memory after the kernel image. Example values are: initramfs initramf.gz 0x00800000 or initramfs init.gz followkernel. NOTE: This option uses different syntax from all the other options, and you should not use a = character here.” – Raspberry Pi Documentation

There will be a kernel message like this “[ 0.143709] Trying to unpack rootfs image as initramfs…” in dmesg, showing that the image is being unpacked.

The initramfs allows the Linux kernel to add modules, and perform other tasks in preparing for the mounting of the actual file system. It thus makes for a very versatile boot system, without having to recompile the kernel / add overhead to the kernel. For instance for users with an encrypted filesystem, initramfs will ask for the passphrase before it can decrypt and mount the file systems.

Depending on the CPU in your Raspberry Pi, different initramfs files will be used, as set in config.txt:

  • boot/initramfs-rpi for Pi 1 / 1B+ and Pi Zero / Zero W, and Compute Module 1
  • boot/initramfs-rpi2 for all other Pi’s (2, 3B, 3B+, …)

The initramfs file can be extracted using, for instance, on Windows 7Zip. (You have to extract it twice: the first file contains a gzipped second file)

initramfs images can be in different formats, depending on which algorithms were compiled statically into the kernel (e.g. gzip / bzip2 / LZMA / XZ / LZO / LZ4).

clip_image005

This second file packages a folder and file structure, which you will recognize if you have used Linux before. It is in a special format, “cpio“, which is similar to tar.

clip_image007

if you are curious, you can read more about cpio here:

https://en.wikipedia.org/wiki/Cpio

This is what the file contains:

clip_image008

in bin, it contains:

in sbin, it contains

the init file is launched by the Linux kernel as first process, and basically sets up Alpine.

Here is the tool used for generating the initramfs for Alpine:

https://github.com/alpinelinux/mkinitfs

init

this is a sequential excerpt from what the init script launched from the initramfs does:

  • creates directories /usr/bin, /usr/sbin, /proc, /sys, /dev, /sysroot , /media/cdrom, /media/usb, /tmp, /run recursively
    • some of these directories seem to be present already in the initramfs
  • installs busybox (sets up symlinks to busybox’ embedded commands)
  • sets up PATH to /usr/bin:/bin:/usr/sbin:/sbin
  • creates /dev/null using mknod
  • mounts proc to /proc
    • proc is a process information pseudo-file system
    • you can obtain information about the kernel by reading and configure certain things by writing to some files in this directory
    • the numbered entries are directories containing information about the process with this particular process id
    • https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html
  • … and sysfs to /sys
  • parses the options passed via cmdline.txt, the following are recognized:
    • alpine_dev autodetect autoraid chart cryptroot cryptdm cryptheader cryptoffset
    • cryptdiscards cryptkey debug_init dma init_args keep_apk_new modules ovl_dev
    • pkgs quiet root_size root usbdelay ip alpine_repo apkovl alpine_start splash
    • blacklist overlaytmpfs rootfstype rootflags nbd resume s390x_net dasd ssh_key

Tip: try changing “quiet” in cmdline.txt to “noquiet” to get a more verbose boot, and see many messages from init

  • mounts devtmpfs to /dev
  • loads drivers to be able to mount modloop later on
  • if the parameter nbd is present and configured (network block device), tries to bring up an ip address, and configure the network block device using nbd-client
  • if the root parameter is set
    • if [ -n “$KOPT_root” ]; then
    • n evaluates if the length of “$KOPT_root” is nonzero
    • then nlplug-findfs is executed
    • overlaytmpfs is optionally handled
    • the current mount points are migrated to /sysroot
    • switch_root is run and control is passed to /sbin/init
    • (this is an entry point for installing Alpine Linux in a regular mode, not in read-only mode)

otherwise:

  • nlplug-findfs is executed to find the boot media,
  • if the parameter apkovl is set,
    • and it is empty – and /tmp/apkovls exists (that is, files were found by nlplug-findfs) – the ovl is set from the first line of /tmp/apkovls
    • and it starts with http:// / https:// / ftp:// – a network connection is tried to pull the apkovl
    • otherwise the ovl is set to the string in the apkovl option
  • if the apkovl exists,
    • and is a .gz file, the apkovl is unpackaged tar -C “$dest” -zxvf “$ovl” > $ovlfiles and /tmp/ovlfiles is filled in with tar’s output
  • there is code for a splash screen, fbsplash is one of the functions which is included in busybox – it displays fbsplash*.ppm files
  • if [ -z “$ALPINE_REPO” ] -> means, if the string $ALPINE_REPO is empty
    • under some circumstances default bootservices are added, for the runlevel sysinit, boot, shutdown and default (e.g. the service firstboot)
  • apk repositories are added and expanded
    • the package alpine-base depends on alpine-baselayout, alpine-conf and some other packages
      • it also includes the file /etc/os-release -> which shows the name, version, and pretty name of Alpine Linu
    • alpine-baselayout is the base directory layout
    • alpine-conf includes, amongst others, the lbu script and setup scripts
    • busybox includes the busybox binary
    • busybox-initscripts includes some initscripts
  • finally, the root directory is switched, and /sbin/init is started
    • this will start OpenRC and execute the runlevels
    • /sbin/init is a symlink to /bin/busybox

Please note: The apkovl is mounted before modloop and before installing the packages. This allows the apkovl to configure which packages are going to be installed, amongst other things.

OpenRC Runlevels

sysinit -> boot -> default -> shutdown

https://hoverbear.org/2014/09/29/init-runlevels-and-targets/

Runlevel sysinit:

clip_image010

modloop will be started after dev-mount, and before checkfs, fsck, hwdriver, modules, hwclock, dev, sysfs

hwdrivers will be started after modloop. hwdrivers will load the drivers using modprobe.

Runlevel boot:

clip_image012

Runlevel default:

clip_image014

Modloop

There is also the modloop file (modloop-rpi or modloop-rpi2).

how it is mounted

The right file for your system is determined in the script /etc/init.d/modloop in the function find_modloop()

The result of uname -r is checked against the directory modules/ – the appropriately named subdirectory needs to be in it.

For example: 4.14.69-0-rpi2

This way the right modules for your kernel are used (uname -r prints the kernel release).

modloop is mounted in the directory /.modloop

clip_image015

Symlinks are created from /lib/firmware to /lib/modules/firmware, and /lib/modules is symlinked to /.modloop/modules

Hint: if you want to patch the Raspberry Pi 3B+ WiFi firmware, you have to put it into modloop.

Modloop’s contents

To explore modloop’s contents, you can use (tested on a mint system):

apt-get install squashfs-tools

unsquashfs modloop-rpi2

It contains firmware and drivers.

clip_image016

clip_image017

References:

further reading: