Alpine boot process on the Raspberry Pi

By Maximilian Batz | 2019-01-10

Today we will have a look at the Alpine Linux boot process on a Raspberry Pi in some detail.

The picture shows the contents of a “virgin” SD card with the Alpine image, which has not been booted yet.

This picture shows the contents of the boot folder.

Initial boot Stages on the Raspberry Pi

Refer to this link for an in-depth guide at different boot media:

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/bootflow.md

The Raspberry Pi does not have a classical “BIOS” as in x86 based IBM PC compatible computers. It’s CPU should be thought of as an add-on to the VideoCore IV for boot purposes, rather than the other way around.

Thus, initially the VideoCore IV, the GPU, has control. It starts off from it’s own ROM (first stage), and checks the boot sources for a file called bootcode.bin. For our purposes we will assume that this file is loaded from the SD card.

bootcode.bin (51 KB) is loaded into the local 128K L2 cache of the GPU. It enables RAM and loads the third stage: start.elf (2759 KB)

start.elf will read config.txt, and set up the appropriate settings for the VideoCore IV. config.txt is a parameter file for the Raspberry Pi where you can set e.g. the video mode, overscan, audio options, MPEG codecs, overclocking settings, etc. – it is “like” the BIOS settings part of the traditional computer. start.elf itself is the firmware for the GPU, which is it’s own operating system called “VideoCore OS”.

NB: There are different versions of start.elf, including start_x.elf which includes camera drivers, and start_cd.elf – which is a cut down version of start.elf, to be used when the GPU memory is reduced to the maximum: gpu_mem=16. This latter version might be interesting for embedded systems. Refer to this site for more information: https://www.raspberrypi.org/documentation/configuration/boot_folder.md

config.txt is optional. If you use the same SD card (or image) for several Pi’s, you can set conditional filters to apply to certain scenarios; you can even apply a condition based on the serial of the Raspberry Pi, or on the state of a GPIO. There is also the possibility to include additional files into the config.txt. Taken together, you get a quite powerful system for the basic configuration of the Raspberry Pi.

Of course, there are also boot options in config.txt, which control how the further boot proceeds.

start.elf then loads the kernel (as specified by config.txt / defaults), and launches it with cmdline.txt as kernel command line .

config.txt boot options

have a look at:

https://www.raspberrypi.org/documentation/configuration/config-txt/boot.md

to read more about the available boot options.

Alpine’s default config.txt

Let’s have a look at the config.txt which ships with Alpine’s 3.8.1 ARMHF release:

disable_splash=1

this will disable the rainbow screen on start (purely cosmetic)

boot_delay=0

do not wait in start.elf before loading the kernel. (a boot delay might be necessary for some SD cards to “get ready”)

gpu_mem=256

sets split of memory between GPU and CPU. The Pi’s memory is shared. Here we set 256 MB for the GPU, you might want to reduce this on embedded systems which do not drive displays.

gpu_mem_256=64

sets the split of memory for Pi’s with only 256 MB of total memory (only some old generation Pi 1B’s)

[pi0]

conditional statement for Pi Zero; this covers Pi Zero and Pi Zero W
all the statemens until the next conditional are going to be applied only if we run on a Pi Zero / Zero W
the Pi Zero / Zero W still use an older ARMv6 ISA CPU (ARM11 @ 1 GHz)
we will use a kernel compiled for ARMv6

kernel=boot/vmlinuz-rpi

this specifies the kernel to load
this must be an uncompressed kernel image file
32 bit kernels are loaded to address 0x8000 by default

initramfs boot/initramfs-rpi

“the initramfs command specifies both the ramfs filename and the memory address to which to load it”
initramfs = initial RAM filesystem
this file can be extracted with 7Zip, for instance (extract twice)
NOTE: this configuration parameter is specified without the “=”

[pi1]

this will apply the following configuration lines to all Raspberry Pi 1 – will not match on Pi 2, etc.

kernel=boot/vmlinuz-rpi
initramfs boot/initramfs-rpi
[pi2]
kernel=boot/vmlinuz-rpi2

notice how we use a different kernel, since here we have a CPU which can run ARMv7 code

initramfs boot/initramfs-rpi2

and a different initramfs …

[pi3]

kernel=boot/vmlinuz-rpi2
initramfs boot/initramfs-rpi2

[pi3+]

kernel=boot/vmlinuz-rpi2
initramfs boot/initramfs-rpi2

[all]

the following line(s) will again apply to all Pi’s

include usercfg.txt

will include a file usercfg.txt into the configuration, which does not yet exist in the default Alpine image.
here you could put additional custom configuration which you want, e.g. HDMI resolution, etc.
alternatively you could just paste it below in config.txt

Note: the configuration used in Alpine does not specify a second parameter for the initramfs (address at which to mount the initramfs); thus it is not advisable to copy this 1:1. That being said, it seems to work, probably defaulting to a safe default!

The Kernel and the Initramfs

The kernel

Depending on the config.txt settings, the correct kernel will be selected and loaded into RAM. Once it is loaded, the ARM processor core(s) (are/is) released from reset so (it/they) can boot the kernel.

It will be passed the contents of cmdline.txt as kernel command line. start.elf will also pass additional parameters of it’s own, for instance setting DMA channels, the MAC address of the SMSC LAN chip, etc.

Get the cmdline.txt:

cat /proc/cmdline

dmesg | grep “Command line”

cmdline.txt

let’s have a look at Alpine’s default cmdline.txt:

modules=loop,squashfs,sd-mod,usb-storage quiet dwc_otg.lpm_enable=0 console=tty1

This loads the modules

loop

the loop device is used to mount a file as a file system

squashfs

compressed read-ony file system for Linux
especially suitable for embedded systems
https://en.wikipedia.org/wiki/SquashFS
this is used for modloop later on

sd-mod

SCSI disk driver / block driver
apparently copied from Alpine default configuration, e.g. here: https://wiki.alpinelinux.org/wiki/Create_a_Bootable_Compact_Flash

usb-storage

Linux USB mass storage driver

NB: module names may contain both – and _, these symbols can apparently be interchanged.

quiet

sets the default kernel log level to KERN_WARNING, which suppresses all but very serious log messages during boot (source: Raspberry Pi documentation)

dwc_otg.lpm_enable=0

The USB Controller driver, disables USB link power management with this setting. (This is the recommended default)
https://elinux.org/RPI_BCM2708_Parameters#module:_sdhci-bcm2708

console=tty1

defines the serial console

initramfs

” The initramfs command specifies both the ramfs filename and the memory address to which to load it. It performs the actions of both ramfsfile and ramfsaddr in one parameter. The address can also be followkernel (or 0) to place it in memory after the kernel image. Example values are: initramfs initramf.gz 0x00800000 or initramfs init.gz followkernel. NOTE: This option uses different syntax from all the other options, and you should not use a = character here.” – Raspberry Pi Documentation

There will be a kernel message like this “[ 0.143709] Trying to unpack rootfs image as initramfs…” in dmesg, showing that the image is being unpacked.

The initramfs allows the Linux kernel to add modules, and perform other tasks in preparing for the mounting of the actual file system. It thus makes for a very versatile boot system, without having to recompile the kernel / add overhead to the kernel. For instance for users with an encrypted filesystem, initramfs will ask for the passphrase before it can decrypt and mount the file systems.

Depending on the CPU in your Raspberry Pi, different initramfs files will be used, as set in config.txt:

boot/initramfs-rpi for Pi 1 / 1B+ and Pi Zero / Zero W, and Compute Module 1
boot/initramfs-rpi2 for all other Pi’s (2, 3B, 3B+, …)

The initramfs file can be extracted using, for instance, on Windows 7Zip. (You have to extract it twice: the first file contains a gzipped second file)

initramfs images can be in different formats, depending on which algorithms were compiled statically into the kernel (e.g. gzip / bzip2 / LZMA / XZ / LZO / LZ4).

This second file packages a folder and file structure, which you will recognize if you have used Linux before. It is in a special format, “cpio“, which is similar to tar.

if you are curious, you can read more about cpio here:

https://en.wikipedia.org/wiki/Cpio

This is what the file contains:

in bin, it contains:

busybox

compact executable containing several UNIX utilities in one application
especially for embedded systems
https://busybox.net/about.html

in sbin, it contains

Alpine package management tool

bootchartd

this is a tool for profiling startup time
https://elinux.org/Bootchart
http://www.bootchart.org/

nlplug-findfs

is used to mount block devices and search them for available apkovl’s, if no apkovl= is passed in the kernel command line
https://github.com/alpinelinux/mkinitfs

the init file is launched by the Linux kernel as first process, and basically sets up Alpine.

Here is the tool used for generating the initramfs for Alpine:

https://github.com/alpinelinux/mkinitfs

init

this is a sequential excerpt from what the init script launched from the initramfs does:

creates directories /usr/bin, /usr/sbin, /proc, /sys, /dev, /sysroot , /media/cdrom, /media/usb, /tmp, /run recursively

some of these directories seem to be present already in the initramfs

installs busybox (sets up symlinks to busybox’ embedded commands)
sets up PATH to /usr/bin:/bin:/usr/sbin:/sbin
creates /dev/null using mknod
mounts proc to /proc

proc is a process information pseudo-file system
you can obtain information about the kernel by reading and configure certain things by writing to some files in this directory
the numbered entries are directories containing information about the process with this particular process id
https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html

… and sysfs to /sys

this is information about hardware devices, device drivers and kernel subsystems
https://en.wikipedia.org/wiki/Sysfs

parses the options passed via cmdline.txt, the following are recognized:

alpine_dev autodetect autoraid chart cryptroot cryptdm cryptheader cryptoffset
cryptdiscards cryptkey debug_init dma init_args keep_apk_new modules ovl_dev
pkgs quiet root_size root usbdelay ip alpine_repo apkovl alpine_start splash
blacklist overlaytmpfs rootfstype rootflags nbd resume s390x_net dasd ssh_key

Tip: try changing “quiet” in cmdline.txt to “noquiet” to get a more verbose boot, and see many messages from init

mounts devtmpfs to /dev
loads drivers to be able to mount modloop later on
if the parameter nbd is present and configured (network block device), tries to bring up an ip address, and configure the network block device using nbd-client
if the root parameter is set

if [ -n “$KOPT_root” ]; then
–n evaluates if the length of “$KOPT_root” is nonzero
then nlplug-findfs is executed
overlaytmpfs is optionally handled
the current mount points are migrated to /sysroot
switch_root is run and control is passed to /sbin/init
(this is an entry point for installing Alpine Linux in a regular mode, not in read-only mode)

otherwise:

nlplug-findfs is executed to find the boot media,

crypto-options are possible and processed here
found apkovls’ paths are added to /tmp/apkovl (using the -a switch, see https://pi3g.com/2019/01/09/nlplug-findfs-documentation/)

apparently the file is only created if apkovl’s are found.

https://github.com/alpinelinux/mkinitfs/blob/master/nlplug-findfs.c
apkovls are matched looking for “*.apkovl.tar.gz*”, that is why backup files which lbu commit creates are not found / applied automatically – they have a structure of hostname.timestamp.tar.gz, e.g. pidoctor.20190110180745.tar.gz

if the parameter apkovl is set,

and it is empty – and /tmp/apkovls exists (that is, files were found by nlplug-findfs) – the ovl is set from the first line of /tmp/apkovls
and it starts with http:// / https:// / ftp:// – a network connection is tried to pull the apkovl
otherwise the ovl is set to the string in the apkovl option

if the apkovl exists,

and is a .gz file, the apkovl is unpackaged tar -C “$dest” -zxvf “$ovl” > $ovlfiles and /tmp/ovlfiles is filled in with tar’s output

there is code for a splash screen, fbsplash is one of the functions which is included in busybox – it displays fbsplash*.ppm files

https://github.com/brgl/busybox/blob/master/miscutils/fbsplash.c

if [ -z “$ALPINE_REPO” ] -> means, if the string $ALPINE_REPO is empty

under some circumstances default bootservices are added, for the runlevel sysinit, boot, shutdown and default (e.g. the service firstboot)

apk repositories are added and expanded

the package alpine-base depends on alpine-baselayout, alpine-conf and some other packages

it also includes the file /etc/os-release -> which shows the name, version, and pretty name of Alpine Linu

alpine-baselayout is the base directory layout
alpine-conf includes, amongst others, the lbu script and setup scripts
busybox includes the busybox binary
busybox-initscripts includes some initscripts

finally, the root directory is switched, and /sbin/init is started

this will start OpenRC and execute the runlevels
/sbin/init is a symlink to /bin/busybox

Please note: The apkovl is mounted before modloop and before installing the packages. This allows the apkovl to configure which packages are going to be installed, amongst other things.

OpenRC Runlevels

sysinit -> boot -> default -> shutdown

https://hoverbear.org/2014/09/29/init-runlevels-and-targets/

Runlevel sysinit:

modloop will be started after dev-mount, and before checkfs, fsck, hwdriver, modules, hwclock, dev, sysfs

hwdrivers will be started after modloop. hwdrivers will load the drivers using modprobe.

Runlevel boot:

Runlevel default:

Modloop

There is also the modloop file (modloop-rpi or modloop-rpi2).

how it is mounted

The right file for your system is determined in the script /etc/init.d/modloop in the function find_modloop()

The result of uname -r is checked against the directory modules/ – the appropriately named subdirectory needs to be in it.

For example: 4.14.69-0-rpi2

This way the right modules for your kernel are used (uname -r prints the kernel release).

modloop is mounted in the directory /.modloop

Symlinks are created from /lib/firmware to /lib/modules/firmware, and /lib/modules is symlinked to /.modloop/modules

Hint: if you want to patch the Raspberry Pi 3B+ WiFi firmware, you have to put it into modloop.

Modloop’s contents

To explore modloop’s contents, you can use (tested on a mint system):

apt-get install squashfs-tools

unsquashfs modloop-rpi2

It contains firmware and drivers.