Guidelines for debugging EasyOS

June 15, 2023 — BarryK

Created: August 12, 2020
Updated: June 14, 2023; June 15, 2023

Making a start on writing this page...

EasyOS boots up from a boot-manager, such as Limine, rEFInd or GRUB, which loads the Linux kernel, file 'vmlinuz', and the tiny complete Linux OS into RAM, file 'initrd'. The 'init' script inside 'initrd' is then run, which does a lot of setting-up, especially creating the aufs layered filesystem, consisting of a read-write folder on top and file 'easy.sfs' on the bottom. There is then a switch_root onto the aufs layers, and hence to the desktop.

The 'initrd' is actually a "initramfs", and the latter is the word you will most likely come across if searching online for help about how it works. It really is a complete tiny Linux OS, commandline only. If anything goes wrong at bootup, or if you want to modify bootup behaviour, you will want to get inside the 'initrd'.

Opening up and editing the 'initrd'

You download EasyOS as a drive-image file, named "easy-<version>[-<language>]-<architecture>.img", for example "easy-5.4.1-amd64.img". This will be written to a drive, most often a USB-stick. In Easy, we have a nice utility to handle this, "easydd", or you could use the "dd" utility. In the target drive, you will find two partitions, a "boot partition" and a "working partition".

The boot-partition has the Limine boot-loader, that can boot on either UEFI-firmware or traditional BIOS computers. Here is a snapshot of the boot-partition, USB-stick, in this case it is /dev/sdc:

...the 'EFI' folder contains files that Limine needs for UEFI booting.

The working-partition contains the installation of EasyOS, which initially is just three files 'vmlinuz', 'initrd' and 'easy.sfs':

...notice the folder 'easyos'. That is an arbitrary name, it could be any name, or the installation could be deeper, for example 'easyos/kirkstone'.

Opening up and editing the contents of 'initrd' is simplicity itself -- just click on it. You can find some more details here:

https://easyos.org/tech/how-easy-works.html

You can, for example, edit the 'init' script. Having made changes, just click on 'initrd' again and it will be updated.

Debugging the 'initrd' at bootup

Perchance the 'initrd' doesn't do what you want, it is possible to dropout to a shell in the initramfs and poke around. If you examine the 'init' script, you will see that it recognises a kernel boot parameter of the format "qfix=<param1>[,<param2>...]", for example "qfix=nox" (which means bootup to commandline, without X. Note that you can have multiple, comma-separated parameters. At the time of writing, these are the recognised parameters:

lock1	Copy session to RAM & unmount all partitions
lock2	Copy session to RAM & disable drives
normal	Remove permanent lockdown, normal bootup
nox	Bootup to commandline, no X
fsck	Filesystem check of working-partition
back	Rollback to last snapshot
new	Rollback to pristine first-bootup
dropout0	Initrd shell: nothing yet mounted
dropout1	Initrd shell: boot & wkg partitions mounted
dropout2	Initrd shell: before recovery & maintenance operations
dropout3	Initrd shell: before version control
dropout4	Initrd shell: before setting up SFS layers
dropout5	Initrd shell: before moving mount points to layered f.s.
dropout6	Initrd shell: just before switch_root

Any of the parameters can be implemented by editing the kernel commandline in the 'limine.cfg' file in the boot-partition.

However, they can also be achieved via two other ways: a menu in the initrd, or the Shutdown menu. The latter assumes you have a working desktop of course:

...the last bottom-two entries have further windows. The "Reboot to initrd" entry will bring up this window:

...so, can achieve the same dropout to a shell in the initrd, as can be done by kernel commandline.

The way that these reboot options are created by the menu, is files are created in the working-partition that are read by the 'init' script in the initrd at next booup. For example, if you had chosen "Reboot to initrd (developers only)" from the Shutdown menu, a file '.debug.flg' gets created.

Dropout to a shell in the 'initrd' is extremely useful for debugging the 'initrd' at bootup.

A file .debug.flg will be created, with a parameter between "dropout1" and "dropout6" in it. The "Reboot later" button may be useful, as you could choose some other reboot option from the menu, such as "Reboot with rollback", and you will then be able to dropout to a shell in the initramfs to debug that mode of bootup.

If perchance X is not running, you can manually create .debug.flg, then reboot.

if the 'init' script hits an error, such that it is unable to continue bootup, it will also dropout to a shell in the initramfs. So, how to poke around inside the initramfs and attempt some debugging?

Debugging aids in the initramfs

the 'initrd' is based on Busybox, plus there are some other utilities, all statically linked. if you type "busybox<ENTER>" in the initramfs shell, you will see the list of applets available from busybox. Other utilities are, in /bin:

argon2 capsh, dialog, e2fsck, e4crypt, fdisk, fscryptctl, gpg, idump, 
mke2fs, mksquashfs, mp, readlink, resize2fs, shfm, tune2fs, vercmp

These are binary executables, except for 'shfm' which is a shell script. 'shfm' is particularly interesting, as it is a console file manager, to make it easy to navigate around in the filesystem. Here is a snapshot of 'shfm', running in a terminal emulator on the desktop ('shfm' is also in the main filesystem, after switch-root):

...very simple, probably more accurately described as a "file browser" than a file manager. Keyboard navigation by the arrow keys, "?" for help, "q" to quit, "!" to spawn a shell. Right-arrow or ENTER key on any text file will launch the 'mp' text editor.

Although Busybox has a simple 'vi' editor, 'mp' is way better, with drop-down menus. Here is a snapshot, in a terminal emulator on the desktop ('mp' is also available in the main filesystem):

After having poked around in the initramfs, type "exit" to continue bootup, to the desktop.

Files created by the Shutdown menu

The choices in the Shutdown menu may create a file that the 'init' script will read at bootup. File '.debug.flg' has already been mentioned. They are all located in the working-partition, for example /mnt/sdc2/easyos. Here is a list of all of them:

File	Purpose
.debug.flg	Exit to shell in initramfs
.lockdown.flg	Copy session to RAM
.fsckme.flg	Filesystem check of working-partition
.rollback.flg	Rollback
.qsfs.flg	Recompress 'easy'sfs' with gz
.zramtl.flg	Top-level zram or direct write to drive

You can examine the 'init' script to see the expected content of these files.

Menu in initrd

There is also a menu in the initrd, with a small number of choices:

Editing kernel commandline in boot-loader

It depends what boot-loader you have, but whatever, you need to append parameters for EasyOS. For example, the USB-stick has the Limine boot-loader, with file 'limine.cfg', with something like this in it:

VERBOSE=no
TIMEOUT=10
DEFAULT_ENTRY=1
INTERFACE_BRANDING=EasyOS Limine Boot Manager
INTERFACE_RESOLUTION=800x600

:EasyOS Kirkstone
    COMMENT=EasyOS Kirkstone bootup
    RESOLUTION=800x600
    PROTOCOL=linux
    KERNEL_PATH=boot://2/easyos/vmlinuz
    MODULE_PATH=boot://2/easyos/initrd
    KERNEL_CMDLINE=rw wkg_uuid=8666a128-0766-11ee-968f-287fcfeb4376 wkg_dir=easyos/

As documented earlier in this page, you can add "qfix=..." parameter, which in the above case would go onto the end of the "KERNEL_CMDLINE" parameter.

When installing EasyOS, or debugging an installation, you might need to change those "wkg_uuid" and "wkg_dir" parameters.

Both of those are essential to boot EasyOS. They tell the 'initrd' where the working-partition is and in what folder Easy is installed.

Note the trailing slash "wkg_dir=easyos/". That is not essential. "wkg_dir=easyos" would suffice.

"wkg_uuid" is the filesystem UUID, that you can obtain by running the 'blkid' utility, for example:

# blkid /dev/sdc2
/dev/sdc2: LABEL="easy2" UUID="8666a128-0766-11ee-968f-287fcfeb4376" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="89263718-02"

What is very useful to know, is that there are two alternatives to "wkg_uuid":

wkg_uuid=8666a128-0766-11ee-968f-287fcfeb4376
wkg_dev=sdc2
wkg_label=easy2

You can use any one of those. However, there is a trap: "wkg_uuid" and "wkg_label" must be unique. Usually, the UUID is unique, but there are situations where there might be two partitions with the same UUID. Same with the partition label. So, whichever you use, be sure that it is unique.

"wkg_dev" is another trap. If booting from a USB device, the device name might change, so you cannot guarantee that it will always be "sdc2". Usually the names assigned to the internal drives will remain constant.

Here is a blog post with further clarification:

https://bkhome.org/news/202206/complete-separation-between-boot-manager-and-easyos.html

Tags: dev