Make Linux servers fun again!

Persistence by Choice

The immutable nature of Lightwhale offers clear advantages, but in order to install, configure, run containers, and write data, a writable filesystem is required. And for the system to be genuinely useful, such changes must persist across reboots.

The Data Filesystem

Lightwhale provides both temporary and persistent writability through an automated subsystem activated early during startup. This mounts the data filesystem at /mnt/lightwhale-data.

All data written by Lightwhale is kept within a single subdirectory: /mnt/lightwhale-data/lightwhale-state. This in turn serves as the writable upper layer in an overlayfs stack, with the immutable root as the lower layer.

By default, Lightwhale mounts a volatile tmpfs as its data filesystem. When persistence is enabled, the data filesystem instead resides on a storage device and is mounted accordingly.

Key Directories

The data filesystem overlay does not cover the entire root filesystem; that would defeat the purpose of immutability and Lightwhale altogether. Instead, the writable overlays apply only to a few strategic directories:

/etc

For customizing system configuration, including networking, password, and sshd settings.

/var

For log and other application data.

/home

For user account customization, including authorized SSH keys, and cloning Git repositories with Docker and Swarm stacks.

Docker Data

Docker is configured with its data root directory located directly on the data filesystem, where all Docker runtime data is stored, including images, containers, volumes, and network state:

/mnt/lightwhale-data/lightwhale-state/docker

Enable Persistence

Persistence must be enabled explictly by writing the magic header to the storage device to be used, e.g. /dev/sdx:

echo "lightwhale-please-format-me" | sudo dd conv=notrunc of=/dev/sdx

Multiple storage devices are supported to have a magic header written, and will be assembled into a Btrfs RAID1 volume.

The next time Lightwhale boots up, it will detect the magic disk, format it, and make it the data filesystem.

Managing Persistence

The persistence subsystem is initiated from /etc/init.d/S11persistence, and proceeds through a sequence of detailed steps, executed fully automatically:

1. Find data filesystem

Scan all disks for a partition with the filesystem label lightwhale-data.

If found, use it as the data filesystem and jump to step 6; otherwise proceed to step 2.

2. Find magic disks

Scan all disks for the magic header, specifically this exact byte sequence at the very start of the device: lightwhale-please-format-me.

If found, treat each as a magic disk and proceed to step 3; otherwise jump to step 6.

3. Create magic partitions

For each magic disk, create a swap partition labeled lightwhale-swap, then create a Linux partition that uses the remaining space and label it lightwhale-data. Then proceed to step 4.

4. Find magic partitions

Scan all disks for swap partitions labeled lightwhale-swap and Linux partitions labeled lightwhale-data. Treat each as a magic swap partition or magic data partition and proceed to step 5.

5. Create data filesystem

All magic swap partitions are formatted and labeled lightwhale-swap.

If only a single magic data partition exists, format it with btrfs --data single --metadata dup.
In case of multiple, join them into a RAID1 and format with btrfs --data raid1 --metadata raid1cn. Subvolumes are created for @lightwhale-data, @lightwhale-state, and @lightwhale-state-snapshots.

Label the data filesystem lightwhale-data, so it can be detected in step 1 at next startup.

6. Mount data filesystem and prepare state directory

If a data filesystem was created or found, mount its subvolume @lightwhale-data at /mnt/lightwhale-data; otherwise mount a tmpfs instead.

7. Mount overlays

Prepare the immutable lower layer: Bind mount /etc on /run/lightwhale/overlay/lower/etc, and mirror the entire directory tree of the immutable root filesystem.

Prepare the writable upper layer: If not present, create a directory on the writable data filesystem at /mnt/lightwhale-data/lightwhale-state/overlay/upper/etc.

Finally use overlayfs to virtually merge the two layers and mount the overlay filesystem at /etc. This effectively replaces the immutable directory with a writable version!

Repeat for remaining key directories /var and /home.