Make Linux servers fun again!

Persistence by Choice

The immutable nature of Lightwhale offers clear advantages,
but in order to install, configure, run containers, and write data, a writable filesystem is required.
And for the system to be genuinely useful,
such changes must persist across reboots.

The Data Filesystem

All data written by Lightwhale is kept within a single subdirectory:
/mnt/lightwhale-data/lightwhale-state.
This in turn serves as the writable upper layer in an

overlayfs
stack,
with the immutable root as the lower layer.

By default, Lightwhale mounts a volatile tmpfs
as its data filesystem.

When persistence is enabled,
the data filesystem instead resides on a storage device
and is mounted accordingly.

Key Directories

The data filesystem overlay does not cover the entire root filesystem;
that would defeat the purpose of immutability
and Lightwhale altogether.
Instead, the writable overlays apply only to a few strategic directories:

/etc

For customizing system configuration,
including networking, password, and sshd settings.

/var

For log and other application data.

/home

For user account customization,
including authorized SSH keys,
and cloning Git repositories with Docker and Swarm stacks.

Docker Data

Enable Persistence

Managing Persistence

The
persistence subsystem
is initiated from /etc/init.d/S11persistence,
and proceeds through a sequence of detailed steps, executed fully automatically:

1. Find data filesystem

Scan all disks for a partition with the filesystem label lightwhale-data.

If found, use it as the data filesystem and jump to step 6;
otherwise proceed to step 2.

2. Find magic disks

Scan all disks for the magic header,
specifically this exact byte sequence at the very start of the device:
lightwhale-please-format-me.

If found, treat each as a magic disk and proceed to step 3;
otherwise jump to step 6.

3. Create magic partitions

For each magic disk, create a swap partition labeled
lightwhale-swap,
then create a Linux partition that uses the remaining space and label it
lightwhale-data.
Then proceed to step 4.

4. Find magic partitions

Scan all disks for swap partitions labeled
lightwhale-swap
and Linux partitions labeled
lightwhale-data.
Treat each as a
magic swap partition
or
magic data partition
and proceed to step 5.

5. Create data filesystem

All magic swap partitions are formatted
and labeled lightwhale-swap.

If only a single magic data partition exists,
format it with
btrfs --data single --metadata dup.
In case of multiple,
join them into a RAID1 and format with
btrfs --data raid1 --metadata raid1cn.
Subvolumes are created for
@lightwhale-data,
@lightwhale-state,
and @lightwhale-state-snapshots.

Label the data filesystem lightwhale-data,
so it can be detected in step 1 at next startup.

6. Mount data filesystem and prepare state directory

If a data filesystem was created or found,
mount its subvolume @lightwhale-data
at /mnt/lightwhale-data;
otherwise mount a tmpfs instead.

7. Mount overlays

Prepare the immutable lower layer:
Bind mount /etc
on /run/lightwhale/overlay/lower/etc,
and mirror the entire directory tree of the immutable root filesystem.

Prepare the writable upper layer:
If not present, create a directory on the writable data filesystem at
/mnt/lightwhale-data/lightwhale-state/overlay/upper/etc.

Finally use overlayfs to virtually merge
the two layers and mount the overlay filesystem at /etc.
This effectively replaces the immutable directory
with a writable version!

Repeat for remaining key directories /var and /home.