Sometimes you may want to do something in isolation from your usual user account. For example, you might want to run an LLM harness without giving it access to basically everything on your system. At the same time, the setup needs to be practical enough that you can actually get some work done. One way to achieve this is to use systemd-nspawn.

Preparing the environment Link to heading

The containers are stored in /var/lib/machines. They are just plain directories,¹ into which you install your preferred Linux distro.

Installing Arch Link to heading

Let’s start by creating a container named jail:

/var/lib/machines$ mkdir jail
/var/lib/machines$ pacstrap -K -c jail base-devel zsh-completions micro mc sudo git-lfs less man-db clang llvm cmake ninja ccache mold git-delta bat eza duf fzf ripgrep fd github-cli gdb lldb docker act

I chose these specific packages because:

• base-devel – I want to use this for development, so plain base is not enough.
• zsh-completions – The shell I use.
• micro – My text editor (I no longer use Vim).
• mc, sudo, git-lfs, less, man-db – Basic necessities for development.
• clang, llvm – I prefer clang over gcc.
• cmake, ninja – The build system toolchain.
• ccache – I want my builds to be fast.
• mold – Fast linker.
• git-delta, bat, eza, duf, fzf, ripgrep, fd – Modern replacements for traditional Unix tools.
• github-cli – GitHub access.
• gdb, lldb – Debugging.
• docker, act – Running GitHub Actions CI locally.

Config file Link to heading

Persistent configuration is stored in the /etc/systemd/nspawn/ directory. Within that directory, create jail.nspawn² with the following contents:

[Exec]
Capability = CAP_NET_ADMIN CAP_BPF
PrivateUsers = no

[Files]
BindReadOnly = /sys/kernel/tracing

[Network]
VirtualEthernet = no

There are some potentially dangerous capabilities that are not available in the container by default. The two capabilities enabled by the Capability setting are needed for Docker to work.

I work on a profiler, so I want to be able to access certain profiling features of the operating system. I need to allow read-only access to the /sys/kernel/tracing directory on the host.³

Disabling the PrivateUsers option makes it easy to share files between the jail and the host system. Typically, you will have just one user on each side, so UIDs will match. We will look at this more closely later.

Turning the VirtualEthernet option off greatly simplifies the network configuration. The network interfaces remain shared. For example, you can bind a service to the host’s loopback interface and connect to it in the jail by accessing localhost, or vice versa.

Note that by default both PrivateUsers and VirtualEthernet are enabled if you’re using the .nspawn file, but disabled if you have a local jail that you access with the systemd-nspawn command. This may come as a surprise.

Starting and stopping the container Link to heading

At this point you can start the container by executing the following command as a regular (non-root) user. You will be prompted for your password, assuming your user is in the wheel group.

$ machinectl start jail

To see the boot process of the container, issue the following command:

$ journalctl -b -u systemd-nspawn@jail.service

To inspect the journal of the container itself, issue the following command:

$ sudo journalctl -b -M jail

You can further control the container’s status with the following commands:

$ machinectl restart jail
$ machinectl poweroff jail

Getting rid of the password prompts Link to heading

Entering the password every time you want to operate on a container quickly becomes tedious. To remove these prompts, create a file named 10-nopassword-machinectl.rules in /etc/polkit-1/rules.d/ containing the following script:

/* Allow members in 'wheel' to use machinectl without password authentication
 *
 * Thanks to the followings:
 * - https://unix.stackexchange.com/a/595725
 * - https://wiki.archlinux.org/index.php/Polkit#For_specific_actions
 */

polkit.addRule(function(action, subject) {
    if (
        (
            action.id == "org.freedesktop.machine1.shell" ||
            action.id == "org.freedesktop.machine1.manage-machines" ||
            action.id == "org.freedesktop.machine1.manage-images" ||
            action.id == "org.freedesktop.machine1.login" ||
            (
                action.id == "org.freedesktop.systemd1.manage-units" &&
                RegExp('systemd-nspawn@[A-Za-z0-9_-]+.service').test(action.lookup("unit")) === true
            )
        )
        && subject.isInGroup("wheel")
    ) {
        return polkit.Result.YES;
    }
});

Initial setup of the container Link to heading

To access the container, simply execute the following command:

$ machinectl shell jail

This will give you shell access to the root account inside the container. In there, perform the required setup, for example:

$ passwd
$ chsh
$ useradd -m -G wheel,docker wolf
$ passwd wolf
$ chsh wolf
$ micro /etc/sudoers

Then copy your config files, change the mc color scheme so it differs from the one on your host, remove the GPG signing config from your .gitconfig, and so on.

With everything properly set up, you can now access the user account on the container directly with the following command:

$ machinectl shell wolf@jail

Establishing file sharing Link to heading

There are two ways in which I have configured sharing files between the host and the container. The first one is simple: a shared directory where the specified host directory is bound at the same place in the container’s file system. Both sides can read and write in that directory. To enable it, open the .nspawn file and expand it by adding:

[Files]
Bind = /home/wolf/Sandbox

For my projects, I want a one-way isolation. That is, the container must reflect what I have on the host, because trying to make sense out of multiple remotes and branches across two copies of the same repository would quickly drive me mad. At the same time, no change made in the container is allowed to be visible on the host. If I want to make the changes permanent, I have to apply them manually. To have such a setup, add the following entries to the .nspawn file:

[Files]
Overlay = /home/wolf/tracy::/home/wolf/tracy
Overlay = /home/wolf/.cache/cpm::/home/wolf/.cache/cpm

Of course, the list of my projects is a bit longer. The .cache/cpm entry is used to share the CPM package manager cache, so that the libraries I depend on in my CMake scripts do not get downloaded twice.⁴

Now, what happens when some file is changed both on the host and in the container? That’s a very good question, and the answer is: don’t do that. Either way, the changes made in the container are ephemeral and do not survive a reboot of the container.

There’s another problem to solve here. Because of how overlays are designed, the overlay mount point is owned by root, not by the user. To fix this, create the /etc/rc.local file in the container with the following contents:

#!/bin/sh
chown wolf:wolf /home/wolf/*
chown wolf:wolf /home/wolf/.cache/cpm

Remember to make it executable with chmod a+x. Then go to the /etc/systemd/system/ directory and create the rc-local.service file containing:

[Unit]
Description = /etc/rc.local
ConditionPathExists = /etc/rc.local

[Service]
Type = forking
ExecStart = /etc/rc.local
TimeoutSec = 0
StandardOutput = tty
RemainAfterExit = yes

[Install]
WantedBy = multi-user.target

The service must be enabled by executing the following command:

$ systemctl enable rc-local.service

Now restart the container, and the overlay mounts should have the correct owner.

Docker setup Link to heading

The docker package was installed during the initial setup, and the user was added to the docker group when the account was created. What remains is to enable the docker socket by executing the following command:

$ systemctl enable --now docker.socket

You may need to set the value of /proc/sys/net/ipv4/ip_forward to 1 on the host system.

That should be enough to run CI actions:

$ act -l        # list workflows
$ act -j build  # select workflow to build

Summary Link to heading

And that’s it. The container is now ready to use.