A Machine or a Process: systemd-nspawn, LXD and containerd

systemd-nspawn, LXD and containerd share a vocabulary - namespaces, cgroups, images, "containers" - and turn out to be far less interchangeable than that suggests. The quickest way to see why is to give each of them the same small job and watch them answer differently.

The job, as a running example: alice needs a shell on this server, and she must not be able to see the server - not its files, not its processes, not its other users. The previous post confined a service with namespaces, cgroups, seccomp and capabilities, one systemd directive at a time. A person is harder to confine than nginx - a person runs arbitrary programs - so this time the answer is a whole container. Which container, exactly, is the question the three tools disagree about.

Restricting a user on the host doesn't hold

The first instinct is to keep alice on the host and take things away: a restricted shell, a chroot, a ForceCommand that runs a menu. Every one of these has the same weakness. alice is still on the host's filesystem, in the host's process table, next to the host's users and setuid binaries. You are hiding parts of a world she lives in, and every program that can start a shell is a door: vim with :set shell=/bin/bash and :shell, less with !, the pager inside git, any interpreter. Each one has to be found and closed, and missing one is enough.

A container turns the problem inside out. Instead of hiding parts of the host, you give alice a separate world: her own root filesystem, her own PID 1, her own users, optionally her own network. A shell escape still works - and lands her in a shell inside the container, which is where she already was. To see the host she has to get out of the namespaces, and that takes a kernel bug or a misconfiguration rather than a clever vim command.

The one thing she still shares with the host is the kernel. That is the difference between a container and a VM, and it comes back near the end.

Who is allowed to open the container

Before building anything, one axis separates the three tools sharply, and it is the one that matters for a hostile user: who is allowed to enter the container in the first place? Whatever wires the shell in - an SSH ForceCommand, a login service, a menu - runs as alice, so alice needs the permission to enter, and how narrow that permission can be is a property of the tool.

Docker. Entering a container means talking to the Docker daemon, which means the docker group - and Docker's own documentation warns that the group grants root-level privileges. Anyone who can run docker can start a container with the host's / mounted inside it. The permission to enter is root on the host.
LXD. Similar shape, with an escape hatch. Full access to the LXD socket - the lxd group - is, in LXD's own words, to be given only to users you would trust with root. But LXD and Incus can also hand a user a confined project through a restricted group (Incus splits this out as incus for confined access versus incus-admin for full), so a locked-down user isn't automatically root.
systemd-nspawn. machinectl shell asks systemd-machined, which asks polkit before it opens anything - passing along which machine, which user and which program. A polkit rule can therefore say exactly "alice may open a shell as alice in alice-box", and nothing more.

That is the narrowest grant of the three, so nspawn is what the demo below builds - though the point of the demo is what a system container is and how it isolates, not the plumbing.

Building one by hand with nspawn

The host is Debian 13 (trixie) on arm64, with systemd 257. The first surprise comes before any container exists:

root@n2:~# machinectl list
-bash: machinectl: command not found

Having systemd is not having nspawn. On Debian and Ubuntu, systemd-nspawn and machinectl ship in a separate package, systemd-container:

apt install systemd-container debootstrap

A root filesystem

A container needs an OS tree to run. debootstrap builds a minimal Debian into a directory, and /var/lib/machines is where machinectl looks for machines:

debootstrap --include=dbus trixie /var/lib/machines/alice-box http://deb.debian.org/debian
du -sh /var/lib/machines/alice-box

I: Base system installed successfully.
339M    /var/lib/machines/alice-box

A whole Debian userspace in 339 MB, and nothing in it is running yet. --include=dbus adds a system bus inside the tree, which is how machinectl shell reaches the container's own systemd later on.

Run by hand, it is not a sandbox

Run a command in that tree with no options at all:

systemd-nspawn -q -D /var/lib/machines/alice-box /bin/sh -c \
  'hostname; ps aux; cat /proc/self/uid_map; grep -E "CapBnd|Seccomp:" /proc/self/status; ip -br addr'

alice-box
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1 33.3  0.1   2408  1484 pts/0    Ss+  23:16   0:00 /bin/sh -c hostname; ps aux; ...
root           3  0.0  0.3   6404  3496 pts/0    R+   23:16   0:00 ps aux
         0          0 4294967295
CapBnd:    00000000fdecafff
Seccomp:    2
lo               UNKNOWN        127.0.0.1/8 ::1/128
enp2s0           UP             172.16.141.159/24 fe80::b256:5b47:704e:f38b/64

Line by line, that is a container with a hole in it:

alice-box - its own hostname, so a UTS namespace.
Two processes, and the shell is PID 1 - a PID namespace. The host's 170-odd processes are not there.
0 0 4294967295 - the UID map is the identity. There is no user namespace, so root in here is root on the host: the same UID 0.
CapBnd: 00000000fdecafff - nspawn did drop capabilities: CAP_SYS_MODULE, CAP_SYS_RAWIO, CAP_SYS_TIME, CAP_NET_ADMIN, CAP_BPF and others are gone. But 26 remain, and CAP_SYS_ADMIN and CAP_SYS_PTRACE are among them.
Seccomp: 2 - a syscall filter is in place.
enp2s0 … 172.16.141.159/24 - that is the host's network card and the host's address. There is no network namespace at all.

The systemd-nspawn documentation is blunt about what that combination means: the sandbox "can easily be circumvented from within the container if user namespaces are not used", and "untrusted code must always be run in a user namespace". With no networking options, the container also has full access to the host's network.

Started as a machine, it is

machinectl start doesn't run that command line. It starts the template unit systemd-nspawn@.service, which adds the options that matter:

ExecStart=systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i

--boot runs the tree's own systemd as PID 1 - a system container, not a single command.
-U turns on user namespaces, with a private range of host UIDs picked for the container.
--network-veth gives the container its own network namespace, joined to the host by a virtual Ethernet pair.

Start the same tree that way:

machinectl start alice-box
machinectl list

MACHINE   CLASS     SERVICE        OS     VERSION ADDRESSES
alice-box container systemd-nspawn debian 13      -

1 machines listed.

Then ask the same questions from inside, as root in the container:

machinectl -q shell alice-box /bin/sh -c 'cat /proc/self/uid_map; ip -br addr; ps -e --no-headers | wc -l; uname -r'

         0 1790443520      65536
lo               UNKNOWN        127.0.0.1/8 ::1/128
host0@if3        DOWN
10
6.12.107+deb13-arm64

0 1790443520 65536 - root in here is UID 1790443520 out there, and the container owns the 65,536 UIDs from that point on. A process that breaks out as "root" arrives on the host as an unprivileged UID nobody else uses.
host0@if3 DOWN - a network interface of its own: the container's end of the veth pair, with nothing configured on it. The host's enp2s0 is gone. Bringing it up takes systemd-networkd on both ends - 80-container-ve.network configures the host side, 80-container-host0.network (DHCP on host0) the container side - and Debian doesn't enable networkd by default, so a debootstrapped tree comes up with no network at all. For a shell box that may be exactly right; if alice needs egress, enable systemd-networkd inside the container.
10 - its whole process table, counting the shell and ps that did the counting.
6.12.107+deb13-arm64 - the host's kernel. Hold on to that one.

And from the host:

LEADER=$(machinectl show alice-box -p Leader --value)
ps -o user,pid,cmd -p "$LEADER"
lsns -p "$LEADER"

USER         PID CMD
vu-alic+   11433 /usr/lib/systemd/systemd
        NS TYPE   NPROCS   PID USER           COMMAND
4026531834 time      171     1 root           /sbin/init
4026532571 cgroup      6 11433 vu-alice-box-0 /usr/lib/systemd/systemd
4026532572 user        6 11433 vu-alice-box-0 /usr/lib/systemd/systemd
4026532573 mnt         5 11433 vu-alice-box-0 /usr/lib/systemd/systemd
4026532574 uts         5 11433 vu-alice-box-0 /usr/lib/systemd/systemd
4026532575 ipc         6 11433 vu-alice-box-0 /usr/lib/systemd/systemd
4026532576 pid         6 11433 vu-alice-box-0 /usr/lib/systemd/systemd
4026532577 net         6 11433 vu-alice-box-0 /usr/lib/systemd/systemd

The container's PID 1 is host PID 11433, an ordinary process owned by vu-alice-box-0 - the name systemd-machined publishes for that UID through its userdb service, which nss-systemd resolves. (Since systemd 246, the older nss-mymachines only maps hostnames.) lsns lists the namespace half of what a container is made of: new cgroup, user, mount, UTS, IPC, PID and network namespaces. The time namespace is still the host's, shared with the host's /sbin/init and 170 other processes.

The counts hold one more detail. Six processes share the container's user, PID and network namespaces, but only five share its mount and UTS namespaces. Listing each process's namespaces from inside finds the sixth:

machinectl -q shell alice-box /bin/sh -c \
  'for p in /proc/[0-9]*; do echo "$(readlink $p/ns/uts) $(readlink $p/ns/mnt) $(cat $p/comm)"; done' | sort

uts:[4026532570] mnt:[4026532702] systemd-logind
uts:[4026532574] mnt:[4026532573] agetty
uts:[4026532574] mnt:[4026532573] cron
uts:[4026532574] mnt:[4026532573] dbus-daemon
uts:[4026532574] mnt:[4026532573] systemd
uts:[4026532574] mnt:[4026532573] systemd-journal

systemd-logind runs in a mount and UTS namespace of its own, because its unit file uses the same kind of sandboxing directives as the nginx override in the previous post. A sandboxed service, inside a container, on one kernel.

And because the machine is a systemd unit, everything from the sandboxing post applies to it directly:

systemctl set-property systemd-nspawn@alice-box.service MemoryMax=512M TasksMax=512 CPUQuota=100%
systemctl status systemd-nspawn@alice-box.service

● systemd-nspawn@alice-box.service - Container alice-box
    Drop-In: /etc/systemd/system.control/systemd-nspawn@alice-box.service.d
             └─50-CPUQuota.conf, 50-MemoryMax.conf, 50-TasksMax.conf
     Active: active (running) since Mon 2026-09-14 23:16:24 EDT; 29s ago
   Main PID: 11430 (systemd-nspawn)
     Status: "Container running: Ready."
      Tasks: 7 (limit: 512)
     Memory: 13.3M (max: 512M, available: 498.6M, peak: 17.3M)
        CPU: 225ms

A booted Debian - its own systemd and a handful of services - in 13.3 MB, after 225 ms of CPU. The seventh task is systemd-nspawn itself, which sits in the unit's cgroup but outside the container's namespaces. set-property applied the limits to the running machine immediately and wrote them to drop-ins under /etc/systemd/system.control/, so they survive a restart.

The narrow grant

Give the container an alice account, then write the one rule that lets her - and only her - open a shell in it. polkit isn't in a minimal Debian, so install it first (apt install polkitd):

machinectl -q shell alice-box /usr/sbin/useradd -m -s /bin/bash alice

// /etc/polkit-1/rules.d/50-alice-box.rules
polkit.addRule(function (action, subject) {
  if (
    action.id == "org.freedesktop.machine1.shell" &&
    subject.user == "alice" &&
    action.lookup("machine") == "alice-box" &&
    action.lookup("user") == "alice"
  ) {
    return polkit.Result.YES
  }
})

That is the whole access-control story, and it is worth testing directly - exactly one of these should get through:

runuser -u alice -- machinectl --no-ask-password shell alice@alice-box /usr/bin/id
runuser -u alice -- machinectl --no-ask-password shell root@alice-box /usr/bin/id
runuser -u ryan  -- machinectl --no-ask-password shell alice@alice-box /usr/bin/id

Connected to machine alice-box. Press ^] three times within 1s to exit session.
uid=1000(alice) gid=1000(alice) groups=1000(alice)
Failed to get shell PTY: Interactive authentication required.
Failed to get shell PTY: Interactive authentication required.

alice asking for alice gets a shell - uid=1000(alice) is the container's alice, from the container's own /etc/passwd. alice asking for root, and ryan asking for alice, are both refused. That is the narrow grant the comparison above pointed at: one machine, one user, and nothing else.

Wiring this to a login is the demo's least interesting part: an sshd ForceCommand of machinectl -q --no-ask-password shell alice@alice-box in a Match User alice block, a key-only host account, and DisableForwarding yes so the connection can't carry a tunnel past the forced command. None of that changes what a container is - which is the actual subject - so I'll leave the SSH knobs as a footnote and look instead at what the shell finds once it is inside.

What the shell finds inside

Open it as alice - machinectl -q shell alice@alice-box - and run top:

alice@n2:~$ top
top - 23:23:20 up  2:21,  0 users,  load average: 0.17, 0.08, 0.03
Tasks:   9 total,   1 running,   8 sleeping,   0 stopped,   0 zombie
MiB Mem :    961.9 total,    142.0 free,    230.4 used,    676.6 buff/cache

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0   23180  12816  10016 S   0.0   1.3   0:00.06 systemd
     16 root      20   0   34776  11500  10392 S   0.0   1.2   0:00.01 systemd-journal
     70 root      20   0    4392   2524   2280 S   0.0   0.3   0:00.00 cron
     71 message+  20   0    8064   4000   3504 S   0.0   0.4   0:00.00 dbus-daemon
     72 root      20   0   18620   7724   6652 S   0.0   0.8   0:00.00 systemd-logind
     81 root      20   0    4764   2228   1992 S   0.0   0.2   0:00.00 agetty
    124 alice     20   0    4600   3732   3176 S   0.0   0.4   0:00.00 bash
    126 alice     20   0   23908   3644   2004 S   0.0   0.4   0:00.00 (sd-pam)
    133 alice     20   0    7544   4956   2880 R   0.0   0.5   0:00.00 top

Nine tasks: the container's systemd as PID 1, its journal, cron, D-Bus, logind, a getty, and alice's own three. The host's 170 processes don't exist from here.

Two things in that screen are not what they seem.

The header is the host's. 961.9 total is the host's memory, not the container's 512M limit, and the uptime and load average are the host's too. /proc/meminfo and its neighbours are not namespaced, so a process in a container reads the real kernel's numbers. The limit is still enforced - past 512M the OOM killer acts inside the unit - it just isn't what top and free report. LXD ships LXCFS to present container-sized versions of these files; nspawn doesn't.

The prompt says alice@n2. Run by hand, the container was called alice-box. Booted, its systemd took the hostname from /etc/hostname in the tree - and debootstrap had copied that file from the host. Nothing leaked, but a user who sees the host's name in her prompt will assume she is on the host, and so will an administrator reading over her shoulder. Fix it in the tree:

echo alice-box > /var/lib/machines/alice-box/etc/hostname
machinectl reboot alice-box

Before the fix, both /etc/hostname on the host and the copy in the tree read n2. After it, open the shell again:

alice@alice-box:~$ hostname; id; ls -A /home; ls /var/lib/machines; cat /proc/self/uid_map; free -m | head -2
alice-box
uid=1000(alice) gid=1000(alice) groups=1000(alice)
alice
ls: cannot access '/var/lib/machines': No such file or directory
         0 1790443520      65536
               total        used        free      shared  buff/cache   available
Mem:             961         230         141           0         676         731

alice-box - the prompt tells the truth now.
/home holds only alice. On the host it holds alice and ryan; from in here, ryan doesn't exist.
/var/lib/machines doesn't exist. The directory her whole world lives in can't be seen from inside it.
0 1790443520 65536 - the same map root saw, so alice's UID 1000 is host UID 1790444520.
961 total - still the host's memory, exactly as top reported.

What each tool thinks a container is

Everything above - a PID namespace, a mount namespace, a user namespace, a veth pair, cgroup limits, a seccomp filter, a trimmed capability set - is the same list of kernel features the nginx sandbox used, and the same list LXD and containerd use. None of these tools has an isolation mechanism the others lack. They differ in what they wrap around the mechanism, and in their answer to one question: is a container a machine or a process?

A system container is a machine. It runs a full OS userspace with its own init, several services and several users. It is long-lived, you log in to it, and you upgrade packages inside it. It behaves like a lightweight VM, and alice's box is one.

An application container is a process. It runs one program - or a small tree of them - from an image, usually with no init at all. Its lifetime is the process's lifetime. You don't upgrade it; you build a new image and replace it. Kubernetes runs these by the thousand.

systemd-nspawn: the minimal one

nspawn comes from the systemd project and stays close to it. A container is a directory or a disk image under /var/lib/machines; systemd-machined (which is a daemon, with D-Bus and, since v257, Varlink APIs) registers it, machinectl manages it, a template unit runs it, journalctl -M reads its logs and systemctl set-property limits it. What it doesn't have is the application-container machinery: no registry client of its own in the versions most distros ship (v260 added OCI pulls via importctl), and its unit of distribution is a filesystem tree or a disk image, not a stack of content-addressed layers.

That minimalism is both the appeal and the limit. It fits a handful of long-lived machines on a host that already runs systemd, and build or test environments. It does not fit hundreds of anything. And as the hand-run test showed, its safe defaults live in machinectl start, not in systemd-nspawn itself.

LXD and Incus: the system-container manager

LXD is built for the same kind of container as nspawn and wraps far more around it: a daemon with a REST API and the lxc client, image servers to launch from, storage pools with snapshots, profiles, managed networks, clustering across hosts - and it runs virtual machines as well as containers, from the same command line. Containers are unprivileged by default, meaning they run in a user namespace, and LXD's documentation is explicit that a privileged container, where UID 0 inside is UID 0 on the host, can be escaped.

For alice, LXD is the most comfortable of the three: a box that feels like a real server, with snapshots and limits behind a clean interface. The catch is the entry grant from earlier - full socket access is root-equivalent, so a locked-down user needs a confined project through the restricted group rather than the lxd group itself.

Since 2023 there are two of them. Canonical, which created LXD and was its main contributor, moved it out of the Linux Containers project on 4 July 2023 and under its own infrastructure; the community fork, Incus, joined Linux Containers the following month. Commands and concepts are still close to identical.

containerd: the application-container engine

containerd is built for the other kind entirely. It was born at Docker in 2014 as the layer underneath the Docker engine, joined the CNCF in 2017 and graduated in 2019. Its job is the life of image-based containers: pull and unpack OCI images, keep them in a content store, create containers from them, and supervise those containers while they run.

It doesn't create namespaces itself. The chain on a Kubernetes node is:

kubelet → CRI → containerd → containerd-shim-runc-v2 → runc → kernel

The kubelet talks to containerd over the Container Runtime Interface. Kubernetes removed its built-in Docker support, dockershim, in 1.24; since then a node runs a CRI runtime such as containerd or CRI-O directly.
containerd's CRI plugin creates the pod's network namespace and configures it with CNI plugins. containerd's core has no networking of its own - under Docker, dockerd sets up the network.
The shim is a small process - one per pod under the CRI runtime. It invokes runc with the container's configuration and stays behind as the containers' parent, so they survive a containerd restart.
runc is the OCI runtime that actually asks the kernel for namespaces and cgroups, starts the process, and exits.

Nothing in that chain expects a human to log in. A container is meant to do one job, log to stdout, and be replaced rather than repaired. You can keep a container alive with sleep infinity and exec into it, but alice would get a machine with no init and no services, behind a group that is root on the host.

Side by side

	systemd-nspawn	LXD / Incus	containerd
A container is	a machine, or one command	a machine - or a VM	a process from an image
Init inside	the tree's systemd, with `--boot`	the image's own	usually none
Images	a directory or a disk image	image servers, snapshots	OCI images from registries
Driven by	`machinectl`, systemd units	`lxc` or `incus`, a REST API	the kubelet over CRI, `ctr`, Docker
User namespaces	only with `-U` - `machinectl start` adds it	on by default	off by default
Who may enter	polkit: per machine, user and program	`lxd`/`incus-admin` (full); confined projects for others	root, or the `docker` group via Docker
For alice's shell	yes, with a narrow polkit rule	yes, the most comfortable	no

Which one for alice

nspawn, if the host already runs systemd and you want the narrowest grant: a polkit rule naming one machine and one user, -U and a private network from machinectl start, and resource limits from systemctl set-property. You assemble more of it by hand.
LXD or Incus, if alice's box should feel like a real server, with images, snapshots and a proper management layer - entering it through a confined project rather than the root-equivalent lxd group.
containerd or Docker, not for this. They are excellent at what they were designed for, and a person's shell isn't it.

A container is not a VM

There is exactly one kernel on the host, and alice's box is running on it. Every syscall she makes goes through the host's syscall table, so the container boundary is only as strong as that kernel: a privilege escalation bug reachable from inside a container is a host compromise. Copy Fail and Dirty Frag were exactly that. User namespaces, seccomp and dropped capabilities shrink how much of the kernel is reachable. They do not give the container a kernel of its own.

A VM does: its own kernel on virtual hardware, so a guest kernel bug stays in the guest. The price has traditionally been weight - a full boot, a fixed slice of memory - which is what microVMs go after. Firecracker, the VMM behind AWS Lambda, boots a minimal guest in a fraction of a second, and Kata Containers wires that into container tooling. LXD reaches the same place from the other side, launching a VM with the same command it uses for a container.

For alice, a system container with user namespaces is a sound boundary against a user who might poke around. If alice is a stranger running whatever she likes on shared hardware, give her a kernel of her own.

Why the names are confusing

Two lineages that started in the same place, and systemd's own tool beside them:

2008 - LXC is released: userspace tools for the namespaces and cgroups that IBM and others were adding to the kernel.
2013 - Docker launches, running its containers through LXC.
2014 - Docker 0.9 replaces LXC with its own Go library, libcontainer; the LXC driver is gone entirely by Docker 1.10. containerd starts inside Docker the same year.
2015 - Docker donates its container format and runtime, runC, to the new Open Container Initiative. LXD, Canonical's manager on top of LXC, is part of the Linux Containers project.
2017 - containerd joins the CNCF, and graduates in 2019.
2022 - Kubernetes 1.24 removes dockershim; nodes talk CRI to containerd or CRI-O directly.
2023 - Canonical takes LXD in-house, and the community fork, Incus, takes its place in Linux Containers.

So LXC → LXD → Incus is the system-container line, libcontainer → runc and containerd is the application-container line, and nspawn sits beside both as part of systemd.

The takeaway

systemd-nspawn, LXD and containerd don't differ in how they isolate. They ask the same kernel for the same namespaces, cgroups and filters. They differ in what they believe a container is - a machine you log in to, or a process you replace - and in how much machinery they build around that belief.

The shell that can't see the host was the running example, and it lands on the machine end: a system container, started with user namespaces and a network of its own. The demo's one lasting lesson beyond "it isolates" is that the permission to enter the container is itself part of the boundary - as narrow as a polkit rule for one user, or as wide as a group that is root on the host, depending on which tool you picked. Everything else about wiring in a shell is plumbing; the choice of container is the design.