A Process Gets Everything Until You Take It Away

Install nginx on a fresh Debian server and start it. Before it serves a single request, ask what that process is allowed to do. The honest answer is: nearly anything the kernel lets root do.

systemd-analyze security nginx

→ Overall exposure level for nginx.service: 9.6 UNSAFE 😨

That number is not a verdict on nginx, which is careful software. It is a verdict on the defaults. A service started by systemd gets almost no restrictions unless its unit file asks for them, and the packaged nginx unit asks for none. Whatever the process can do, anyone who takes over the process can do too.

("Almost" because there is one default: since v228 systemd gives every unit a TasksMax of 15% of the kernel's PID limit. It is a loose backstop against a fork bomb, not a fit to the service - the point below is that memory and CPU have no ceiling at all, and the tighter limits are yours to set.)

What a plain service is allowed to do

Without hardening, the nginx process tree on that host has:

No memory or CPU ceiling. A memory leak grows until the whole machine is short of memory. Every other service slows down under that pressure first, and when the kernel's OOM killer finally acts, it chooses from every process on the host - not necessarily the one that leaked. (Processes it can start are capped by the default TasksMax, but a service can still consume all the memory and CPU on the box.)
The whole filesystem. The nginx master runs as root, so it can write to /etc, /usr and /boot. The workers run as www-data, and they can still read every world-readable file on the machine and write into the /tmp that every other service shares.
Every process. ps aux from inside nginx lists everything running on the host, command lines included - and command lines carry passwords and tokens more often than they should.
Hundreds of syscalls, including the ones a web server will never make. Each one is kernel code an attacker can reach, and a kernel bug is how a foothold in one process becomes root on the host.
Every capability. Root is not one privilege but around forty capabilities: load kernel modules, trace any process, set the clock, mount filesystems. The master process holds all of them.

None of this is a bug. It is what "a process" means on Linux: it starts with everything its user has, and nothing is taken away unless something is configured to take it. Hardening a service is mostly the work of taking things away before someone else gets to use them.

A container is a process with things taken away

A container is not a small virtual machine. It is an ordinary process on the host's kernel, with a set of kernel features applied to it:

cgroups decide how much memory and CPU it can use, and how many processes it can start.
Mount namespaces decide which parts of the filesystem it sees, and whether it can write to them.
/proc options and PID namespaces decide which other processes it can see.
seccomp decides which syscalls it can make.
Capabilities decide how much of root's power it keeps.

Docker, containerd and the kubelet assemble these for you, together with an image and a network of its own. But the features belong to the kernel, and systemd exposes each of them as a directive in a unit file. A distro-packaged nginx - installed and patched by apt, configured in /etc/nginx, started at boot - can sit behind the same walls with no image and no container runtime.

Everything below goes in a drop-in, so the packaged unit file stays untouched:

systemctl edit nginx      # creates /etc/systemd/system/nginx.service.d/override.conf
systemctl restart nginx   # the sandbox is built when the process starts
systemctl revert nginx    # back to the stock unit, if you need it

systemctl edit reloads systemd's configuration when the editor closes, so there is no separate daemon-reload. The restart is still yours to do.

How much it can use: cgroups

[Service]
MemoryHigh=200M
MemoryMax=256M
CPUQuota=50%
TasksMax=100
Restart=on-failure

MemoryHigh= is the soft limit. Above it the kernel slows the service down and reclaims its memory aggressively, but kills nothing. It is the warning shot.
MemoryMax= is the hard limit. If usage can't be held under it, the OOM killer runs inside the unit - it picks from nginx's processes and nothing else on the host.
Restart=on-failure is here for what happens next. systemd's default OOMPolicy is stop, so when the OOM killer takes one worker, systemd stops the whole service - and the Debian unit sets no Restart=, so without this line an OOM would leave the site down until someone intervened. Restart=on-failure brings it back.
CPUQuota=50% is half of one CPU. 200% would be two full CPUs.
TasksMax=100 caps processes and threads, each thread counted individually. There is a default here already - 15% of the kernel's PID limit, thousands of tasks - so this line tightens a loose backstop down to what nginx actually needs.

The limits belong to the unit's cgroup, so they cover the master and every worker together, not each process separately. systemctl status shows them next to live usage:

Tasks: 2 (limit: 100)
Memory: 2.6M (high: 200M, max: 256M, available: 197.3M, peak: 2.7M)

And because systemd is only writing cgroup files, you can read them back directly:

cat /sys/fs/cgroup/system.slice/nginx.service/memory.max   # 268435456
cat /sys/fs/cgroup/system.slice/nginx.service/cpu.max      # 50000 100000
cat /sys/fs/cgroup/system.slice/nginx.service/pids.max     # 100

systemd-cgtop shows the same tree live, per service and per slice - the quickest way to find out what is actually eating a host.

This is the change that separates "one broken service" from "one broken server". A leak under MemoryMax= takes down nginx and nothing else - then Restart=on-failure brings it back, instead of the host-wide OOM killer picking a victim at random. It is also exactly what a Kubernetes resources.limits turns into on the node: the same memory.max and cpu.max files, written for the container's cgroup. Learn it on a systemd unit and you have learned it for pods.

What it can touch on disk: the mount namespace

ProtectSystem=strict
ReadWritePaths=/var/log/nginx /var/lib/nginx /run
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

systemd applies all of these inside a mount namespace it creates for the service. The host's view of the filesystem does not change; nginx's does.

ProtectSystem=strict mounts the entire filesystem read-only, except the API filesystems /dev, /proc and /sys - which the settings further down lock down separately.
ReadWritePaths= reopens only the directories nginx needs to write to.
ProtectHome=true makes /home, /root and /run/user empty and inaccessible.
PrivateTmp=true gives the service its own /tmp and /var/tmp, so it can neither read nor plant files in the ones other services use.
PrivateDevices=true replaces /dev with a minimal one - null, zero, random and the like - with no disks and no raw memory devices.

A mount applies to everything inside the namespace, root included, so this is one check a root shell can make honestly:

PID=$(systemctl show nginx -p MainPID --value)

ls -A /root                      # on the host
nsenter -t $PID -m ls -A /root   # inside nginx's mount namespace

.bash_history  .bashrc  .lesshst  .local  .profile  .ssh  .viminfo  .wget-hsts

The first command lists root's home directory, .ssh included. The second prints nothing at all. Keep that in mind for the /proc section: a mount changes what everyone in the namespace sees, and not every restriction works that way.

Read the config instead of guessing

ProtectSystem=strict only blocks writes, so the question is what nginx writes. Its own config answers it:

pid /run/nginx.pid;                  # needs /run
error_log /var/log/nginx/error.log;  # needs /var/log/nginx
access_log /var/log/nginx/access.log;
root /var/www/html;                  # only read - nothing to add

Add /var/lib/nginx, where the Debian and Ubuntu packages keep request-body and proxy temp files, and the list is complete. /run is broader than nginx needs, but the packaged config writes its pid file straight into it rather than into a subdirectory.

One trap turns a hardening change into an outage: every path in ReadWritePaths= must exist. systemd bind-mounts each one while building the namespace, and a missing path fails that step before nginx ever runs - the unit dies with status=226/NAMESPACE. The nginx.org packages keep their temp files in /var/cache/nginx, which the Debian package never creates, so copying a list from a guide written for the other package is enough to take the service down. Prefix a path with - and systemd skips it when it doesn't exist:

ReadWritePaths=/var/log/nginx /var/lib/nginx /run -/var/cache/nginx

Which processes it can see: /proc

ProtectProc=invisible
ProcSubset=pid

Neither setting creates a PID namespace - nginx still runs with its real host PIDs. What changes is the /proc mounted inside its mount namespace:

ProtectProc=invisible mounts it with hidepid=invisible: the /proc/<pid> directories of other users' processes are simply not there.
ProcSubset=pid mounts it with subset=pid: only per-process directories remain. /proc/meminfo, /proc/cpuinfo, /proc/sys and the rest of the system-wide files are gone. This one is worth applying deliberately - systemd notes it suits only simple services, and a process that reads /proc/meminfo or /proc/sys to size itself will break. Stock nginx doesn't; some modules and helper scripts do.

The detail that matters about hidepid: an ordinary root shell sees through it. systemd's documentation shortens this to "the root user is unaffected", but the mechanism is more precise - hidepid is a ptrace-permission check, so what bypasses it is CAP_SYS_PTRACE, which a normal root shell holds. That is why systemd says the option only bites a service running under User= and stripped of that capability. Both are true of nginx: the workers run as www-data, and the override below drops CAP_SYS_PTRACE from the whole service, so even the root master is covered. And the workers are the processes parsing whatever the internet sends.

The check that looked like proof

The obvious test is to step into nginx's namespaces and list processes:

PID=$(systemctl show nginx -p MainPID --value)
nsenter -t $PID -m -p ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0ps: Unable to get total memory

One process, PID 1. It reads like "nginx can only see itself". It shows nothing of the sort, for three separate reasons:

ps stopped on the first row. It printed USER, PID and %CPU, then needed the machine's total memory to compute %MEM. subset=pid had removed /proc/meminfo, so ps gave up. There could have been a thousand more rows.
PID 1 is the host's systemd. nginx has no PID namespace, so -p entered the host's, and PID 1 there is init - not nginx.
nsenter only enters namespaces. It does not pick up the service's user, capability bounding set, seccomp filter or cgroup. The shell that ran it was root with every capability - exactly the viewer hidepid hides nothing from.

A tool that exits early, a PID that belongs to something else, and a viewer the restriction doesn't apply to: any one of them makes the output meaningless. A useful check looks at the mount itself, and then looks through the eyes of the user the restriction is for:

grep ' /proc ' /proc/$PID/mountinfo
nsenter -t $PID -m -S $(id -u www-data) -G $(id -g www-data) ls /proc

209 207 0:49 / /proc rw,nosuid,nodev,noexec,relatime shared:174 - proc proc rw,hidepid=invisible,subset=pid
1869  1968  self  thread-self

The first line is the mount itself, and its last field reads hidepid=invisible,subset=pid - both settings made it into nginx's namespace. The second is /proc as www-data sees it, because -S and -G set the uid and gid used inside. Two PIDs remain: the nginx worker, and the ls doing the looking. No meminfo, no sys, not even nginx's own root master - and none of the other processes on the machine.

Then close the loop on the third reason by counting what root sees through the very same mount:

nsenter -t $PID -m ls /proc | grep -c '^[0-9]'   # inside nginx's namespace: 157
ls /proc | grep -c '^[0-9]'                       # on the host: 157

157 and 157. Same mount, same hidepid=invisible, and root sees every process through it - which is why the first test could never have shown anything, whatever the sandbox was doing.

Which syscalls it can make: seccomp

SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native

@system-service is a group systemd maintains: the syscalls a typical long-running service needs, and none of the ones for loading kernel modules, rebooting, mounting filesystems, setting the clock or calling ptrace(). (It is not a paranoid minimum - it still allows process_vm_readv, for instance - but it removes the whole dangerous tail.) systemd-analyze syscall-filter @system-service prints the exact list.

By default, a service that makes a filtered syscall is killed on the spot with SIGSYS. SystemCallErrorNumber=EPERM makes the call fail with "operation not permitted" instead, which is kinder while you are still discovering what a service needs - it logs an error rather than vanishing. SystemCallArchitectures=native limits the service to the host's own syscall ABI, so it can't reach the kernel through the 32-bit compatibility layer, historically a generous source of kernel bugs.

The rest of this group works the same way. Each line removes a way into the kernel that a web server has no use for:

RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
MemoryDenyWriteExecute=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true

RestrictAddressFamilies= allows IPv4, IPv6 and Unix sockets only. No netlink, no packet sockets, and no AF_ALG - the kernel crypto socket family Copy Fail was reached through.
RestrictNamespaces=true stops the service creating namespaces of its own, a common step in turning a kernel bug into root.
MemoryDenyWriteExecute=true forbids memory that is writable and executable at once, the classic landing spot for injected code. It also breaks anything that compiles code at runtime, so check before copying it into a Node.js or Java unit.
LockPersonality=true, RestrictRealtime=true and RestrictSUIDSGID=true block changing the kernel execution domain, taking realtime scheduling that could starve the host, and creating setuid or setgid files.

How much of root it keeps: capabilities

NoNewPrivileges=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE

CapabilityBoundingSet= is the most direct way to say "root, but not all of it". Every capability not on the list is gone - from the service and from everything it ever starts - even though the master still runs as uid 0. The four that remain each have a job:

CAP_NET_BIND_SERVICE - bind port 80, below 1024.
CAP_SETUID and CAP_SETGID - switch the workers to www-data.
CAP_DAC_OVERRIDE - open files root doesn't own, such as log files that belong to www-data.

Everything else - CAP_SYS_ADMIN, CAP_SYS_MODULE, CAP_SYS_PTRACE, CAP_NET_RAW and the rest - is removed for good. A process can drop capabilities from its bounding set, but nothing can add them back.

NoNewPrivileges=true closes the other road to more power: running a setuid binary such as sudo no longer raises privileges. It is the same kernel flag Kubernetes sets when a container says allowPrivilegeEscalation: false.

A last group removes the host-wide controls a service has no business touching, mostly by making their files read-only and dropping the matching capabilities:

ProtectKernelTunables=true - /proc/sys and /sys become read-only.
ProtectKernelModules=true - no loading or unloading kernel modules.
ProtectKernelLogs=true - no reading the kernel log buffer.
ProtectControlGroups=true - /sys/fs/cgroup becomes read-only, so the service can't raise its own limits.
ProtectClock=true and ProtectHostname=true - no changing the system clock or the hostname.

The whole override

[Service]
# how much it can use
MemoryHigh=200M
MemoryMax=256M
CPUQuota=50%
TasksMax=100
Restart=on-failure

# what it can touch on disk
ProtectSystem=strict
ReadWritePaths=/var/log/nginx /var/lib/nginx /run
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

# which processes it can see
ProtectProc=invisible
ProcSubset=pid

# which syscalls it can make
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
MemoryDenyWriteExecute=true
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true

# how much of root it keeps
NoNewPrivileges=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true

Keep comments on their own lines. Unit files have no trailing comments: PrivateTmp=true # private /tmp is read as the value true # private /tmp, which is not a boolean, so systemd logs a warning and ignores the whole line. The service starts, the setting silently isn't there, and nothing tells you unless you read the journal.

Then restart, and check both halves of the job:

systemctl restart nginx
systemd-analyze security nginx   # 9.6 UNSAFE -> about 2.5 OK
curl -I http://localhost

HTTP/1.1 200 OK
Server: nginx

The curl is not optional. A sandbox that stops the service doing its job is an outage with extra steps. The goal was never a locked-down process; it was a working process with nothing extra.

What stays red, and why that's right

The ✗ lines left in systemd-analyze security are worth reading, because some of them are meant to stay.

The service runs as root. The master starts as root, binds port 80, and switches its workers to www-data. You can go further: User=www-data together with AmbientCapabilities=CAP_NET_BIND_SERVICE lets an unprivileged process bind port 80 by itself. The cost is everything the master used to do as root - the pid file and log ownership need reworking, and nginx ignores its own user directive, with a warning, when it isn't started as root. That is a real trade, not a free win.

The service has access to the host's network. PrivateNetwork=true would give nginx a network namespace containing nothing but loopback. For a web server that isn't hardening, it is switching the service off.

Parts of @privileged are still allowed. @system-service keeps calls such as setuid(), setgid() and chown(). nginx needs the first two to drop its workers to www-data, and chown() to hand its temp directories under /var/lib/nginx to that user. That last one hides a trap worth knowing: the override drops CAP_CHOWN, so the chown() only succeeds on files the service already owns. It works here because Debian's first, un-sandboxed start already created /var/lib/nginx/* owned by www-data. Apply this override before nginx has ever started - on a fresh install where those directories don't exist yet - and it fails with chown(...) failed (1: Operation not permitted). Start nginx once, or add CAP_CHOWN, before locking it down.

A score of zero is not the target - zero is a service that can't do its job. The target is that everything still allowed is there because the service needs it, and you can say what for.

Try a setting before you commit it

You don't need to restart a production service to find out what a directive does. systemd-run starts a throwaway unit with whatever properties you pass and attaches it to your terminal:

systemd-run --pty -p ProtectSystem=strict touch /etc/test
# touch: cannot touch '/etc/test': Read-only file system

systemd-run --pty -p ProtectHome=true ls -A /root
# (nothing)

systemd-run --pty -p ProcSubset=pid cat /proc/meminfo
# cat: /proc/meminfo: No such file or directory

systemd-run --pty -p MemoryMax=64M -p MemorySwapMax=0 python3 -c 'b"x" * (512 * 1024**2)'
# OOM-killed inside the unit at 64M; nothing else on the host notices

Every line of the override can be tried this way first, one property at a time - much cheaper than learning about it from a unit that won't start.

Where this sits next to containers

The mechanisms are the same, and so are most of the settings, under different names:

MemoryMax=, CPUQuota= - resources.limits, both becoming memory.max and cpu.max.
TasksMax= - the kubelet's podPidsLimit, both becoming pids.max.
ProtectSystem=strict - readOnlyRootFilesystem: true.
CapabilityBoundingSet= - capabilities: { drop: ["ALL"], add: [...] }.
SystemCallFilter= - seccompProfile: { type: RuntimeDefault }.
NoNewPrivileges=true - allowPrivilegeEscalation: false.

What a container adds on top is mostly packaging: a root filesystem that comes from an image instead of the host, its own network namespace and IP, its own PID namespace, and a single artifact you can ship anywhere. What the systemd route keeps is the host's package - apt upgrade still patches nginx, its config still lives in /etc/nginx, and nothing about how the service runs changes except what it is allowed to do.

The takeaway

A Linux process starts with everything its user has, and a service started as root starts with nearly everything the kernel offers. That isn't broken; it is the default, and the default assumes the process will never be taken over.

Containers made taking things away routine, but the tools were never theirs. cgroups decide how much a service can use, mount namespaces what it can touch, /proc options what it can see, seccomp which kernel code it can reach, and the capability bounding set how much of root it keeps. systemd puts every one of them a drop-in away from any service already on the host - and systemd-analyze security plus a curl tell you whether you took away enough, and not too much.