Evicted on One Node, and Only That Node

A customer reports that pods are being evicted at random on one node, while every other node in the cluster behaves. You have SSH to the node. Before you touch it, notice that the complaint already contains two pieces of evidence, and they point in different directions.

Eviction is the kubelet's decision, not the scheduler's

The sequence matters, because getting it backwards sends you looking in the wrong component:

The kubelet on the node observes a resource running out - memory, disk, inodes, PIDs.
Under disk pressure it tries to reclaim node-level resources first: garbage collect dead pods and containers, then delete unused images. (Memory and PID pressure have no such node-level step - the kubelet goes straight to evicting.)
If that is not enough, it starts terminating pods on that node to get back under the threshold.
Only then does the scheduler get involved, placing the replacement pod somewhere.

The scheduler never does node-pressure eviction. It picks placements (and it can preempt - killing lower-priority pods to make room for a pending higher-priority one - which is a different mechanism, covered below). So if a customer asks "why did the scheduler kill my pod under memory pressure", the answer is that it didn't - a component on one specific node did, for reasons local to that node. That reframing is most of the diagnosis, and it is why your SSH session is the right place to be.

Step 4 is also why this repeats rather than resolves. More on that below.

"One node" is the strongest clue in the ticket

If the cause were the pod spec - a memory limit set too low, a container leaking, a missing request - it would follow the workload wherever it ran. Symptoms that travel with the workload are workload bugs.

This one doesn't travel. It stays on a node. So the variable is the node, and you are looking for something this node has that its peers don't:

a filesystem that has filled up over a long uptime,
container logs nothing is rotating,
an image cache that has grown unbounded,
a noisy neighbour pod without limits, sitting on this node by luck,
or a node that is simply different - a smaller disk, an older instance type, an odd member of an otherwise uniform pool.

That last one is worth checking early, because "one weird node" is very often "one node someone added by hand".

First move: ask which pressure fired

kubectl describe node <node>

Read the Conditions block. One of three will be True:

MemoryPressure - from the memory.available signal.
DiskPressure - from nodefs.available, nodefs.inodesFree, imagefs.available, or imagefs.inodesFree.
PIDPressure - from pid.available.

That single field tells you which reclaim path the kubelet took, and it costs one command. Pair it with the events, which survive the pod:

kubectl get events -A --field-selector reason=Evicted
kubectl get pods -A --field-selector status.phase=Failed

Evicted pods stay around as Failed objects until pod garbage collection prunes them, so they double as a history of what happened and when it started. (The Events, by contrast, expire on the API server's --event-ttl, one hour by default - so a quiet morning can erase the eviction events while the Failed pods remain.)

The kubelet's default hard thresholds are worth memorising, because they explain the timing:

memory.available < 100Mi
nodefs.available < 10%
nodefs.inodesFree < 5%
imagefs.available < 15%
imagefs.inodesFree < 5%

There is no default threshold for pid.available - PID pressure only fires if someone configured one. That is why a fork-bombing container can wedge a node without a single eviction event: the kubelet never had a rule that would fire. (One sharp edge: setting any evictionHard value replaces the whole default map, so a hand-tuned memory.available silently drops the disk and inode defaults unless you re-list them.)

Note also that the kubelet tries to reclaim before it evicts. Under disk pressure it runs container garbage collection and deletes unused images first; only if that doesn't clear the threshold does it start killing pods. (Image GC has its own trigger too - imageGCHighThresholdPercent, default 85 - but the two are separate mechanisms, so don't lean on a tidy "85 then 90" story.) The useful consequence is the same: if a node is evicting for disk, reclaim has already run and failed to free enough, which means the space is held by something it will not touch - running containers' writable layers, emptyDir volumes, or logs. That is a much sharper starting point than "disk is full".

Nothing about the victims is random

The usual shorthand is that the kubelet evicts by QoS class - BestEffort, then Burstable, then Guaranteed. That is a good predictor and the wrong mechanism, and the difference shows up exactly when you need it. The kubelet ranks pods by:

whether the pod's usage exceeds its requests,
then Priority,
then how far usage exceeds requests.

QoS falls out of that ranking rather than driving it. A BestEffort pod has no requests at all, so its usage always exceeds them - it is at the front of the queue by construction, every time. Burstable pods only join it once they climb past what they asked for. Guaranteed pods, and Burstable pods living inside their requests, are evicted last and only on Priority.

Which is precisely why the customer perceives randomness. If a deployment ships without resource requests, every one of its pods is permanently first in line, and which one dies is decided by whatever the node needed to reclaim at that moment. From inside the application, that looks arbitrary. From the node, it is deterministic.

Under disk pressure the ranking compares disk usage - local volumes, logs, and the container writable layer - against the pod's ephemeral-storage requests rather than memory. Same shape, different resource: a pod writing 40GB into an emptyDir is not being singled out, it is being measured. The one place the "usage vs requests" step drops out is inode or PID starvation, where there is nothing to request against, so the kubelet ranks by Priority alone.

Why the pod comes straight back

Here is the loop that makes this feel unfixable to the customer:

The scheduler places pods by requests. The kubelet evicts by actual usage.

A pod with no requests costs the scheduler nothing - it fits anywhere, including on a node whose real utilisation is at 95%. The scheduler's view of that node has plenty of room, because the room is defined by the sum of requests, not by free or df.

While the condition is actually firing, the node protects itself: the control plane taints it (node.kubernetes.io/disk-pressure and friends), so the scheduler places nothing new there, and the kubelet holds the condition for evictionPressureTransitionPeriod (5m by default) to avoid flapping. So the loop is not "instantly back onto the same node" - it is an oscillation. Pressure fires, the node is tainted, victims are evicted; usage drops, the condition clears, the taint lifts; requests-based scheduling refills the node it still sees as empty; usage climbs and pressure fires again. DaemonSet pods are the exception that does bounce straight back, because their controller tolerates those taints.

Missing requests are what open that gap. A nodeSelector, a node affinity rule, or a DaemonSet ties the pod to the node more directly, but even without any of those, requests-versus-usage alone is enough to keep the cycle turning.

Three things people call "eviction"

Getting these apart saves you from answering the wrong question:

Node-pressure eviction - the kubelet reclaims a node. The pod ends in Failed with reason Evicted, and its controller (if it has one) creates a replacement, not necessarily elsewhere. This is what we are debugging.
OOMKilled - a container exceeded its own memory limit and the cgroup OOM killer took it. The pod stays put, the container restarts, RESTARTS climbs, and there is no eviction event at all. Node memory may be perfectly healthy.
Preemption - the scheduler deletes lower-priority pods to fit a pending higher-priority one. Like node-pressure eviction it kills running pods, but it is driven by scheduling, not by a node running out of a resource.
API-initiated eviction - kubectl drain, a descheduler, or a cluster-autoscaler scale-down calling the Eviction API. This one respects PodDisruptionBudgets; node-pressure eviction does not.

If restart counts are climbing, you are on the OOMKilled path and should be reading limits. If pods are vanishing and reappearing, you are on this one - but note the two can overlap: a node-level OOM that fires before the kubelet reacts also shows as OOMKilled with climbing restarts, so on a pressured node, check the kernel log (dmesg) as well as the container's limits.

On the node: disk first

Disk is the most common cause and the one that best explains a single misbehaving node:

df -h                             # space on / and the container filesystem
df -i                             # inodes - exhausted inodes trigger DiskPressure too
sudo du -sh /var/lib/containerd   # image and layer cache
sudo du -sh /var/log/*            # host logs
sudo du -sh /var/log/pods/*       # container logs, if rotation is broken
sudo du -sh /var/lib/kubelet/pods/* | sort -h | tail   # emptyDir and local volumes
journalctl -u kubelet | grep -i evict     # the kubelet's own account

df -i earns its own line. Inode exhaustion produces DiskPressure with plenty of gigabytes free, and a team staring at df -h can spend an hour concluding the node is fine. Anything that creates millions of tiny files - a cache directory, a build workspace - gets there without ever filling the disk.

The last du is where you find the noisy neighbour: a single pod whose emptyDir or writable layer has swallowed the node. That is your answer to "why this node" - the pod happened to land here.

For memory pressure, be careful with free -h. The kubelet computes memory.available from the working set, which excludes inactive_file - the reclaimable inactive page cache - so a node showing almost no "free" memory can be entirely healthy. (Active file cache still counts, so this is not all page cache.) Trust the node condition and kubectl top node over the raw free column.

Stop the bleeding, then fix it

Order matters here, and the first command is the one that helps the customer immediately:

kubectl cordon <node>                 # nothing new lands here
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data   # if pods are still cycling

Cordon alone stops the churn from being fed. Drain moves the survivors off, and note the flags: it will not proceed past DaemonSet pods or pods with local data unless you say so, and --delete-emptydir-data genuinely deletes that data - confirm it is scratch before you pass it.

Then reclaim, on the node:

sudo crictl images                    # what is actually cached
sudo crictl rmi --prune               # drop unreferenced images
sudo journalctl --vacuum-size=500M    # if systemd journal is the hog

And uncordon once the condition clears. Clean up the debris while you are there:

kubectl delete pods -A --field-selector status.phase=Failed

Make it not happen again

The fixes divide neatly by owner, which is the useful part when you are writing the ticket up:

The node / platform side. Tune imageGCHighThresholdPercent and imageGCLowThresholdPercent for how fast this cluster churns images. Confirm container log rotation is actually configured - the kubelet's containerLogMaxSize (10Mi) and containerLogMaxFiles (5) are per container, and an application logging to a file inside the container bypasses them entirely. Give imagefs its own volume so an image cache cannot starve the root filesystem. And check whether this node is the same shape as its peers, because the odd node is usually the odd instance type or the one with the longest uptime.

The application side. Set resource requests on everything - and for the disk case, that means ephemeral-storage requests, not just CPU and memory. A LimitRange per namespace fills in defaultRequest where a manifest omits it, without editing every one (its min also rejects pods that ask for less). This is the change that makes the evictions stop looking random - a pod inside its requests is at the back of the queue instead of permanently at the front, and the scheduler stops treating the node as empty when it isn't.

The monitoring side. Alert on the node conditions - MemoryPressure, DiskPressure, PIDPressure - and on disk trending past 80%, not on pod restart counts. Restarts are how you find out afterwards; conditions are how you find out during.

The takeaway

Two words in the original report did the work. One node said the node is the variable, not the pod spec, which is what put you in an SSH session instead of a manifest. Randomly said the workload probably has no resource requests, which is what turns a one-off cleanup into a recurring incident.

And the ownership split follows from the same reading: the full disk is the platform's to fix, the missing requests are the application's - and if you only fix the first one, you will be back on this node in a month.