Evicted on One Node, and Only That Node

A customer reports that pods are being evicted at random on one node, while every other node in the cluster behaves. You have SSH to the node. Before you touch it, notice that the complaint already contains two pieces of evidence, and they point in different directions.

Eviction is the kubelet's decision, not the scheduler's

The sequence matters, because getting it backwards sends you looking in the wrong component:

The kubelet on the node observes a resource running out - memory, disk, inodes, PIDs.
It tries to reclaim node-level resources first: garbage collect dead pods and containers, then delete unused images.
If that is not enough, it starts terminating pods on that node to get back under the threshold.
Only then does the scheduler get involved, placing the replacement pod somewhere.

The scheduler never evicts anything. It picks placements. So if a customer asks "why did the scheduler kill my pod", the answer is that it didn't - a component on one specific node did, for reasons local to that node. That reframing is most of the diagnosis, and it is why your SSH session is the right place to be.

Step 4 is also why this repeats rather than resolves. More on that below.

"One node" is the strongest clue in the ticket

If the cause were the pod spec - a memory limit set too low, a container leaking, a missing request - it would follow the workload wherever it ran. Symptoms that travel with the workload are workload bugs.

This one doesn't travel. It stays on a node. So the variable is the node, and you are looking for something this node has that its peers don't:

a filesystem that has filled up over a long uptime,
container logs nothing is rotating,
an image cache that has grown unbounded,
a noisy neighbour pod without limits, sitting on this node by luck,
or a node that is simply different - a smaller disk, an older instance type, an odd member of an otherwise uniform pool.

That last one is worth checking early, because "one weird node" is very often "one node someone added by hand".

First move: ask which pressure fired

kubectl describe node <node>

Read the Conditions block. One of three will be True:

MemoryPressure - from the memory.available signal.
DiskPressure - from nodefs.available, nodefs.inodesFree, imagefs.available, or imagefs.inodesFree.
PIDPressure - from pid.available.

That single field tells you which reclaim path the kubelet took, and it costs one command. Pair it with the events, which survive the pod:

kubectl get events -A --field-selector reason=Evicted
kubectl get pods -A --field-selector status.phase=Failed

Evicted pods stay around as Failed objects until pod garbage collection prunes them, so they double as a history of what happened and when it started.

The kubelet's default hard thresholds are worth memorising, because they explain the timing:

memory.available < 100Mi
nodefs.available < 10%
nodefs.inodesFree < 5%
imagefs.available < 15%

There is no default threshold for pid.available - PID pressure only fires if someone configured one. That is why a fork-bombing container can wedge a node without a single eviction event: the kubelet never had a rule that would fire.

Note also that image garbage collection starts earlier than eviction: imageGCHighThresholdPercent defaults to 85, so the kubelet begins deleting unused images at 85% disk usage and only starts evicting pods at 90% (10% available). If a node is evicting for disk, image GC has already run and failed to free enough - which means the space is held by something GC will not touch. That is a much sharper starting point than "disk is full".

Nothing about the victims is random

The usual shorthand is that the kubelet evicts by QoS class - BestEffort, then Burstable, then Guaranteed. That is a good predictor and the wrong mechanism, and the difference shows up exactly when you need it. The kubelet ranks pods by:

whether the pod's usage exceeds its requests,
then Priority,
then how far usage exceeds requests.

QoS falls out of that ranking rather than driving it. A BestEffort pod has no requests at all, so its usage always exceeds them - it is at the front of the queue by construction, every time. Burstable pods only join it once they climb past what they asked for. Guaranteed pods, and Burstable pods living inside their requests, are evicted last and only on Priority.

Which is precisely why the customer perceives randomness. If a deployment ships without resource requests, every one of its pods is permanently first in line, and which one dies is decided by whatever the node needed to reclaim at that moment. From inside the application, that looks arbitrary. From the node, it is deterministic.

Under disk pressure the ranking uses disk usage - local volumes, logs, and the container writable layer - rather than memory. Same shape, different resource: a pod writing 40GB into an emptyDir is not being singled out, it is being measured.

Why the pod comes straight back

Here is the loop that makes this feel unfixable to the customer:

The scheduler places pods by requests. The kubelet evicts by actual usage.

A pod with no requests costs the scheduler nothing - it fits anywhere, including on a node whose real utilisation is at 95%. The scheduler's view of that node has plenty of room, because the room is defined by the sum of requests, not by free or df. So the evicted pod is rescheduled, quite possibly right back onto the node that just evicted it, where the kubelet finds it over threshold again.

Missing requests are what open that gap. A nodeSelector, a node affinity rule, or a DaemonSet will pin the pod there outright, but even without any of those, requests-versus-usage alone is enough to keep the cycle turning.

Three things people call "eviction"

Getting these apart saves you from answering the wrong question:

Node-pressure eviction - the kubelet reclaims a node. The pod ends in Failed with reason Evicted, and a new pod object is created elsewhere. This is what we are debugging.
OOMKilled - a container exceeded its own memory limit and the cgroup OOM killer took it. The pod stays put, the container restarts, RESTARTS climbs, and there is no eviction event at all. Node memory may be perfectly healthy.
API-initiated eviction - kubectl drain, a descheduler, or a cluster-autoscaler scale-down calling the Eviction API. This one respects PodDisruptionBudgets; node-pressure eviction does not.

If restart counts are climbing, you are on the OOMKilled path and should be reading limits. If pods are vanishing and reappearing, you are on this one.

On the node: disk first

Disk is the most common cause and the one that best explains a single misbehaving node:

df -h                             # space on / and the container filesystem
df -i                             # inodes - exhausted inodes trigger DiskPressure too
sudo du -sh /var/lib/containerd   # image and layer cache
sudo du -sh /var/log/*            # host logs
sudo du -sh /var/log/pods/*       # container logs, if rotation is broken
sudo du -sh /var/lib/kubelet/pods/* | sort -h | tail   # emptyDir and local volumes
journalctl -u kubelet | grep -i evict     # the kubelet's own account

df -i earns its own line. Inode exhaustion produces DiskPressure with plenty of gigabytes free, and a team staring at df -h can spend an hour concluding the node is fine. Anything that creates millions of tiny files - a cache directory, a build workspace - gets there without ever filling the disk.

The last du is where you find the noisy neighbour: a single pod whose emptyDir or writable layer has swallowed the node. That is your answer to "why this node" - the pod happened to land here.

For memory pressure, be careful with free -h. The kubelet computes memory.available from the working set, which excludes reclaimable page cache; a node showing almost no "free" memory can be entirely healthy. Trust the node condition and kubectl top node over the raw free column.

Stop the bleeding, then fix it

Order matters here, and the first command is the one that helps the customer immediately:

kubectl cordon <node>                 # nothing new lands here
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data   # if pods are still cycling

Cordon alone stops the churn from being fed. Drain moves the survivors off, and note the flags: it will not proceed past DaemonSet pods or pods with local data unless you say so, and --delete-emptydir-data genuinely deletes that data - confirm it is scratch before you pass it.

Then reclaim, on the node:

sudo crictl images                    # what is actually cached
sudo crictl rmi --prune               # drop unreferenced images
sudo journalctl --vacuum-size=500M    # if systemd journal is the hog

And uncordon once the condition clears. Clean up the debris while you are there:

kubectl delete pods -A --field-selector status.phase=Failed

Make it not happen again

The fixes divide neatly by owner, which is the useful part when you are writing the ticket up:

The node / platform side. Tune imageGCHighThresholdPercent and imageGCLowThresholdPercent for how fast this cluster churns images. Confirm container log rotation is actually configured - the kubelet's containerLogMaxSize (10Mi) and containerLogMaxFiles (5) are per container, and an application logging to a file inside the container bypasses them entirely. Give imagefs its own volume so an image cache cannot starve the root filesystem. And check whether this node is the same shape as its peers, because the odd node is usually the odd instance type or the one with the longest uptime.

The application side. Set resource requests on everything. A LimitRange per namespace gives you a floor without editing every manifest. This is the change that makes the evictions stop looking random - a pod inside its requests is at the back of the queue instead of permanently at the front, and the scheduler stops treating the node as empty when it isn't.

The monitoring side. Alert on the node conditions - MemoryPressure, DiskPressure, PIDPressure - and on disk trending past 80%, not on pod restart counts. Restarts are how you find out afterwards; conditions are how you find out during.

The takeaway

Two words in the original report did the work. One node said the node is the variable, not the pod spec, which is what put you in an SSH session instead of a manifest. Randomly said the workload probably has no resource requests, which is what turns a one-off cleanup into a recurring incident.

And the ownership split follows from the same reading: the full disk is the platform's to fix, the missing requests are the application's - and if you only fix the first one, you will be back on this node in a month.