~/ryan
← Blog/kubernetes

Ready, Endpoints Populated, Nothing Gets Through

The Service is right, the selector is right, the endpoint is populated, the pod is Ready - and every request fails. Debugging this well is less about knowing the causes than about reading the symptom, distrusting the tests that pass, and bisecting in the right order.

Aug 25, 2026·10 min readkubernetesnetworking

An application inside a pod listens on 127.0.0.1:8080. Can anything outside the pod reach it? No - and the interesting part is not the fix, which is one line of application config. It is that nothing in the cluster will tell you, and that the obvious test you would run to rule it out passes.

Loopback never leaves the namespace

127.0.0.1 is a loopback address. The kernel never puts a packet with a loopback destination onto a real interface, and it drops arriving packets that claim one. This is routing and kernel behaviour, not policy: there is no iptables chain to fix, no NetworkPolicy to loosen, no capability to add.

The part worth internalising is which loopback. Every network namespace has its own lo device. "Localhost" does not mean "this machine" and it does not mean "this container" - it means this network namespace, and nothing else.

The boundary is the pod, not the container

This is where most mental models are off by one. All containers in a pod share a single network namespace, so the boundary localhost draws is the pod.

  • Same container - works.
  • A sidecar in the same pod - works. This is exactly how mesh proxies and log shippers reach the app, and why they are packaged as sidecars rather than as separate pods.
  • Another pod - fails, even on the same node.
  • Through the Service or the Ingress - fails.
  • From the node itself - fails. The node is a different namespace.
  • kubectl port-forward - works, which is the trap. More on that below.

The exception is hostNetwork: true. That pod has no namespace of its own; it uses the node's, so 127.0.0.1 is the node's loopback - reachable by anything else on that node, still unreachable from anywhere off it.

Why everything looks healthy

The pod is Running. The kubelet reports on the process, not on whether anyone can reach it. A process listening on the wrong address is still a live process.

The endpoint is populated. A Ready pod goes into the EndpointSlice, so kubectl get endpoints shows the pod IP and kube-proxy happily DNATs Service traffic to <pod-ip>:8080.

The readiness probe passes - sometimes. An httpGet probe with no host field makes the kubelet dial the pod IP, so a loopback bind fails the probe, the pod never becomes Ready, and the bug catches itself. The shapes that let it through:

  • no readiness probe at all - the pod is Ready the moment the container starts,
  • an exec probe that runs curl http://localhost:8080/healthz inside the container, which of course succeeds,
  • an httpGet probe with host: 127.0.0.1 set explicitly.

And port-forward works. kubectl port-forward is proxied by the kubelet from inside the pod's network namespace, so it reaches a loopback listener like any in-pod client would. A developer testing through port-forward gets a 200 and concludes the app is fine. It is fine - to itself.

Read the failure mode before you read anything else

Before running a single command, the error string the client returned has already narrowed the field. Three outcomes, three different worlds:

  • connection refused - the packet arrived somewhere and something actively rejected it. A TCP RST came back. Wrong port, or nothing bound to the address you dialled.
  • timeout - nothing came back at all. The packet went out and vanished. Something dropped it: a NetworkPolicy, a firewall or security group, a routing problem.
  • no such host - the name never resolved, so no packet was ever sent. This is DNS, and nothing else.

Timeout is the strongest signal you get for NetworkPolicy, and the reason is mechanical: policies drop traffic, they do not reject it. A default-deny ingress policy with no rule allowing the source produces exactly a hang, every time. Listing the causes is knowledge; connecting the symptom to the cause is diagnosis, and only the second one is fast.

Which also tells you where a loopback bind sits. Traffic reaches the pod's namespace, arrives at <pod-ip>:8080, and finds no listener on that address - so the kernel sends a RST and the caller gets connection refused, quickly. If your symptom is a hang, look at policy and routing first; if it is refused, look at what the process bound to.

Distrust the test that passes

Someone will report that curl localhost:8080 works inside the pod, and offer it as proof the application is fine. It isn't proof of anything - because a process bound to 127.0.0.1:8080 answers that call and is unreachable from everywhere else.

That successful curl is consistent with the bug. So verify it rather than build on it:

kubectl exec -it <pod> -- ss -tlnp
  • 0.0.0.0:8080 or *:8080 - listening on every interface in the namespace. Healthy; look elsewhere.
  • 127.0.0.1:8080 - found it.

Being suspicious of evidence that appears to clear a component is most of what separates reading a system from guessing at one.

The order that halves the problem

# 1. is the app listening on an interface that can receive external traffic?
kubectl exec -it <pod> -- ss -tlnp

# 2. does the Service have backends at all?
kubectl get endpoints <svc> -n <ns>

# 3. bypass the Service entirely - talk to the pod IP
kubectl exec -it <other-pod> -- curl -sS -m 5 <pod-ip>:8080

# 4. name resolution, separately from connectivity
kubectl exec -it <other-pod> -- nslookup <svc>.<ns>.svc.cluster.local

# 5. what policy applies in this namespace?
kubectl get networkpolicy -n <ns>

Step 2 is where an empty result means the selector does not match the pod labels, or the pods are Running but not Ready.

Step 3 is the one that earns its place. It cuts the problem in half in a single command:

  • pod IP works, Service name does not → the Service, its ports, or DNS.
  • pod IP also fails → the network layer, below the Service entirely. And the way it fails re-applies the rule above: refused points back at the bind address, timeout points at policy.

Run it from another pod, not from the node - the node has different routing and, on some CNIs, different policy treatment.

When the image has no shell

Distroless and scratch images ship no ss, no curl, no shell. Attach an ephemeral container instead; it joins the pod's network namespace and sees the same listeners:

kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container> -- ss -tlnp

Or read the kernel's table directly, which needs nothing but cat:

kubectl exec <pod> -- cat /proc/net/tcp

The local_address column is little-endian hex. 0100007F:1F90 is 127.0.0.1:8080; 00000000:1F90 is 0.0.0.0:8080. For IPv6, check /proc/net/tcp6, where ::1 appears as 00000000000000000000000001000000 and :: as all zeros - a service bound to ::1 fails in exactly the same way, and on a dual-stack cluster it is the same bug wearing a different address.

The fix is one line, and it is in the application

Kubernetes has no setting for this. The bind address is chosen by the process:

uvicorn.run(app, host="0.0.0.0", port=8080)  # not the 127.0.0.1 default

The equivalents, since it is never the framework you expect:

  • Go - http.ListenAndServe(":8080", h) binds every interface; "localhost:8080" does not.
  • Node / Express - app.listen(8080) is fine; app.listen(8080, "127.0.0.1") is not.
  • nginx - listen 8080; rather than listen 127.0.0.1:8080;.
  • Spring Boot - server.address=0.0.0.0.
  • ASP.NET - ASPNETCORE_URLS=http://0.0.0.0:8080.
  • Rails / Puma - -b tcp://0.0.0.0:8080.

The common thread: frameworks default to loopback because that is the safe default on a laptop, where binding every interface exposes your dev server to the coffee shop. Containerising an app carries that default somewhere it is simply wrong.

"But isn't 0.0.0.0 insecure?"

Not here. Inside a pod, 0.0.0.0 means every interface in that network namespace: lo and one veth holding the pod IP. It does not expose the process to the node, the cluster or the internet - who can reach that pod IP is decided by NetworkPolicy, the Service and the Ingress. Binding loopback in Kubernetes is not a security control, it is an outage. If you want the control, write a default-deny NetworkPolicy.

The one place the objection is real is hostNetwork: true, where 0.0.0.0 genuinely means every interface on the node, public ones included. There, bind deliberately.

Make the cluster catch the next one

Give every workload a readiness probe that dials the pod IP - which is what you get by leaving host unset:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080 # no `host:` - the kubelet dials the pod IP 
  periodSeconds: 5

Now a loopback bind fails readiness, the pod never enters the EndpointSlice, and a rollout stalls loudly instead of quietly serving nothing. If you enforce policy with Kyverno or Gatekeeper, the rule worth writing is: no host: 127.0.0.1 in probes, and no exec probe that curls localhost. Those two patterns are what turn a loud failure into a silent one.

The takeaway

Nothing in the cluster is misconfigured. The Service, the selector, the EndpointSlice, kube-proxy and the CNI all did their jobs and delivered a packet to an address the application declined to listen on. That is also the ownership line: this is not a platform bug, so it does not escalate to your infrastructure team or your provider - it goes back to whoever ships the image, as a one-line change.

Drawing that line in five minutes, with ss -tlnp output attached, is most of the value in triaging it. And the habit that gets you there is not memorising causes. It is reading the failure mode, refusing to trust the test that passed, and bisecting where the split is cheapest.

← All posts