~/ryan
← Blog/kubernetes

Ready, Endpoints Populated, Nothing Gets Through

The Service is right, the selector is right, the endpoint is populated, the pod is Ready - and every request fails. Debugging this well is less about knowing the causes than about reading the symptom, distrusting the tests that pass, and bisecting in the right order.

Aug 25, 2026·11 min readkubernetesnetworking

An application inside a pod listens on 127.0.0.1:8080. Can anything outside the pod reach it? No - and the interesting part is not the fix, which is one line of application config. It is that nothing in the cluster will tell you, and that the obvious test you would run to rule it out passes.

Loopback never leaves the namespace

127.0.0.1 is a loopback address. By default the kernel never puts a packet with a loopback destination onto a real interface, and it drops arriving packets that claim one (the route_localnet sysctl can lift the latter, but nothing in a normal pod does). This is routing and kernel behaviour, not policy: there is no iptables chain to fix, no NetworkPolicy to loosen, no capability to add.

The part worth internalising is which loopback. Every network namespace has its own lo device. "Localhost" does not mean "this machine" and it does not mean "this container" - it means this network namespace, and nothing else.

The boundary is the pod, not the container

This is where most mental models are off by one. All containers in a pod share a single network namespace, so the boundary localhost draws is the pod.

  • Same container - works.
  • A sidecar in the same pod - works. This is how a log shipper or a metrics scraper in the same pod reaches a loopback-bound app. (Service meshes are not a good example here: since Istio 1.10 and Linkerd 2.11 the mesh proxy forwards to the pod IP, so an app bound only to lo is unreachable through the mesh too - the same bug, one hop later.)
  • Another pod - fails, even on the same node.
  • Through the Service or the Ingress - fails.
  • From the node itself - fails. The node is a different namespace.
  • kubectl port-forward - works, which is the trap. More on that below.

The exception is hostNetwork: true. That pod has no namespace of its own; it uses the node's, so 127.0.0.1 is the node's loopback - reachable by node-level processes and other hostNetwork pods on that node (ordinary pods still have their own lo), and unreachable from anywhere off it.

Why everything looks healthy

The pod is Running. The kubelet reports on the process, not on whether anyone can reach it. A process listening on the wrong address is still a live process.

The endpoint is populated. A Ready pod goes into the EndpointSlice, so kubectl get endpoints shows the pod IP and kube-proxy happily DNATs Service traffic to <pod-ip>:8080.

The readiness probe passes - sometimes. An httpGet probe with no host field makes the kubelet dial the pod IP, so a loopback bind fails the probe, the pod never becomes Ready, and the bug catches itself. The shapes that let it through:

  • no readiness probe at all - the pod is Ready the moment the container starts,
  • an exec probe that runs curl http://localhost:8080/healthz inside the container, which of course succeeds,
  • an httpGet probe pointed at a different port - a health endpoint the app happens to bind on 0.0.0.0 while the main listener sits on loopback.

(Setting host: 127.0.0.1 on the probe does not do this - the kubelet probes from the node's own network namespace, so host makes it dial the node's loopback, which fails just as an unset host would. It is a way to break a working probe, not a way to hide the bug.)

And port-forward works. kubectl port-forward is relayed by the kubelet to the CRI runtime, which dials the target from inside the pod's network namespace, so it reaches a loopback listener like any in-pod client would. A developer testing through port-forward gets a 200 and concludes the app is fine. It is fine - to itself.

Read the failure mode before you read anything else

Before running a single command, the error string the client returned has already narrowed the field. Three outcomes, three different worlds:

  • connection refused - the packet arrived somewhere and something actively rejected it. A TCP RST came back. Wrong port, nothing bound to the address you dialled - or a Service whose endpoints are all not-ready, which kube-proxy rejects rather than black-holes.
  • timeout - nothing came back at all. The packet went out and vanished. Something dropped it: a NetworkPolicy, a firewall or security group, a routing problem.
  • no such host - the name never resolved, so no packet was ever sent. This is DNS, and nothing else.

Timeout is the strongest signal you get for NetworkPolicy, and the reason is mechanical: most CNIs drop denied traffic rather than reject it. A default-deny ingress policy with no rule allowing the source produces exactly a hang on those plugins. (It is a default, not a law - Calico and Cilium can both be configured to send a reject instead - so treat a hang as pointing at policy, then confirm.) Listing the causes is knowledge; connecting the symptom to the cause is diagnosis, and only the second one is fast.

Which also tells you where a loopback bind sits. Traffic reaches the pod's namespace, arrives at <pod-ip>:8080, and finds no listener on that address - so the kernel sends a RST and the caller gets connection refused, quickly. If your symptom is a hang, look at policy and routing first; if it is refused, look at what the process bound to.

Distrust the test that passes

Someone will report that curl localhost:8080 works inside the pod, and offer it as proof the application is fine. It isn't proof of anything - because a process bound to 127.0.0.1:8080 answers that call and is unreachable from everywhere else.

That successful curl is consistent with the bug. So verify it rather than build on it:

kubectl exec -it <pod> -- ss -tlnp
  • 0.0.0.0:8080 or *:8080 - listening on every interface in the namespace. Healthy; look elsewhere.
  • 127.0.0.1:8080 - found it.

Being suspicious of evidence that appears to clear a component is most of what separates reading a system from guessing at one.

The order that halves the problem

# 1. is the app listening on an interface that can receive external traffic?
kubectl exec -it <pod> -- ss -tlnp

# 2. does the Service have backends at all?
kubectl get endpoints <svc> -n <ns>

# 3. bypass the Service entirely - talk to the pod IP
kubectl exec -it <other-pod> -- curl -sS -m 5 <pod-ip>:8080

# 4. name resolution, separately from connectivity
kubectl exec -it <other-pod> -- nslookup <svc>.<ns>.svc.cluster.local

# 5. what policy applies in this namespace?
kubectl get networkpolicy -n <ns>

Step 2 is where an empty result means the selector does not match the pod labels, or the pods are Running but not Ready.

Step 3 is the one that earns its place. It cuts the problem in half in a single command:

  • pod IP works, Service name does not → the Service, its ports, or DNS.
  • pod IP also fails → the network layer, below the Service entirely. And the way it fails re-applies the rule above: refused points back at the bind address, timeout points at policy.

Run it from another pod, not from the node - the node has different routing and, on some CNIs, different policy treatment.

When the image has no shell

Distroless and scratch images ship no ss, no curl, no shell. Attach an ephemeral container instead; it joins the pod's network namespace and sees the same listeners:

kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container> -- ss -tlnp

Or read the kernel's table directly. A distroless image has no cat either, so read it from the ephemeral container (which shares the pod's network namespace) rather than exec-ing into the app container:

kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container> -- cat /proc/net/tcp

In the local_address column the address is printed in host byte order - so on a little-endian CPU the bytes read reversed - while the port after the colon is plain hex. 0100007F:1F90 is 127.0.0.1:8080; 00000000:1F90 is 0.0.0.0:8080. For IPv6, check /proc/net/tcp6, where ::1 appears as 00000000000000000000000001000000 and :: as all zeros - a service bound to ::1 fails in exactly the same way, and on a dual-stack cluster it is the same bug wearing a different address.

The fix is one line, and it is in the application

Kubernetes has no setting for this. The bind address is chosen by the process:

uvicorn.run(app, host="0.0.0.0", port=8080)  # not the 127.0.0.1 default

The equivalents, since it is never the framework you expect:

  • Go - http.ListenAndServe(":8080", h) binds every interface; "localhost:8080" does not.
  • Node / Express - app.listen(8080) is fine; app.listen(8080, "127.0.0.1") is not.
  • nginx - listen 8080; rather than listen 127.0.0.1:8080;.
  • Spring Boot - server.address=0.0.0.0.
  • ASP.NET - ASPNETCORE_URLS=http://0.0.0.0:8080.
  • Rails / Puma - -b tcp://0.0.0.0:8080.

The common thread: frameworks default to loopback because that is the safe default on a laptop, where binding every interface exposes your dev server to the coffee shop. Containerising an app carries that default somewhere it is simply wrong.

"But isn't 0.0.0.0 insecure?"

Not here. Inside a pod, 0.0.0.0 means every interface in that network namespace: lo and one veth holding the pod IP. It does not expose the process to the node, the cluster or the internet - who can reach that pod IP is decided by NetworkPolicy, the Service and the Ingress. Binding loopback in Kubernetes is not a security control, it is an outage. If you want the control, write a default-deny NetworkPolicy.

The one place the objection is real is hostNetwork: true, where 0.0.0.0 genuinely means every interface on the node, public ones included. There, bind deliberately.

Make the cluster catch the next one

Give every workload a readiness probe that dials the pod IP - which is what you get by leaving host unset:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080 # no `host:` - the kubelet dials the pod IP 
  periodSeconds: 5

Now a loopback bind fails readiness, the pod never becomes a ready endpoint (it may appear in the EndpointSlice, but flagged not-ready, so proxies skip it), and a rollout stalls loudly instead of quietly serving nothing. If you enforce policy with Kyverno or Gatekeeper, the rule worth writing is: no host: 127.0.0.1 in probes, and no exec probe that curls localhost. Those two patterns are what turn a loud failure into a silent one.

The takeaway

Nothing in the cluster is misconfigured. The Service, the selector, the EndpointSlice, kube-proxy and the CNI all did their jobs and delivered a packet to an address the application declined to listen on. That is also the ownership line: this is not a platform bug, so it does not escalate to your infrastructure team or your provider - it goes back to whoever ships the image, as a one-line change.

Drawing that line in five minutes, with ss -tlnp output attached, is most of the value in triaging it. And the habit that gets you there is not memorising causes. It is reading the failure mode, refusing to trust the test that passed, and bisecting where the split is cheapest.

← All posts