Skip to main content

Cluster Networking Model: Every Pod Gets an IP

What This Concept Is​

The Kubernetes networking model is four rules:

  1. Every Pod gets its own IP address.
  2. Every container in a Pod shares that IP (same net namespace).
  3. Pods can reach each other by IP without NAT, regardless of which node they run on.
  4. Agents on a node (kubelet, system daemons) can reach Pods on that node without NAT.

That flat, no-NAT pod network is assumed by every higher layer (Services, DNS, Ingress). Kubernetes itself does not implement it. It is implemented by a CNI (Container Network Interface) plugin installed at cluster creation: Calico, Cilium, Flannel, AWS VPC CNI, Azure CNI, etc.

A CNI plugin:

  • allocates a pod IP from a per-node CIDR (or directly from VPC space)
  • wires up a veth pair: one end in the pod's net namespace, the other on a bridge or routing table on the host
  • programs host routes or encapsulation (VXLAN, IP-in-IP, BGP, eBPF) so that a packet from Pod A on Node 1 reaches Pod B on Node 2

Why It Matters Here​

Everything about Services, DNS, and Ingress is a simplification built on top of "every pod has a routable IP." If that assumption breaks, nothing above it works. The common failure modes are:

  • two nodes with overlapping pod CIDRs -> pod-to-pod traffic routes to the wrong node
  • CNI plugin crash on a node -> new pods get stuck in ContainerCreating because no IP can be allocated
  • NetworkPolicy misconfiguration -> "pods can reach each other by IP without NAT" is denied and you see timeouts with no error logs

Concrete Example​

A single node, two pods, one Service:

Node 10.0.0.5
 ├── cni0 bridge (10.244.1.0/24)
 │    ├── veth-aa ──── Pod A (10.244.1.7) ── eth0 ── container
 │    └── veth-bb ──── Pod B (10.244.1.8) ── eth0 ── container
 └── iptables rules (kube-proxy) -> Service 10.96.0.12 -> [10.244.1.7, 10.244.1.8]

Cross-node traffic, with an overlay:

Pod A (10.244.1.7) on Node 1
        │
   veth -> cni0 -> host route (10.244.2.0/24 via VXLAN tunnel)
        │
   ─── encapsulated over 10.0.0.6:4789 ───
        │
   host route -> cni0 -> veth -> Pod C (10.244.2.4) on Node 2

Common Confusion / Misconception​

"Pods have a DNS name."

Pods do not. Services do. By default a Pod's only stable identifier is its name within a StatefulSet (covered in Cluster 4). A Pod's IP changes on every restart. This is why client code must never hard-code a Pod IP; it should resolve a Service name and let DNS and kube-proxy do the rest.

A second confusion: "NetworkPolicy is enforced by Kubernetes itself." It is not. Policy is enforced by the CNI plugin. On a cluster with a CNI that does not implement NetworkPolicy (Flannel alone, for example), kind: NetworkPolicy resources exist but are ignored.

How To Use It​

Draw the path from source to destination before reading any logs:

If a request fails, you can work this graph left-to-right and identify the first hop that does not respond. That is almost always faster than reading logs.

CNI Plugins: What They Actually Do​

The Container Network Interface (CNI) is a small specification and an executable contract. When a Pod is created, the kubelet invokes the CNI binary with:

  • the network namespace handle for the new Pod
  • a JSON config object describing the network
  • the desired command (ADD, DEL, CHECK)

The CNI binary is responsible for:

  1. Allocating an IP (from IPAM -- either per-node CIDR or the cloud's VPC allocator).
  2. Creating the veth pair and moving one end into the pod's net namespace.
  3. Programming routes so packets from that IP go out the correct host interface or tunnel.
  4. Installing any firewall rules implied by NetworkPolicy (if the plugin supports it).
  5. Returning the result to the kubelet, which records the Pod IP via the api-server.

Common CNI plugins differ mostly in step 3:

PluginCross-node path
Flannel (VXLAN)UDP encapsulation between nodes
Calico (BGP)Real L3 routes between nodes on the same L2, plus BGP with ToR
Cilium (eBPF)eBPF programs on each node; overlay or native routing
AWS VPC CNIPod IPs are real VPC IPs on the node's ENIs
Azure / GCP nativeSimilar to AWS; Pod IPs come from the cloud VPC

When performance surprises you (tail latency, packet loss), knowing which plugin you run determines which knobs exist.

Check Yourself​

  1. What are the four network guarantees of the Kubernetes model?
  2. Where does a Pod IP come from, and who assigns it?
  3. Why is NetworkPolicy not a self-enforcing feature?

NetworkPolicy: a Default-Deny Pattern​

A common production pattern is to apply a namespace-wide default-deny and then explicitly allow specific flows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: prod
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-api
  namespace: prod
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: web } }
      ports:
        - { protocol: TCP, port: 8080 }

Remember: NetworkPolicy is additive. Two policies that each allow one flow allow both flows; a policy never "overrides" another to deny. For denial you rely on the default-deny.

Mini Drill or Application​

On a running cluster, list pod IPs:

kubectl get pods -A -o wide

Pick two pods on the same node and two on different nodes. From a debug pod (kubectl run -it --rm debug --image=nicolaka/netshoot -- bash), ping or curl each target pod's IP. Then list the routes and interfaces on the debug node with ip route and ip link. Write down which packets crossed only the bridge and which traversed the overlay or cloud routing.

Which CNI Should I Use?​

There is no single right answer, but three questions collapse the decision quickly:

  1. Am I on a managed cloud platform? Use the cloud's native CNI (AWS VPC CNI, Azure CNI, GKE dataplane) unless you have a specific reason not to. Pod IPs become real VPC IPs -- no overlay, no double-encapsulation cost, and cloud security groups apply.
  2. Do I need NetworkPolicy enforcement? Pick a plugin that implements it (Calico, Cilium). Flannel does not enforce policy on its own.
  3. Do I need L7-aware / eBPF-based features (identity-aware policy, transparent encryption, observability, service-mesh bypass)? Cilium is the answer most teams land on.

"We should migrate CNI plugins" is a one-way-door decision in production (all pods need recreation, policy is not portable across plugins). Write an ADR before switching.

Read This Only If Stuck​