NetworkPolicy blocking traffic
A NetworkPolicy selects the pod, and the connection you need is not on its allow-list, so packets are silently dropped.
$ kubectl exec web-6d5f7c9b8-hx2lm -- wget -qO- -T 5 http://api:8080/healthzwget: bad address 'api:8080'
$ kubectl get networkpolicyNAME POD-SELECTOR AGE api-ingress app=api 3d web-egress app=web 14m
$ kubectl describe networkpolicy web-egressName: web-egress
Namespace: default
Spec:
PodSelector: app=web
Not affecting ingress traffic
Allowing egress traffic:
To Port: 8080/TCP
To:
PodSelector: app=api
Policy Types: EgressWatch it happen
How NetworkPolicy blocking traffic unfolds
Step 3 · The failure
The forgotten DNS rule
A second policy selects the web pod for egress and allows only api on port 8080. Nobody listed DNS. The lookup is dropped, the resolver waits 5 seconds and tries again, and that is dropped too. The app reports it cannot resolve the name. The request it was allowed to make is never sent.
- lookup
- retry
- dropped
- waiting
Step 3 · The failure
The forgotten DNS rule
A second policy selects the web pod for egress and allows only api on port 8080. Nobody listed DNS. The lookup is dropped, the resolver waits 5 seconds and tries again, and that is dropped too. The app reports it cannot resolve the name. The request it was allowed to make is never sent.
- lookup
- retry
- dropped
- waiting
Where it breaks: The network plugin (CNI) drops the packets, on the instructions of a NetworkPolicy. The policy is what is wrong: it selects the pod and does not list the connection the pod needs.
In short
What NetworkPolicy blocking traffic means
A NetworkPolicy is an allow-list, enforced by the cluster's network plugin. A pod that no policy selects is open in both directions. As soon as one policy selects a pod for ingress or for egress, that direction flips to deny by default, and only connections allowed by some policy get through. Policies add up; there is no rule that removes access, and no order between them. Anything not listed is dropped without a reply, so the caller sees a timeout, never a refusal. The most common way to hurt yourself is an egress policy that names the destination the app needs and leaves out DNS. The pod can then no longer resolve any name, including the one it is allowed to call. The opposite surprise also exists: on a network plugin that does not implement policies, they are accepted and do nothing.
In order
Diagnose in 60 seconds
- 1
kubectl get networkpolicy -n <namespace>- Look for
- Which policies exist, their pod selectors and their age. Check the destination's namespace as well as the source's.
- What it tells you
- No policies in either namespace rules this page out. A policy created just before the trouble started is the first suspect.
- 2
kubectl get pod <pod> --show-labels- Look for
- Whether the labels of the source pod and of the destination pod match any policy's pod selector.
- What it tells you
- A pod no policy selects is not restricted. A selected pod is deny-by-default for the directions the policy covers.
- 3
kubectl describe networkpolicy <policy>- Look for
- Policy Types, and the lists after “Allowing ingress traffic” and “Allowing egress traffic”.
- What it tells you
- The connection you need must appear in a list for that direction: the right pods or namespaces and the right port. If it is in none, it is dropped.
- 4
kubectl exec <source-pod> -- nslookup kubernetes.default- Look for
- Whether a lookup is answered or times out.
- What it tells you
- A timeout from a pod under an egress policy means port 53 is not allowed. Fix that first: nothing else can be tested by name until it works.
- 5
kubectl exec <source-pod> -- wget -qO- -T 5 http://<destination-pod-ip>:<port>- Look for
- Timeout, refusal or an answer when calling the destination by IP.
- What it tells you
- A timeout fits a policy drop. “Connection refused” means the packet arrived and nothing was listening, which is not a policy problem.
Most common first
Root causes
1An egress policy allows the application's destinations and leaves out DNS.
- Confirm
- Lookups time out from the selected pods, while calling the destination by IP address works.
- Fix now
- Add an egress rule for UDP and TCP port 53 to the cluster DNS pods.
- Prevent
- Start every egress policy from a template that already contains the DNS rule.
2A default-deny policy was applied to a namespace before the allow rules for its real traffic were in place.
- Confirm
- Everything to or from the namespace started timing out at the moment the policy was created.
- Fix now
- Add allow policies for each flow the workloads need, or remove the default-deny until they are ready.
- Prevent
- Write and test the allow rules first, in a staging namespace, then add the default-deny.
3The selectors in a rule do not mean what was intended: a namespace and a pod selector written as two list entries allow either, while one entry with both requires both.
- Confirm
- kubectl describe shows two separate “From” blocks where one was meant, or the namespace label named in the rule does not exist.
- Fix now
- Restructure the rule, and select namespaces by the built-in kubernetes.io/metadata.name label.
- Prevent
- Review the describe output of every new policy, not only its YAML.
4Traffic from outside the namespace was not listed: the ingress controller, a monitoring agent or another team's service.
- Confirm
- In-namespace calls work while requests through the ingress, or metric scrapes, time out.
- Fix now
- Add an ingress rule with a namespace selector for the namespace that traffic comes from.
- Prevent
- List every caller of a workload before isolating it.
5The opposite problem: the cluster's network plugin does not implement NetworkPolicy, so policies are accepted and nothing is blocked.
- Confirm
- A default-deny policy is in place and a test connection that it should block still succeeds.
- Fix now
- Install or enable a network plugin, or a policy engine alongside it, that enforces NetworkPolicy.
- Prevent
- Test enforcement with one deliberate deny in every new cluster.
The tempting wrong fix
Delete the NetworkPolicy
Before / after
The most common fix: add DNS to the egress allow-list
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-egress
spec:
podSelector:
matchLabels:
app: web
policyTypes: [Egress]
egress:
- to:
- podSelector:
matchLabels:
app: api
ports:
- protocol: TCP
port: 8080apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-egress
spec:
podSelector:
matchLabels:
app: web
policyTypes: [Egress]
egress:
- to:
- podSelector:
matchLabels:
app: api
ports:
- protocol: TCP
port: 8080
- to: # DNS
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53The namespace and pod selectors sit in one list entry, so both must match: DNS pods in kube-system only. Replies to allowed connections are let through automatically, so no matching ingress rule is needed on the web pod.
In the real world
Depends on version and provider
Your turn
What would you do?
A team adds an egress NetworkPolicy to their pods that allows TCP 5432 to the database pods. Straight away the app logs “could not translate host name db to address”. The database is healthy. What is missing?
Related errors
NetworkPolicy blocking traffic questions
- Why did my pod lose connectivity after I added a NetworkPolicy?
- Because a policy turns the pods it selects into an allow-list for that direction. Every connection that is not explicitly allowed by some policy is dropped, including ones you did not think about, such as DNS.
- Do I need to allow DNS in a NetworkPolicy?
- Yes, if the policy restricts egress. Allow UDP and TCP port 53 to the cluster's DNS pods, otherwise the selected pods cannot resolve any name.
- Why does my NetworkPolicy have no effect?
- NetworkPolicies are enforced by the cluster's network plugin. If the plugin does not implement them, or enforcement is not enabled on your managed cluster, the policy is stored and nothing is blocked.
Sources: Kubernetes docs: Network Policies ↗ · Kubernetes docs: Declare Network Policy ↗ · Kubernetes docs: Debugging DNS resolution ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.