Skip to content
Breaks at CNI plugin · Your containerCan take the service downCommon

NetworkPolicy blocking traffic

A NetworkPolicy selects the pod, and the connection you need is not on its allow-list, so packets are silently dropped.

What you'll seeIllustrative output
$ kubectl exec web-6d5f7c9b8-hx2lm -- wget -qO- -T 5 http://api:8080/healthz
wget: bad address 'api:8080'
$ kubectl get networkpolicy
NAME          POD-SELECTOR   AGE
api-ingress   app=api        3d
web-egress    app=web        14m
$ kubectl describe networkpolicy web-egress
Name:         web-egress
Namespace:    default
Spec:
  PodSelector:     app=web
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: 8080/TCP
    To:
      PodSelector: app=api
  Policy Types: Egress

Watch it happen

How NetworkPolicy blocking traffic unfolds

Step 3 · The failure

The forgotten DNS rule

A second policy selects the web pod for egress and allows only api on port 8080. Nobody listed DNS. The lookup is dropped, the resolver waits 5 seconds and tries again, and that is dropped too. The app reports it cannot resolve the name. The request it was allowed to make is never sent.

  • lookup
  • retry
  • dropped
  • waiting

Where it breaks: The network plugin (CNI) drops the packets, on the instructions of a NetworkPolicy. The policy is what is wrong: it selects the pod and does not list the connection the pod needs.

In short

What NetworkPolicy blocking traffic means

A NetworkPolicy is an allow-list, enforced by the cluster's network plugin. A pod that no policy selects is open in both directions. As soon as one policy selects a pod for ingress or for egress, that direction flips to deny by default, and only connections allowed by some policy get through. Policies add up; there is no rule that removes access, and no order between them. Anything not listed is dropped without a reply, so the caller sees a timeout, never a refusal. The most common way to hurt yourself is an egress policy that names the destination the app needs and leaves out DNS. The pod can then no longer resolve any name, including the one it is allowed to call. The opposite surprise also exists: on a network plugin that does not implement policies, they are accepted and do nothing.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get networkpolicy -n <namespace>
    Look for
    Which policies exist, their pod selectors and their age. Check the destination's namespace as well as the source's.
    What it tells you
    No policies in either namespace rules this page out. A policy created just before the trouble started is the first suspect.
  2. 2
    kubectl get pod <pod> --show-labels
    Look for
    Whether the labels of the source pod and of the destination pod match any policy's pod selector.
    What it tells you
    A pod no policy selects is not restricted. A selected pod is deny-by-default for the directions the policy covers.
  3. 3
    kubectl describe networkpolicy <policy>
    Look for
    Policy Types, and the lists after “Allowing ingress traffic” and “Allowing egress traffic”.
    What it tells you
    The connection you need must appear in a list for that direction: the right pods or namespaces and the right port. If it is in none, it is dropped.
  4. 4
    kubectl exec <source-pod> -- nslookup kubernetes.default
    Look for
    Whether a lookup is answered or times out.
    What it tells you
    A timeout from a pod under an egress policy means port 53 is not allowed. Fix that first: nothing else can be tested by name until it works.
  5. 5
    kubectl exec <source-pod> -- wget -qO- -T 5 http://<destination-pod-ip>:<port>
    Look for
    Timeout, refusal or an answer when calling the destination by IP.
    What it tells you
    A timeout fits a policy drop. “Connection refused” means the packet arrived and nothing was listening, which is not a policy problem.

Most common first

Root causes

  1. 1An egress policy allows the application's destinations and leaves out DNS.

    Confirm
    Lookups time out from the selected pods, while calling the destination by IP address works.
    Fix now
    Add an egress rule for UDP and TCP port 53 to the cluster DNS pods.
    Prevent
    Start every egress policy from a template that already contains the DNS rule.
  2. 2A default-deny policy was applied to a namespace before the allow rules for its real traffic were in place.

    Confirm
    Everything to or from the namespace started timing out at the moment the policy was created.
    Fix now
    Add allow policies for each flow the workloads need, or remove the default-deny until they are ready.
    Prevent
    Write and test the allow rules first, in a staging namespace, then add the default-deny.
  3. 3The selectors in a rule do not mean what was intended: a namespace and a pod selector written as two list entries allow either, while one entry with both requires both.

    Confirm
    kubectl describe shows two separate “From” blocks where one was meant, or the namespace label named in the rule does not exist.
    Fix now
    Restructure the rule, and select namespaces by the built-in kubernetes.io/metadata.name label.
    Prevent
    Review the describe output of every new policy, not only its YAML.
  4. 4Traffic from outside the namespace was not listed: the ingress controller, a monitoring agent or another team's service.

    Confirm
    In-namespace calls work while requests through the ingress, or metric scrapes, time out.
    Fix now
    Add an ingress rule with a namespace selector for the namespace that traffic comes from.
    Prevent
    List every caller of a workload before isolating it.
  5. 5The opposite problem: the cluster's network plugin does not implement NetworkPolicy, so policies are accepted and nothing is blocked.

    Confirm
    A default-deny policy is in place and a test connection that it should block still succeeds.
    Fix now
    Install or enable a network plugin, or a policy engine alongside it, that enforces NetworkPolicy.
    Prevent
    Test enforcement with one deliberate deny in every new cluster.

The tempting wrong fix

Delete the NetworkPolicy

It does bring the traffic back, by removing the isolation for every pod the policy covered, including whatever the policy was written to protect. If it was the only policy selecting those pods, they are now open to the whole cluster. The missing piece is usually one rule; add that rule and keep the rest.

Before / after

The most common fix: add DNS to the egress allow-list

Before
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-egress
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 8080
After
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-egress
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 8080
    - to:                       # DNS
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

The namespace and pod selectors sit in one list entry, so both must match: DNS pods in kube-system only. Replies to allowed connections are let through automatically, so no matching ingress rule is needed on the web pod.

In the real world

Depends on version and provider

Enforcement belongs to the network plugin, not to Kubernetes itself. Some plugins do not implement NetworkPolicy at all, and on several managed services it has to be switched on or needs a particular dataplane; until then policies are stored and ignored. Clusters that run a node-local DNS cache send lookups to an address on the node, not to the kube-dns pods, so the DNS rule may need an IP block instead of a pod selector. Traffic between a pod and its own node is always allowed, and behaviour for hostNetwork pods differs between plugins. Many plugins also offer their own policy types with explicit deny rules, which the standard API does not have.

Your turn

What would you do?

A team adds an egress NetworkPolicy to their pods that allows TCP 5432 to the database pods. Straight away the app logs “could not translate host name db to address”. The database is healthy. What is missing?

NetworkPolicy blocking traffic questions

Why did my pod lose connectivity after I added a NetworkPolicy?
Because a policy turns the pods it selects into an allow-list for that direction. Every connection that is not explicitly allowed by some policy is dropped, including ones you did not think about, such as DNS.
Do I need to allow DNS in a NetworkPolicy?
Yes, if the policy restricts egress. Allow UDP and TCP port 53 to the cluster's DNS pods, otherwise the selected pods cannot resolve any name.
Why does my NetworkPolicy have no effect?
NetworkPolicies are enforced by the cluster's network plugin. If the plugin does not implement them, or enforcement is not enabled on your managed cluster, the policy is stored and nothing is blocked.

Sources: Kubernetes docs: Network Policies ↗ · Kubernetes docs: Declare Network Policy ↗ · Kubernetes docs: Debugging DNS resolution ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play