Skip to content
Breaks at kubelet · kube-apiserverDegrades the serviceCommon

ConfigMap / Secret issues

A container cannot be created because a ConfigMap or Secret it refers to is missing, or it keeps running with old values after an update.

What you'll seeIllustrative output
$ kubectl get pods
NAME                   READY   STATUS                       RESTARTS   AGE
api-84c6f7d5b9-kq2vd   0/1     CreateContainerConfigError   0          2m5s
$ kubectl describe pod api-84c6f7d5b9-kq2vd
    State:          Waiting
      Reason:       CreateContainerConfigError
    Environment:
      DATABASE_URL:  <set to the key 'database-url' in secret 'api-secrets'>  Optional: false
Events:
  Type     Reason  Age                  From     Message
  Normal   Pulled  12s (x10 over 2m4s)  kubelet  Container image "registry.example.com/api:1.4.2" already present on machine
  Warning  Failed  12s (x10 over 2m4s)  kubelet  Error: secret "api-secrets" not found
$ kubectl get secret api-secrets
Error from server (NotFound): secrets "api-secrets" not found

Watch it happen

How ConfigMap / Secret issues unfolds

Step 3 · The failure

The Secret does not exist

The pod refers to a Secret named api-secrets, and there is none in this namespace. The API server answers “not found”, so the kubelet cannot build the container's configuration and never creates it. It retries, and gets the same answer every time.

  • request
  • not found
  • retry

Where it breaks: The kubelet is where it stops: it must fetch every referenced ConfigMap and Secret from the API server before it can create the container. The cause is the reference itself, which names an object or key that does not exist in the pod's namespace.

In short

What ConfigMap / Secret issues means

ConfigMaps and Secrets reach a container in two ways, and each has its own failure. When a pod starts, the kubelet fetches every object the container refers to. If a referenced Secret, ConfigMap or key does not exist, the container is not created at all and the pod shows CreateContainerConfigError. The kubelet keeps retrying, so creating the missing object is enough to unblock it. The second failure is quieter: the object is changed and the pods carry on with the old values. Environment variables are copied once, when the container is created, and never change afterwards. Files mounted from a volume are refreshed by the kubelet, typically within a minute or so. Files mounted with subPath are never refreshed. Even a refreshed file only helps if the application reads it again. In practice, changing configuration means restarting the pods.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get pod <pod>
    Look for
    STATUS CreateContainerConfigError with RESTARTS at 0. Or ContainerCreating that never ends.
    What it tells you
    Rules in: the container was never created, so it is about what the pod refers to, not about your application. ContainerCreating points at a volume instead of an environment variable.
  2. 2
    kubectl describe pod <pod>
    Look for
    The Failed event: it names the object or the key. For volumes, a FailedMount event does the same.
    What it tells you
    “secret … not found” or “configmap … not found”: the object is missing. “couldn't find key … in Secret”: the object exists, the key does not.
  3. 3
    kubectl get secret,configmap -n <namespace>
    Look for
    Whether the named object is in the same namespace as the pod, spelled the same way.
    What it tells you
    Rules in a typo, an object created in another namespace, or one that was never applied. Pods cannot read ConfigMaps or Secrets across namespaces.
  4. 4
    kubectl get secret <name> -o jsonpath='{.data}{"\n"}'
    Look for
    The keys the object really has. Use kubectl describe configmap <name> for a ConfigMap.
    What it tells you
    Rules in a wrong or misspelled key name in the pod's reference.
  5. 5
    kubectl exec <pod> -- printenv <VARIABLE>
    Look for
    For stale values: what the running container holds, compared with the object now. Use cat on the mounted path for files.
    What it tells you
    An old value in an environment variable is expected: it is fixed at start. An old value in a mounted file after a few minutes points at subPath.
  6. 6
    kubectl rollout restart deployment/<name>
    Look for
    New pods starting and reading the current values.
    What it tells you
    Confirms the stale-value case: if the restart fixes it, the config was right and the pods were simply older than the change.

Most common first

Root causes

  1. 1The Secret or ConfigMap does not exist in the pod's namespace: never applied, applied after the Deployment, misspelled, or created in another namespace.

    Confirm
    The event says the object is “not found”, and kubectl get in the pod's namespace does not list it.
    Fix now
    Create the object in that namespace with the name the pod uses. The kubelet picks it up at its next retry.
    Prevent
    Ship ConfigMaps and Secrets in the same release as the workload that needs them, and apply them first.
  2. 2The object exists but the key the pod asks for does not.

    Confirm
    The event says “couldn't find key … in Secret” or “… in ConfigMap”.
    Fix now
    Correct the key in the pod's reference, or add the key to the object.
    Prevent
    Generate both sides from one template so key names cannot drift apart.
  3. 3The ConfigMap or Secret was changed, and the pods still run with what they read at start.

    Confirm
    printenv in the container shows the old value while the object shows the new one.
    Fix now
    Restart the pods: kubectl rollout restart deployment/<name>.
    Prevent
    Roll the pods whenever config changes, for example by putting a checksum of the config in a pod template annotation.
  4. 4A file is mounted with subPath, so the kubelet never refreshes it.

    Confirm
    The volumeMount has a subPath, and the file inside the container keeps its old content long after the change.
    Fix now
    Restart the pods. To get live updates, mount the whole volume as a directory instead.
    Prevent
    Avoid subPath for configuration that is expected to change.
  5. 5A missing ConfigMap or Secret is used as a volume, so the pod hangs in ContainerCreating.

    Confirm
    Events show FailedMount with “MountVolume.SetUp failed … not found”.
    Fix now
    Create the object, or correct its name in the volume definition.
    Prevent
    Mark references optional only when the application can really run without them.

The tempting wrong fix

Edit the ConfigMap and wait for the pods to pick it up

Values used as environment variables are copied into the container once, at creation, and files mounted with subPath are never refreshed. Even a plain mounted file changes only after the kubelet's next sync, and then the application must read it again. Without a restart most pods keep the old configuration indefinitely.

Before / after

The most common fix: refer to a name and key that exist

Before
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: api-secret        # the Secret is called api-secrets
        key: database_url       # its key is database-url
After
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: api-secrets
        key: database-url

The Secret must be in the same namespace as the pod. Add optional: true only if the container can start without the value: the variable is then simply left unset.

In the real world

Depends on version and provider

How fast a mounted file follows a change depends on the kubelet: its sync period, one minute by default, plus the delay of its change-detection strategy (watch, a time-based cache, or direct reads), set by configMapAndSecretChangeDetectionStrategy. Managed providers choose these values, so expect anything from seconds to a couple of minutes. ConfigMaps and Secrets marked immutable are never refreshed at all. Tools outside Kubernetes itself, such as config reloaders or Helm checksum annotations, are what usually trigger restarts on change.

Seen in a real outage

Settings that are applied only at start-up are not put back when something changes underneath. At Heroku in June 2025, the network routes for app containers were configured only at boot, so when the network service restarted nothing restored them.

Break it: Routes only at boot

Your turn

What would you do?

You change a value in a ConfigMap. The app reads it from an environment variable. Ten minutes later the pods still use the old value. Why?

ConfigMap / Secret issues questions

What does CreateContainerConfigError mean?
The kubelet could not put together the container's configuration, almost always because a ConfigMap or Secret the container refers to, or a key inside it, does not exist in the pod's namespace. The container is not created until it does.
Do pods pick up ConfigMap changes automatically?
Only partly. Files mounted from a ConfigMap volume are refreshed after a delay. Environment variables and files mounted with subPath are not updated at all. Restart the pods to be sure they use the new values.
How do I make pods reload a changed Secret or ConfigMap?
Restart them, for example with kubectl rollout restart deployment/<name>. Many teams automate this by adding a checksum of the configuration to the pod template, so that any change starts a normal rollout.

Sources: Kubernetes docs: ConfigMaps ↗ · Kubernetes docs: Secrets ↗ · Kubernetes docs: Configure a pod to use a ConfigMap ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play