ConfigMap / Secret issues
A container cannot be created because a ConfigMap or Secret it refers to is missing, or it keeps running with old values after an update.
$ kubectl get podsNAME READY STATUS RESTARTS AGE api-84c6f7d5b9-kq2vd 0/1 CreateContainerConfigError 0 2m5s
$ kubectl describe pod api-84c6f7d5b9-kq2vd State: Waiting
Reason: CreateContainerConfigError
Environment:
DATABASE_URL: <set to the key 'database-url' in secret 'api-secrets'> Optional: false
Events:
Type Reason Age From Message
Normal Pulled 12s (x10 over 2m4s) kubelet Container image "registry.example.com/api:1.4.2" already present on machine
Warning Failed 12s (x10 over 2m4s) kubelet Error: secret "api-secrets" not found$ kubectl get secret api-secretsError from server (NotFound): secrets "api-secrets" not found
Watch it happen
How ConfigMap / Secret issues unfolds
Step 3 · The failure
The Secret does not exist
The pod refers to a Secret named api-secrets, and there is none in this namespace. The API server answers “not found”, so the kubelet cannot build the container's configuration and never creates it. It retries, and gets the same answer every time.
- request
- not found
- retry
Step 3 · The failure
The Secret does not exist
The pod refers to a Secret named api-secrets, and there is none in this namespace. The API server answers “not found”, so the kubelet cannot build the container's configuration and never creates it. It retries, and gets the same answer every time.
- request
- not found
- retry
Where it breaks: The kubelet is where it stops: it must fetch every referenced ConfigMap and Secret from the API server before it can create the container. The cause is the reference itself, which names an object or key that does not exist in the pod's namespace.
In short
What ConfigMap / Secret issues means
ConfigMaps and Secrets reach a container in two ways, and each has its own failure. When a pod starts, the kubelet fetches every object the container refers to. If a referenced Secret, ConfigMap or key does not exist, the container is not created at all and the pod shows CreateContainerConfigError. The kubelet keeps retrying, so creating the missing object is enough to unblock it. The second failure is quieter: the object is changed and the pods carry on with the old values. Environment variables are copied once, when the container is created, and never change afterwards. Files mounted from a volume are refreshed by the kubelet, typically within a minute or so. Files mounted with subPath are never refreshed. Even a refreshed file only helps if the application reads it again. In practice, changing configuration means restarting the pods.
In order
Diagnose in 60 seconds
- 1
kubectl get pod <pod>- Look for
- STATUS CreateContainerConfigError with RESTARTS at 0. Or ContainerCreating that never ends.
- What it tells you
- Rules in: the container was never created, so it is about what the pod refers to, not about your application. ContainerCreating points at a volume instead of an environment variable.
- 2
kubectl describe pod <pod>- Look for
- The Failed event: it names the object or the key. For volumes, a FailedMount event does the same.
- What it tells you
- “secret … not found” or “configmap … not found”: the object is missing. “couldn't find key … in Secret”: the object exists, the key does not.
- 3
kubectl get secret,configmap -n <namespace>- Look for
- Whether the named object is in the same namespace as the pod, spelled the same way.
- What it tells you
- Rules in a typo, an object created in another namespace, or one that was never applied. Pods cannot read ConfigMaps or Secrets across namespaces.
- 4
kubectl get secret <name> -o jsonpath='{.data}{"\n"}'- Look for
- The keys the object really has. Use kubectl describe configmap <name> for a ConfigMap.
- What it tells you
- Rules in a wrong or misspelled key name in the pod's reference.
- 5
kubectl exec <pod> -- printenv <VARIABLE>- Look for
- For stale values: what the running container holds, compared with the object now. Use cat on the mounted path for files.
- What it tells you
- An old value in an environment variable is expected: it is fixed at start. An old value in a mounted file after a few minutes points at subPath.
- 6
kubectl rollout restart deployment/<name>- Look for
- New pods starting and reading the current values.
- What it tells you
- Confirms the stale-value case: if the restart fixes it, the config was right and the pods were simply older than the change.
Most common first
Root causes
1The Secret or ConfigMap does not exist in the pod's namespace: never applied, applied after the Deployment, misspelled, or created in another namespace.
- Confirm
- The event says the object is “not found”, and kubectl get in the pod's namespace does not list it.
- Fix now
- Create the object in that namespace with the name the pod uses. The kubelet picks it up at its next retry.
- Prevent
- Ship ConfigMaps and Secrets in the same release as the workload that needs them, and apply them first.
2The object exists but the key the pod asks for does not.
- Confirm
- The event says “couldn't find key … in Secret” or “… in ConfigMap”.
- Fix now
- Correct the key in the pod's reference, or add the key to the object.
- Prevent
- Generate both sides from one template so key names cannot drift apart.
3The ConfigMap or Secret was changed, and the pods still run with what they read at start.
- Confirm
- printenv in the container shows the old value while the object shows the new one.
- Fix now
- Restart the pods: kubectl rollout restart deployment/<name>.
- Prevent
- Roll the pods whenever config changes, for example by putting a checksum of the config in a pod template annotation.
4A file is mounted with subPath, so the kubelet never refreshes it.
- Confirm
- The volumeMount has a subPath, and the file inside the container keeps its old content long after the change.
- Fix now
- Restart the pods. To get live updates, mount the whole volume as a directory instead.
- Prevent
- Avoid subPath for configuration that is expected to change.
5A missing ConfigMap or Secret is used as a volume, so the pod hangs in ContainerCreating.
- Confirm
- Events show FailedMount with “MountVolume.SetUp failed … not found”.
- Fix now
- Create the object, or correct its name in the volume definition.
- Prevent
- Mark references optional only when the application can really run without them.
The tempting wrong fix
Edit the ConfigMap and wait for the pods to pick it up
Before / after
The most common fix: refer to a name and key that exist
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-secret # the Secret is called api-secrets
key: database_url # its key is database-urlenv:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-secrets
key: database-urlThe Secret must be in the same namespace as the pod. Add optional: true only if the container can start without the value: the variable is then simply left unset.
In the real world
Depends on version and provider
Seen in a real outage
Settings that are applied only at start-up are not put back when something changes underneath. At Heroku in June 2025, the network routes for app containers were configured only at boot, so when the network service restarted nothing restored them.
Break it: Routes only at bootYour turn
What would you do?
You change a value in a ConfigMap. The app reads it from an environment variable. Ten minutes later the pods still use the old value. Why?
Related errors
ConfigMap / Secret issues questions
- What does CreateContainerConfigError mean?
- The kubelet could not put together the container's configuration, almost always because a ConfigMap or Secret the container refers to, or a key inside it, does not exist in the pod's namespace. The container is not created until it does.
- Do pods pick up ConfigMap changes automatically?
- Only partly. Files mounted from a ConfigMap volume are refreshed after a delay. Environment variables and files mounted with subPath are not updated at all. Restart the pods to be sure they use the new values.
- How do I make pods reload a changed Secret or ConfigMap?
- Restart them, for example with kubectl rollout restart deployment/<name>. Many teams automate this by adding a checksum of the configuration to the pod template, so that any change starts a normal rollout.
Sources: Kubernetes docs: ConfigMaps ↗ · Kubernetes docs: Secrets ↗ · Kubernetes docs: Configure a pod to use a ConfigMap ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.