← Blog/kubernetes

A Fingerprint, Not a Feature: the Helm Checksum Annotation

The checksum annotation is in every serious Helm chart and it does nothing at runtime - nothing reads it, no controller understands it. Its entire job is to make a change visible to a component that only compares bytes.

You edit a value, run helm upgrade, and it reports success. No pod restarts. The application keeps serving the old configuration. Nothing is broken, nothing errored, and nothing happened.

This is not a Helm bug. It falls straight out of how a Deployment decides that something changed.

What Kubernetes considers a change

A Deployment starts a rollout when spec.template changes. That is the whole rule. The controller hashes the pod template, and that hash becomes the pod-template-hash label identifying a ReplicaSet. Same template, same hash, same ReplicaSet - no new pods, no rollout.

Now look at how the template refers to configuration:

spec:
  template:
    spec:
      volumes:
        - name: config
          configMap:
            name: my-config # a name, not the contents 

That is a reference. You can rewrite every key inside my-config and this line still reads my-config. The template is byte-for-byte identical, so the hash is identical, so as far as the Deployment controller is concerned you did not change your application at all.

The ConfigMap is mutable; the reference to it is not versioned. That gap is the entire problem.

The trick: annotations are part of the template

Annotations under spec.template.metadata are part of the pod template, and Kubernetes does not interpret them. It has no idea what checksum/config means - it only sees whether the value differs from last time.

So you put something there whose value is derived from the config content:

spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

At render time Helm evaluates configmap.yaml as a template, hashes the rendered output with SHA-256, and pastes the digest in. The chain is mechanical:

  • config content changes → rendered output changes → digest changes → annotation changes → pod template changes → the controller sees a diff → rolling update.

And the punchline worth saying out loud in an interview: the annotation itself does nothing. No process reads it. No controller acts on it. It is a fingerprint whose only purpose is to be different when the config is different.

That framing is also what makes the naming clear. checksum/ is a community convention, not something Helm parses - foo/bar would work identically. The same line for secrets is the same idea:

checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

You have already used this mechanism

kubectl rollout restart deployment/x has no special "restart" API behind it. It writes an annotation into the pod template:

kubectl.kubernetes.io/restartedAt: "2026-08-25T10:14:03Z"

New timestamp, changed template, rolling update. Identical trick, shipped in kubectl. If someone asks how rollout restart works, that is the answer - and noticing that the checksum annotation is the same move applied to config is the thing that shows you understand the mechanism rather than the recipe.

Helm used to offer --recreate-pods for this. It was deprecated in Helm 3 in favour of the annotation, precisely because the annotation goes through the normal rolling update strategy instead of killing pods out from under you.

The mistake: putting it in the wrong metadata

metadata:
  annotations:
    checksum/config: abc123 # on the Deployment - does nothing 
spec:
  template:
    metadata:
      annotations:
        checksum/config: abc123 # on the pod template - this is the one

The top-level metadata.annotations belongs to the Deployment object. Changing it updates the Deployment and triggers no rollout whatsoever, because the pod template is untouched. This is the single most common way the trick silently does nothing, and it looks correct in review.

Whether you even need it depends on how the config is consumed

Three consumption modes, three different runtime behaviours - and only one of them updates on its own:

  • Environment variables (envFrom, configMapKeyRef): the docs are blunt - "ConfigMaps consumed as environment variables are not updated automatically and require a pod restart." Values are injected once at container start. You need the annotation.
  • Volume mount: the kubelet refreshes projected keys on its periodic sync, so files do change in the running container. The delay is "kubelet sync period + cache propagation delay". But the file changing is not the application noticing - unless your process watches the file or reloads on SIGHUP, the new bytes sit on disk unread.
  • subPath volume mount: "A container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates." Never, at any delay. You need the annotation.

The subPath case is the one that bites teams, because mounting a single file into an existing directory is the natural way to drop a config file next to a binary, and it quietly opts you out of every update.

So the honest rule: with env vars or subPath, the annotation is required for correctness. With plain volume mounts you are choosing between a hot reload (fast, applies everywhere at once, needs app support) and a rolling restart (slower, one pod at a time, works with any application). The restart is usually the right default because it is the one that fails safely - a bad config takes out one pod and stalls the rollout instead of every replica simultaneously.

What the fingerprint does not give you

A checksum makes a change visible. It does not make config versioned, and the difference shows up at exactly the wrong moment:

kubectl rollout undo deployment/my-app

That restores the previous pod template, old checksum annotation and all. The pods restart. But the ConfigMap object in the cluster still holds the new content, because the Deployment revision never contained the config - only a name and a hash of it. You get the old pod spec running the new configuration, with an annotation now advertising a digest that matches nothing.

helm rollback behaves correctly here, because Helm reapplies the previous release's manifests as a set - the ConfigMap reverts along with the annotation. That is a practical rule worth carrying: when config is involved, roll back with Helm, not with kubectl rollout undo.

If you want config that genuinely versions with the workload, the pattern is different: put the content hash in the name.

configMap:
  name: my-config-7b1d84f0 # a new object per generation 

Kustomize's configMapGenerator does this by default. Now the pod template changes because the reference changed, old ReplicaSets keep pointing at ConfigMaps that still exist, and rollout undo restores the whole picture. Pair it with immutable: true and the kubelet stops watching those objects entirely, which is a real API-server saving on large clusters. The cost is garbage: every generation leaves a ConfigMap behind, and something has to prune them.

Two patterns, one trade: Helm's checksum is one line and mutates in place; hashed names are real generations and real cleanup work.

When the checksum cannot see the config

The annotation only works for config the chart renders. If the ConfigMap is created by another release, applied by a platform team, or synced from an external secret store, $.Template.BasePath has nothing to hash, and a chart that hashes only its own values will happily report "no change" while the mounted content shifts underneath it.

That is the case for a watcher instead. A controller like Stakater's Reloader watches ConfigMaps and Secrets and patches the workloads that reference them:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Note that this one is read by something - a controller you installed. It is the opposite kind of annotation to the checksum, and worth keeping straight: one is inert data that Kubernetes diffs, the other is an instruction to a program that is watching.

Variants, and what each one misses

# hashes the rendered template - catches values AND template logic changes
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

# hashes a raw file in the chart - misses anything computed by the template
checksum/config: {{ .Files.Get "config/app.conf" | sha256sum }}

# hashes a values subtree - misses template changes and files entirely
checksum/config: {{ toYaml .Values.config | sha256sum }}

The first is the one to default to. The third is common and looks equivalent until someone edits the template rather than the values, and gets no rollout.

To check what a running pod is actually carrying:

kubectl get pod <pod> -o jsonpath="{.metadata.annotations['checksum/config']}"

Comparing that against a fresh helm template render tells you in one line whether a pod predates the config it is supposed to have - which is a genuinely useful debugging property that came along for free.

The takeaway

Kubernetes only redeploys when the pod spec changes, and the pod spec points at a ConfigMap by name - so changing what is inside it is invisible. The checksum annotation makes that change visible by hashing the config content into the pod template. Different content, different hash, different spec, automatic rollout.

The annotation does nothing. It is a fingerprint. Everything else - env vars not refreshing, subPath never updating, rollback not restoring content - follows from taking that sentence literally.

References

← All posts