A Green Pipeline Should Mean a Healthy Pod

On most CD setups, “deploy succeeded” means one thing: helm upgrade exited with code zero. That is a much weaker claim than it sounds. Helm returning success only means the manifests were accepted by the API server. The new pod can still crash loop, fail to pull its image, get rejected by a resource quota, or hang on a readiness probe, and the pipeline stays green the whole time. Developers see the checkmark, move on, and find out half an hour later that the new version never actually came up.

On the microservices platform I work on, we decided a deploy stage in Azure DevOps is not allowed to turn green until the cluster proves the release is serving. Every service deploys through a shared template: helm upgrade, then a Verify Pod Health step that interrogates Kubernetes before declaring success. A few design decisions make it genuinely smart rather than just a sleep-and-check.

It verifies the exact artifact, not “some healthy pods”. The step computes the image tag the build just pushed and checks that the Deployment spec carries it. Checking “pods are ready” is worthless during a rollout, because the previous release’s pods are ready too. Pinning every check to the expected image is what turns “something is running” into “the thing we shipped is running”.

It walks the ownership chain the way a human would debug. Four checks, each answering one question: does the Deployment exist with the new image, did it create a ReplicaSet for that image, did the ReplicaSet produce pods, and is at least one of those pods Running with its Ready condition True. It follows ownerReferences rather than label selectors, so it never gets confused by lookalike resources.

When pods never appear, it reads the events for you. No pods is the most opaque failure in Kubernetes. The step pulls the ReplicaSet’s FailedCreate events and classifies the message into a named cause: INSUFFICIENT_CPU, QUOTA_EXCEEDED, FORBIDDEN, PVC_ERROR. The root cause lands in the pipeline log, no kubectl spelunking required.

Success is strict and borrowed from the app itself:

kubectl get pods -n "$NS" -o json | jq -r --arg img "$EXPECTED_IMAGE" '
  .items[]
  | select(.spec.containers[].image == $img)
  | select(.status.phase == "Running")
  | select(.status.conditions[]
      | select(.type == "Ready" and .status == "True"))
  | .metadata.name'

Ready means the readiness probe passed, and the probe is the application’s own definition of healthy. So a green stage means the service answered its own health check, not that a script guessed it was probably fine.

Failure is fast and honest. Transient states keep polling, unrecoverable ones (crash loops, bad image references) break out early, and the last meaningful diagnosis is what gets reported to Azure DevOps, so the run summary names the cause instead of “timed out”.

The takeaway: a CD pipeline’s job does not end when Helm returns. It ends when the cluster proves the new version is alive. Once the pipeline holds that line, a green run becomes something developers actually trust.