Distroless Delivery: Shipping Signed Containers You Can Trust

“We build containers that are smaller than a GIF, yet an attacker still found BusyBox in one of them.” That was the opening line from a post-incident review a platform team shared with us in early 2021. Their story echoed research that Google’s Ian Lewis and Chainguard’s Dan Lorenc have voiced repeatedly on Medium and KubeCon stages: supply-chain threats grow every time we ship an overstuffed container image. The antidote is distroless builds backed by signature verification and policy-as-code.

Distroless means stripping an image down to the runtime essentials—no package manager, shell, or glibc baggage. But distroless alone does not guarantee trust. Without cryptographic signatures and enforcement at deployment time, a bad actor can still slip in a malicious image tag. The magic happens when we combine three pillars:

1. Distroless base images curated by Google, Chainguard, or your internal platform team.
2. Signing and transparency via Sigstore (Cosign, Fulcio, Rekor) or Notary v2.
3. Admission control & policy to verify signatures before Kubernetes schedules a pod.

Let’s walk through how to adopt distroless delivery without breaking developer experience.

Start with a minimal, observable build

The first question every developer asks: “How do I debug a distroless container if there is no shell?” The answer lies in instrumenting the application instead of diving into the container. That means:

• Build with multi-stage Dockerfiles. Compile your binaries in a builder stage with full tooling, copy only the final binary into the distroless stage.
• Add structured logging and OpenTelemetry instrumentation so you can diagnose issues from the outside. Charity Majors’ observability-first mantra applies; you should not need bash to debug.
• Bundle health checks and profiling endpoints (e.g., /debug/pprof) when appropriate, exposing them via secure sidecars or service mesh routes.

We recommend a build pipeline that leans on Bazel, ko, or CNCF Buildpacks—tools that understand minimal base images. Chainguard’s Wolfi distribution and Google’s Distroless project both ship images preloaded with CA certificates, tzdata, and minimal locale data so you are not reinventing the wheel.

Introduce signatures as part of the build contract

Signing should feel invisible to developers. We wire Sigstore’s Cosign into the CI pipeline using workload identities so secrets remain out of band:

cosign sign --identity "https://github.com/cloudythings/workflows/.github/workflows/build.yml@refs/heads/main" \
  --key k8s://sigstore/system/cosign-key \
  ghcr.io/cloudythings/payment-service:${GIT_SHA}

Key practices we learned from Chainguard’s and Shopify’s supply-chain write-ups:

• Adopt keyless signing with workload identity (GitHub OIDC, SPIFFE). This removes long-lived signing keys while providing traceability.
• Log signatures to Rekor transparency logs so forensic teams can audit changes. Rekor’s append-only guarantee mirrors the “immutable logs” auditors ask about.
• Attach SBOMs (CycloneDX or SPDX) during signing by using cosign attach sbom. Storing these in the registry or artifact repository creates a single lookup point.
• Treat signing as a release gate. If Cosign fails, the build fails. No exceptions.

Medium’s infrastructure team documented how they plumbed Cosign into GitHub Actions with minimal friction; we replicated their pattern for several Cloudythings clients by shipping a reusable .github/workflows/container-sign.yml composite action.

Verify at deployment—every time

Signing alone does not stop drift. You must enforce verification in Kubernetes. We deploy a combination of:

• Policy controllers: Sigstore’s cosigned mutating webhook or Kyverno policies that evaluate Cosign signatures.
• RuntimeClass isolation: Pairing distroless images with gVisor or Firecracker microVM runtime classes (as AWS’s Firecracker team suggests) to reduce the kernel attack surface.
• Admission policies: OPA Gatekeeper constraints ensuring images originate from approved registries and include the right annotations (e.g., image-signature/cloudythings).

A sample Gatekeeper constraint template:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8scosigned
spec:
  crd:
    spec:
      names:
        kind: K8sCosigned
      validation:
        openAPIV3Schema:
          type: object
          properties:
            identities:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8scosigned
        violation[{"msg": msg}] {
          input.review.kind.kind == "Pod"
          container := input.review.object.spec.containers[_]
          not sigstore.cosign.verify(container.image, input.parameters.identities)
          msg := sprintf("image %s is not signed by an approved identity", [container.image])
        }

We configure identities to match the workload identities used during signing. Gatekeeper’s external data providers or Cosign’s policy-controller project handle the verification mechanics.

Bake verification into GitOps

GitOps multiplies the power of signed, distroless images. When Argo CD or Flux applies manifests, we ensure the digests point to signed images. Promotion pull requests run two categories of checks:

• Static checks: cosign verify --policy ensures the manifest’s digest matches a signed artifact.
• Conftest or OPA unit tests: Validate that the manifest references runtime classes, seccomp profiles, and read-only root filesystems.

After merge, Argo CD’s reconciliation triggers Kubernetes policy evaluation. If verification fails, the sync status becomes OutOfSync, and our on-call SREs are paged. We also annotate Grafana dashboards with the PR number that introduced the version, making supply-chain audits quick.

Do not forget developer experience

Developers will love the security posture if they can still iterate quickly:

• Provide container debug images with BusyBox and tooling for sandbox environments only. Access is gated through namespaces like dev-debug. This keeps prod images pure.
• Offer ko or Buildpacks CLIs preconfigured to target the distroless base. We wrap these into make targets so local building feels familiar.
• Document observability expectations. If there is no shell, logs and metrics must be rich. We embed OpenTelemetry exporters by default and point teams toward Honeycomb or Grafana Tempo exemplars—practices widely endorsed across Medium engineering posts.
• Automate signature verification in local dev. Developers can run make verify-image to validate their build before pushing, mirroring what the pipeline enforces.

Extend signatures beyond containers

Images are only one artifact in the release supply chain. We also sign:

• Manifests: Using cosign sign-blob on Kubernetes YAML or Terraform plans, then verifying signatures in GitOps pipelines. This ensures the manifest entering production is the one we reviewed.
• SBOMs: Many compliance frameworks (e.g., US Executive Order 14028) now require SBOM distribution. Signing them proves provenance.
• Policies: Gatekeeper and Kyverno policies are versioned and signed so unauthorized edits are rejected. This protects the guardrails themselves.

Supply-chain leaders like Chainguard, Harness, and GitLab have published Medium articles arguing for “defense in depth signing.” We agree: treat every artifact as tamper-prone until proven otherwise.

Measure and iterate

How do you know the investment worked? Track:

• Unsigned image attempts blocked per week. A rising number signals either adoption gaps or attempted attacks. We feed this metric into Honeycomb to spot trends.
• Time from CVE disclosure to patch deployment. Distroless images reduce patch surface, but you still need automation. Aim for <48 hours.
• Image size reduction. Smaller images cut startup times and reduce the blast radius of vulnerabilities. Teams often see 60–80% reductions.
• Developer satisfaction. We run quarterly surveys; if developers feel slowed down, we partner with them to streamline workflows. The best ideas—like shipping ready-to-use skaffold configs—have come from these sessions.

Recommended adoption timeline

1. Inventory existing images. Use syft or tern to classify base images and dependencies.
2. Introduce signing in CI. Start with a pilot service; integrate Cosign with workload identity.
3. Deploy policy controllers in staging. Monitor for blocks. Publish a migration guide.
4. Flip enforcement in production. Make policy failures loud. Celebrate teams that remediate quickly.
5. Expand to manifests and SBOMs. Once containers are covered, extend the same controls.
6. Continuously educate. Lunch-and-learns, runbooks, and pair programming keep everyone on the same page.

Further learning

• Chainguard’s blog offers deep dives on Wolfi, Sigstore, and distroless best practices.
• Shopify’s Medium engineering story on “Life after SolarWinds” documents how they rebuilt their signing pipeline, a valuable template for any platform team.
• Google’s distroless GitHub repository includes samples and guidance for language-specific builds.
• Sigstore documentation explains transparency logs and keyless workflows in depth.

Distroless delivery is not about shrinking images for sport; it is about hardening every step between developer intent and production reality. When signatures, policy, and GitOps guardrails work in concert, the trust gap closes. Your platform becomes resilient against tampering, auditors see verifiable evidence, and developers ship with the confidence that their code will run exactly as intended—nothing more, nothing less.