CI runners became prime targets after supply-chain compromises like SolarWinds and the CircleCI breach. Teams want the flexibility of self-hosted runners without inheriting attacker persistence. Firecracker microVMs and similar technologies offer a middle ground: ephemeral, isolated build sandboxes with hardware-level isolation. This post explains how we built secure runners for Cloudythings clients.
Immutable build images
Runner images are:
- Built via Packer or Bazel with distroless bases.
- Signed with Sigstore; SBOMs describe build tools.
- Scanned continuously for vulnerabilities.
Images contain build essentials (Git, languages, package managers) and nothing else. Developers request additional tooling via PRs, ensuring review.
MicroVM lifecycle
Each job spawns a microVM:
- Firecracker boots in ~150 ms from a snapshot.
- The VM mounts a temporary volume for workspace data.
- Credentials (OIDC tokens, secrets) inject at runtime and expire when the VM stops.
- After completion, disks shred, and the VM is destroyed.
No state persists between jobs. Attackers cannot linger.
Policy enforcement
We integrate:
- OPA policies to restrict network egress and enforce repo allowlists.
- Sigstore verification to ensure build scripts and dependencies are signed.
- Audit logging capturing every system call via eBPF (Tetragon) for forensic traceability.
If a job attempts to reach banned domains or run unsigned code, the policy controller terminates it and files an alert.
Developer experience
From the developer perspective:
- Jobs define requirements (CPU, memory, GPU) in YAML.
- Tool caches (language artifacts) store in remote caches to offset VM cold starts.
- CLI tools (
ct run build) trigger local builds using the same microVM image via Ignite for parity.
Metrics show minimal performance overhead compared to container-based runners, with vastly improved security posture.
Secure CI runners with microVMs blend isolation, immutability, and policy. They make supply-chain attackers work harder—and keep platform teams ahead of the curve.