Pipeline anatomy

A good pipeline is boring. No stage should surprise you, and no stage should do more than its name implies.

Four stages

Our default pipeline has four: verify, build, stage, and promote. The first three run on every merge request; the fourth runs only on the default branch.

stages: [verify, build, stage, promote]
 
verify:
  stage: verify
  script:
    - npm ci
    - npm run lint
    - npm test -- --coverage
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
 
build:
  stage: build
  script:
    - docker build -t $IMAGE:$CI_COMMIT_SHA .
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE:$CI_COMMIT_SHA
    - cosign sign --yes $IMAGE:$CI_COMMIT_SHA

Note that trivy runs with --exit-code 1. A scan that cannot fail the build is decoration.

Digests, not tags

After the build, the image is referenced by digest rather than tag. Tags are movable; digests are not. That is the difference between knowing what is running and assuming it.

What it actually costs

Thirty-day averages across the services we manage:

If verify takes longer than five minutes, engineers stop waiting for it. They context-switch, and the review loop stretches from minutes to hours. Pipeline latency is a product feature, not an ops metric.

Progressive delivery

promote shifts traffic in four steps: five, twenty-five, fifty, one hundred percent. Between each step it checks the error budget, and rolls back on its own if the budget is being burned.

mizban rollout status payments --watch
mizban rollout undo payments --to-revision 41