Post-mortem: the readiness probe that lied

At 14:03 on a Tuesday, a customer's payments service went from a zero percent error rate to seventy. Kubernetes reported no unhealthy pods. Every replica was Ready.

Timeline

• 14:03 — Progressive rollout of 2.14.0 reached the fifty percent step.
• 14:05 — Error-rate alert fired. Primary on-call acknowledged.
• 14:11 — Rollback started. Errors continued.
• 14:29 — Database connection pool found saturated across every pod.
• 14:45 — Service restored with a raised maxPoolSize. Errors stopped.

What actually broke

The readiness probe pointed at /healthz, which returned a hardcoded 200:

app.get('/healthz', (_req, res) => res.status(200).send('ok'));

That probe could not fail. A pod that had just started and had not yet acquired a database connection was declared ready, and Kubernetes sent it traffic. A pod that could not do its job was being handed work to throw away.

Worse, the rollback hit the same wall. The old revision's pods were also declared ready instantly, and took the full flood of traffic before their pools had warmed. The rollback prolonged the outage rather than ending it.

The fix

Readiness now checks critical dependencies with a short timeout. Liveness deliberately does not:

readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 20
  failureThreshold: 6

The distinction matters. readyz asks "can I serve right now?" and healthz asks "am I alive at all?". Had healthz also checked the database, a slow database would have restarted every pod simultaneously, turning a forty-two minute outage into a multi-hour one.

Action items

Three items, each with an owner and a date. The important one was not fixing this service. It was adding a conftest rule that fails any pipeline where readinessProbe and livenessProbe point at the same path. An incident fixed for one service is still queued up for all the others.