Deployment verification and recovery

Verifies a release in its actual runtime and selects a recovery path that accounts for data and schema effects.

deployment-verification-recovery

See how it helps

When is it useful?

Use it when deployment has finished but health, active revision or safe recovery still need confirmation.

What does it help you do?

Separates startup, liveness and readiness health checks.
Verifies deployed revision and a representative live request.
Chooses recovery based on schema changes and affected data.

Try asking this

Verify a release and plan recovery if its migration has already accepted writes. Confirm the active revision with a real request and inspect new-format records. Choose a forward fix or compatible rollback based on those records.

Detailed documentation

Read the skill’s original instructions, examples and checks in their authored language.

Details…Show less

Deployment Verification & Recovery

A green CI build proves only that the code compiled and passed synthetic tests—it does not prove the release works in the target environment. Every deployment requires live health verification and a pre-planned recovery path.

Liveness vs. Readiness vs. Startup Probes

Never wire all health checks to a single endpoint that indiscriminately restarts healthy containers when a downstream dependency blips:

Probe Type	Question Answered	What to Check	Failure Action by Orchestrator
Startup (`/health/startup`)	Has the process finished booting and loading config?	Env validation, local initialization, listening socket bound	Hold off liveness checks until startup completes or timeout expires
Liveness (`/health/live`)	Is the process alive and not deadlocked?	Event loop / process responsiveness only (no external DB/API calls!)	Restart the container/process if it fails repeatedly
Readiness (`/health/ready`)	Can this instance safely serve traffic right now?	DB connection pool reachable, migrations complete, not in graceful shutdown drain	Remove instance from load balancer rotation without killing it

Post-Deploy Smoke Checks & Progressive Rollout

Automated smoke verification:
- Immediately after deployment, verify:
  1. The deployed /version or health metadata matches the expected git_sha.
  2. At least one representative end-to-end user or API read path succeeds (200 OK with valid payload schema).
  3. Error rates (5xx), startup crash loops, and p95 latency remain within baseline thresholds during the observation window.
Zero-downtime rolling / blue-green / canary cutover:
- Keep old instances serving traffic until new instances pass readiness and smoke checks.
- During cutover, stop sending new requests to retiring instances, wait for in-flight HTTP requests and active jobs to finish (SIGTERM drain period), and only then terminate the old processes.

Rollback vs. Roll-Forward Decision Matrix

When a release degrades production, prioritize restoring service over debugging in place—while accounting for data and schema state:

Failure Scenario	Safe Recovery Action	Critical Caveat
Stateless code bug, bad config, or broken asset bundle (no irreversible data mutation)	Immediate Artifact Rollback: re-point traffic or redeploy the previous known-good immutable image digest (`sha-...`)	Verify config/env variables required by the previous version are still present.
Additive migration ran, new code crashes	Redeploy Previous App Artifact: leave the additive schema change (`NULL` column / new table) in place	Never run automatic `down` migrations during an incident if they `DROP` columns or tables containing newly written data!
Data-corrupting write or destructive schema change	Contain & Roll Forward / Point-in-Time Recovery: pause offending writes/workers, assess affected records, and apply a tested forward fix or scoped data restore	Rolling back application code does not undo mutated database rows or sent external webhooks.

Included files 1

SKILL.md