Deployment verification and recovery
Verifies a release in its actual runtime and selects a recovery path that accounts for data and schema effects.
deployment-verification-recovery
Team members using this skill
When is it useful?
Use it when deployment has finished but health, active revision or safe recovery still need confirmation.
What does it help you do?
- Separates startup, liveness and readiness health checks.
- Verifies deployed revision and a representative live request.
- Chooses recovery based on schema changes and affected data.
Try asking this
Verify a release and plan recovery if its migration has already accepted writes. Confirm the active revision with a real request and inspect new-format records. Choose a forward fix or compatible rollback based on those records.
Detailed documentation
Read the skill’s original instructions, examples and checks in their authored language.
Show less
Deployment Verification & Recovery
A green CI build proves only that the code compiled and passed synthetic tests—it does not prove the release works in the target environment. Every deployment requires live health verification and a pre-planned recovery path.
Liveness vs. Readiness vs. Startup Probes
Never wire all health checks to a single endpoint that indiscriminately restarts healthy containers when a downstream dependency blips:
| Probe Type | Question Answered | What to Check | Failure Action by Orchestrator |
|---|---|---|---|
Startup (/health/startup) |
Has the process finished booting and loading config? | Env validation, local initialization, listening socket bound | Hold off liveness checks until startup completes or timeout expires |
Liveness (/health/live) |
Is the process alive and not deadlocked? | Event loop / process responsiveness only (no external DB/API calls!) | Restart the container/process if it fails repeatedly |
Readiness (/health/ready) |
Can this instance safely serve traffic right now? | DB connection pool reachable, migrations complete, not in graceful shutdown drain | Remove instance from load balancer rotation without killing it |
Post-Deploy Smoke Checks & Progressive Rollout
- Automated smoke verification:
- Immediately after deployment, verify:
- The deployed
/versionor health metadata matches the expectedgit_sha. - At least one representative end-to-end user or API read path succeeds (
200 OKwith valid payload schema). - Error rates (
5xx), startup crash loops, and p95 latency remain within baseline thresholds during the observation window.
- The deployed
- Immediately after deployment, verify:
- Zero-downtime rolling / blue-green / canary cutover:
- Keep old instances serving traffic until new instances pass readiness and smoke checks.
- During cutover, stop sending new requests to retiring instances, wait for in-flight HTTP requests and active jobs to finish (
SIGTERMdrain period), and only then terminate the old processes.
Rollback vs. Roll-Forward Decision Matrix
When a release degrades production, prioritize restoring service over debugging in place—while accounting for data and schema state:
| Failure Scenario | Safe Recovery Action | Critical Caveat |
|---|---|---|
| Stateless code bug, bad config, or broken asset bundle (no irreversible data mutation) | Immediate Artifact Rollback: re-point traffic or redeploy the previous known-good immutable image digest (sha-...) |
Verify config/env variables required by the previous version are still present. |
| Additive migration ran, new code crashes | Redeploy Previous App Artifact: leave the additive schema change (NULL column / new table) in place |
Never run automatic down migrations during an incident if they DROP columns or tables containing newly written data! |
| Data-corrupting write or destructive schema change | Contain & Roll Forward / Point-in-Time Recovery: pause offending writes/workers, assess affected records, and apply a tested forward fix or scoped data restore | Rolling back application code does not undo mutated database rows or sent external webhooks. |
Included files 1
SKILL.md