Runbooks and troubleshooting

Runbooks & Troubleshooting

Write operational and troubleshooting guides for someone diagnosing a problem under pressure: start from observable symptoms, give safe diagnostic checks first, and state the exact blast radius of every remediation command.

1. Structure around symptoms and observable signals

Organize troubleshooting entries so the reader can match what they see immediately:

• Symptom: the exact error message, HTTP status, log event, metric spike, stuck state or failing health check.
• Likely cause(s): what in the code, configuration, database or external dependency produces that symptom.
• How to confirm (read-only): the log query, status endpoint, database SELECT, CLI probe or file check that proves which cause applies without mutating state.
• Resolution: ordered steps to fix or recover, with the working directory, required permissions and target environment.
• Verification: how to confirm the service, queue or workflow has recovered and is processing normally.

2. Document background processes, queues and recovery

For daemons, scheduled tasks, queues and orchestrator loops, document:

• How the process starts, stops and recovers interrupted work on restart.
• The state machine (states, valid transitions, terminal states and blocked/error states).
• Timeouts, retry policies, backoff intervals, rate-limit cooldowns and dead-letter or manual retry paths.
• How to inspect stuck or orphaned runs and how to cancel, retry or clean them up safely.

3. Mark destructive or state-changing operations clearly

• Place a clear warning immediately above any command that deletes data, resets state, stops processes, rotates tokens, rolls back migrations or replays jobs.
• State:
  1. What it changes or deletes (and whether it is reversible).
  2. Which environment it is safe or intended for (local, staging, production).
  3. What backup or check to take before running it.

4. Keep diagnostics grounded in the codebase

• Every log pattern, health endpoint, error code, table name, setting key and recovery command in a runbook must exist in the repository.
• When an external dependency or production topology is not defined in the repo, mark the boundary explicitly rather than inventing infrastructure commands.

Before you finish

• Every troubleshooting item covers symptom, cause, read-only diagnosis, fix and verification.
• Background process lifecycle, states, retries and recovery behavior match the code.
• Destructive or state-mutating commands carry explicit consequence and environment warnings.
• All endpoints, log messages, config keys and CLI commands are verified against the codebase.