Operational observability

Observability

Questions first

• Before instrumenting, write 2–4 questions the on-call person will ask (why do payments fail? is the provider slower than usual?). Every signal answers one of them.
• Metrics say that something is wrong, traces say where, logs say why.

Structured logs

• Events, not prose: a stable event name and fields (payment_failed, paymentId, provider, errorCode, attempt), through the repo's logger.
• Levels: error when an invariant broke and someone may need to act; warn for degraded but handled (a retry succeeded, a fallback was used); info for business events; debug off in production.
• A correlation id is taken or created at the boundary and attached to every line, span, outgoing call and job. Where one log has several entry points (scheduler, webhook, manual run), stamp an entryPoint field at the start and pass it along.
• Log an error once, where it is handled, with its context. Never log secrets, tokens, passwords, cookies, auth headers, full request bodies or personal data; log an allowlist of fields.

Metrics, traces, alerts

• RED for every endpoint and external dependency: rate, errors, duration as a histogram. USE for pools, queues and hosts: utilization, saturation, errors. Read p50, p95 and p99, never averages.
• Labels come from small fixed sets (route template, status class 2xx/5xx, provider). Never user ids, raw URLs or error text as labels; they belong in logs.
• Traces with OpenTelemetry or the repo's tracer; propagate context across HTTP and queue boundaries; manual spans only around meaningful units of work.
• Alert on symptoms users feel (error rate above 1% for 5 minutes, p99 over 2 s, queue age over 10 minutes), not on causes such as CPU. Each alert is actionable, links a short runbook (what it means, first check, who to call), and is either a page or a ticket.
• Health checks: liveness says the process runs; readiness checks its dependencies.
• Verify the telemetry: force an error in staging and find it by request id, check the new metric series and labels, follow one request through the trace, fire each new alert once.

Performance: measure, fix one thing, re-measure

• Turn the symptom into a number (endpoint, percentile, throughput, memory) and record a baseline with realistic data volume and concurrency.
• Find the bottleneck before changing code: one endpoint slow, look at its queries and indexes; everything slow, the connection pool, memory or CPU; intermittent, locks, garbage collection or a dependency. Use a profiler, the slow query log and traces.
• Queries: N+1 is the usual suspect; fetch related rows in one query and paginate every list. EXPLAIN ANALYZE before and after an index: a sequential scan where an index was expected, row estimates off by 10x (stale statistics, run ANALYZE), a sort above the scan. Index equality columns first, then the range or sort column.
• Connection pools: when every endpoint slows at once and time goes to waiting for a connection, find what holds connections. Size pools so instances × pool size stays under the database's limit; a bigger pool only moves the queue.
• Cache only what is expensive and read far more than written; the key holds everything the response depends on (tenant, locale, permissions); state the staleness window; never cache balances, permissions or stock at checkout.
• Change one thing, measure the same way, repeat the runs. A gain inside the noise (3% against ±5% variance) is not a gain: revert it, as you revert a change that is worse or breaks a test. Report before and after numbers and the attempts you reverted.

Production issues

• Start from evidence: the exact error, the time window, who is affected, recent deploys and config changes; correlate by request id and time. Fix the root cause.
• Production is read-only for you: no writes, restarts, migrations or config changes without explicit permission. With permission and users hurt, mitigate first (roll back, flag off), then fix and add the test and signal that would have caught it.

Before you finish

• On-call questions written; each signal answers one.
• Logs structured with correlation ids; no secrets or personal data.
• RED metrics with bounded labels; alerts on symptoms with a runbook.
• Telemetry verified by a forced failure.
• Performance claims backed by before and after numbers.