← Back to Stellan
AgentStellan
Monitoring and actionable alerts
Connects service telemetry to user impact so operators can diagnose failures and act on alerts with clear responsibility.
monitoring-logging-alerting
See how it helpsWhen is it useful?
Use it when logs cannot be correlated or frequent alerts do not tell anyone what needs to be done.
What does it help you do?
- Maps latency, traffic, errors and saturation to service behavior.
- Connects structured logs through correlation identifiers without secrets.
- Pages on user impact with owners, runbooks and meaningful thresholds.
Try asking this
Replace noisy CPU alerts with an actionable stalled-queue alert. Pause a worker in a controlled test and observe increasing job age. Alert the responsible owner with queue identity, user impact and the recovery runbook.
Detailed documentation
Read the skill’s original instructions, examples and checks in their authored language.
Show less
Monitoring, Logging & Alerting
Observability exists to answer two questions quickly during normal operations and incidents: Are users experiencing a problem right now? and Where in the system is the bottleneck or failure?
The Four Golden Signals & USE/RED Metrics
Measure both user-facing service health (RED) and underlying resource health (USE):
| Signal | What to Measure | Why Averages Lie |
|---|---|---|
| Latency | Request & job duration histograms (p50, p95, p99), separated for successful (2xx) vs. failed (5xx) requests |
A fast 500 error (2ms) drags down average latency and hides 30s timeouts on 200 responses! |
| Traffic / Rate | Requests/sec, jobs enqueued/sec, active connections | Establishes baseline load and distinguishes traffic spikes from zero-traffic outages. |
| Errors | HTTP 5xx rate, unhandled exceptions, failed background jobs, dead-letter queue growth, synthetic check failures |
Direct indicator of broken user flows. |
| Saturation | DB connection pool utilization, queue depth / consumer lag, CPU throttling, memory working set vs. limit, disk Space/IOPS | Predicts impending failure before hard errors begin (often degrades steeply above 80–85% utilization). |
Structured, Correlated & Redacted Logging
- Single-line JSON structured logs in production:
- Include standard fields on every log entry:
timestamp(ISO-8601 UTC),level,service,version(git_sha),env,request_id/trace_id,job_id,duration_ms, anderror.code/error.stack.
- Include standard fields on every log entry:
- Propagate correlation IDs across boundaries:
- Accept or generate an incoming
x-request-id/ W3Ctraceparentheader at the edge, attach it to all logs, pass it to outbound HTTP calls and enqueued background jobs, and echo it in error responses so users and engineers can locate the exact trace.
- Accept or generate an incoming
- Never log secrets or high-cardinality PII:
- Redact
Authorizationheaders, cookies, session tokens, API keys, passwords, and raw payment/personal data at the logger serializer layer.
- Redact
Actionable Alerting Discipline
Every page/notification must require human action and point to a clear next step:
- Page on user-visible symptoms (SLO burn rate, high error rate, queue stall, disk near full) rather than noisy transient CPU spikes that self-heal.
- Mandatory alert metadata: Every alert rule must define:
- Impact: What user or system capability is degraded?
- Owner / Routing: Which role or channel is responsible?
- Runbook / First Triage Steps: Exact dashboard link, log query, and safe mitigation options (e.g., rollback, scale worker, drain node).
- Never silence a recurring alert permanently without fixing the underlying cause or recalibrating the threshold with measured evidence.
Included files 1
SKILL.md