LLM evaluation and observability
Makes LLM quality and failures inspectable by connecting execution traces to focused evaluations and reusable regression evidence.
llm-eval-observability
See how it helpsWhen is it useful?
Use it when an agent’s answer quality, tool choice or completion behavior deteriorates and the cause is unclear.
What does it help you do?
- Traces agent runs, turns, tools and usage hierarchically.
- Combines deterministic contracts, recorded trajectories and outcome evaluations.
- Classifies context, tool, runtime and model failures separately.
Try asking this
Diagnose a task regression and turn its redacted trace into an evaluation fixture. Replay the failing tool sequence against the previous and changed versions. Record the expected outcome and categorize the first divergence instead of blaming the model generally.
Detailed documentation
Read the skill’s original instructions, examples and checks in their authored language.
Show less
LLM Evaluation & Observability
Treat prompts, tool schemas, context compaction rules, and model selections as testable software artifacts. Never ship or modify agent behavior on "vibes"—verify changes with structured traces and reproducible evaluations.
Hierarchical Trace Instrumentation
Instrument every agent run as a correlated tree of spans (run_id $\rightarrow$ turn_id $\rightarrow$ llm_call / tool_call / retrieval_call):
| Span Type | Required Attributes to Record |
|---|---|
agent.run |
run_id, agent_role, workflow_phase, status, stop_reason, turn_count, total_cost_usd, wall_time_ms |
llm.generate |
provider, model, prompt_version, temperature, effort, ttft_ms, latency_ms, input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens, finish_reason |
agent.tool_call |
tool_name, tool_call_id, latency_ms, is_error, error_code, output_bytes, was_truncated, retry_count |
agent.compaction |
tokens_before, tokens_after, turns_compacted, compaction_latency_ms |
Support privacy-safe payload capture: store full prompt/response bodies in access-controlled artifact storage keyed by content hash so engineers can replay any failing turn without leaking secrets into standard metrics indices.
Three-Tier Evaluation Strategy
Combine fast deterministic checks with trajectory and outcome graders:
- Deterministic Unit & Contract Evals (CI on every PR):
- Schema conformance of structured outputs and tool call arguments.
- Exact assertion checks on deterministic fixtures (e.g., forbidden tools never called, required citation IDs present, no injected instructions followed).
- Trajectory & Tool-Selection Evals (Recorded / Mocked Tools):
- Replay a golden conversation prefix up to turn $T$ and assert that the next turn selects the expected tool category, passes valid parameters, and completes within step/token budgets (
pass@kandpass^kfor reliability across multiple trials).
- Replay a golden conversation prefix up to turn $T$ and assert that the next turn selects the expected tool category, passes valid parameters, and completes within step/token budgets (
- End-to-End Outcome & Rubric Evals:
- Grade the final artifact (e.g., tests pass in sandbox, SQL query returns expected rows, or pairwise LLM-as-judge rubric with explicit binary criteria and reference answers).
- Calibrate any LLM judge against human-labeled golden examples and measure agreement before trusting automated scores.
Failure Diagnosis & Flywheel
When diagnosing a failing agent run:
- Classify the root cause precisely: Context Deficit (needed info missing or compacted away), Tool Contract Flaw (ambiguous schema or unhelpful error message), Runtime/Control Bug (premature stop, timeout, broken state resume), or Model Capability Limit (reasoning failure despite clean context).
- Convert every production bug or regression into a permanent anonymized fixture in the evaluation dataset before closing the fix.
Included files 1
SKILL.md