Orqestria Download
← Back to the catalog

Skill · Version 1.0.0

LLM evaluation and observability

Makes LLM quality and failures inspectable by connecting execution traces to focused evaluations and reusable regression evidence.

llm-eval-observability

See how it helps

When is it useful?

Use it when an agent’s answer quality, tool choice or completion behavior deteriorates and the cause is unclear.

What does it help you do?

  • Traces agent runs, turns, tools and usage hierarchically.
  • Combines deterministic contracts, recorded trajectories and outcome evaluations.
  • Classifies context, tool, runtime and model failures separately.

Try asking this

Diagnose a task regression and turn its redacted trace into an evaluation fixture. Replay the failing tool sequence against the previous and changed versions. Record the expected outcome and categorize the first divergence instead of blaming the model generally.

Detailed documentation

Read the skill’s original instructions, examples and checks in their authored language.

Details…Show less

LLM Evaluation & Observability

Treat prompts, tool schemas, context compaction rules, and model selections as testable software artifacts. Never ship or modify agent behavior on "vibes"—verify changes with structured traces and reproducible evaluations.

Hierarchical Trace Instrumentation

Instrument every agent run as a correlated tree of spans (run_id $\rightarrow$ turn_id $\rightarrow$ llm_call / tool_call / retrieval_call):

Span Type Required Attributes to Record
agent.run run_id, agent_role, workflow_phase, status, stop_reason, turn_count, total_cost_usd, wall_time_ms
llm.generate provider, model, prompt_version, temperature, effort, ttft_ms, latency_ms, input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens, finish_reason
agent.tool_call tool_name, tool_call_id, latency_ms, is_error, error_code, output_bytes, was_truncated, retry_count
agent.compaction tokens_before, tokens_after, turns_compacted, compaction_latency_ms

Support privacy-safe payload capture: store full prompt/response bodies in access-controlled artifact storage keyed by content hash so engineers can replay any failing turn without leaking secrets into standard metrics indices.

Three-Tier Evaluation Strategy

Combine fast deterministic checks with trajectory and outcome graders:

  1. Deterministic Unit & Contract Evals (CI on every PR):
    • Schema conformance of structured outputs and tool call arguments.
    • Exact assertion checks on deterministic fixtures (e.g., forbidden tools never called, required citation IDs present, no injected instructions followed).
  2. Trajectory & Tool-Selection Evals (Recorded / Mocked Tools):
    • Replay a golden conversation prefix up to turn $T$ and assert that the next turn selects the expected tool category, passes valid parameters, and completes within step/token budgets (pass@k and pass^k for reliability across multiple trials).
  3. End-to-End Outcome & Rubric Evals:
    • Grade the final artifact (e.g., tests pass in sandbox, SQL query returns expected rows, or pairwise LLM-as-judge rubric with explicit binary criteria and reference answers).
    • Calibrate any LLM judge against human-labeled golden examples and measure agreement before trusting automated scores.

Failure Diagnosis & Flywheel

When diagnosing a failing agent run:

  • Classify the root cause precisely: Context Deficit (needed info missing or compacted away), Tool Contract Flaw (ambiguous schema or unhelpful error message), Runtime/Control Bug (premature stop, timeout, broken state resume), or Model Capability Limit (reasoning failure despite clean context).
  • Convert every production bug or regression into a permanent anonymized fixture in the evaluation dataset before closing the fix.

Included files 1

  • SKILL.md

About this skill package

llm-eval-observability

llm-eval-observability: Build AI agent features with explicit tool contracts, controlled permissions, durable state and observable model behavior.

Details…Show less

llm-eval-observability

llm-eval-observability: Build AI agent features with explicit tool contracts, controlled permissions, durable state and observable model behavior.

Includes the complete skill and its supporting files (1 files). The exact authored scope is preserved below.

Original skill description (source language)

Use when instrumenting LLM and agent runs, designing offline or online evaluations, building regression datasets from production traces, or diagnosing agent failures, latency bottlenecks, and quality regressions.

Install the skill, review its source and trust settings, then enable it for the relevant project or agent.

Requirements

  • Orqestria installed on your computer.