Structured AI output and streaming

Structured Outputs & Streaming

Never parse free-form LLM prose with brittle regexes when downstream application code expects typed data. Use constrained decoding or strict tool-input schemas paired with runtime validation and explicit refusal/truncation handling.

Enforcing Schema-Conformant Outputs

1. Choose the right mechanism:
   • Native Structured Output (response_format / output_config.format with json_schema): Best when the final assistant response itself must conform to a strict schema.
   • Strict Tool Call (strict: true): Best when the model may either invoke helper tools or emit a final structured payload via a terminal submission tool (submit_review, propose_plan).
2. Design schemas that help the model reason:
   • Order fields so reasoning/evidence fields (analysis, evidence_paths, root_cause) appear before conclusion or classification fields (severity, verdict, action). In autoregressive generation, early fields condition the tokens generated for later fields.
   • Keep schema nesting shallow ($\le 3$ levels), set "additionalProperties": false, use explicit enum values instead of open strings, and avoid complex regex patterns or recursive self-references that break provider grammar compilers.
3. Always validate at runtime (Zod / Pydantic / Ajv):
   • Even with constrained decoding, check finish_reason / stop_reason before parsing JSON:
     • max_tokens / length: The JSON stream was cut off mid-object. Do not attempt to treat partial JSON as complete—either increase max_tokens, request a more compact output, or trigger a continuation/repair turn.
     • refusal / content_filter: Surface a typed refusal result rather than throwing a SyntaxError: Unexpected token on non-JSON refusal text.
   • Enforce domain invariants that JSON Schema cannot express (e.g., cross-field consistency, referenced file paths actually exist in the workspace). If domain validation fails, feed the exact validation errors back to the model in a bounded repair turn (max_repair_attempts = 2).

Streaming Deltas & Partial Tool Calls

When streaming responses to a UI or downstream consumer over SSE / WebSockets / JSONL:

• State-tracked event reducer: Process provider stream events through a deterministic state machine (message_start $\rightarrow$ content_block_start $\rightarrow$ content_block_delta (text_delta / input_json_delta / thinking_delta) $\rightarrow$ content_block_stop $\rightarrow$ message_delta $\rightarrow$ message_stop).
• Never execute a tool on partial input_json_delta chunks: Buffer input_json_delta fragments per content_block index and parse/validate the tool arguments only after receiving content_block_stop and confirming the turn was not truncated.
• Incremental UI rendering: If rendering progressive structured fields in a UI, use a tolerant streaming JSON parser strictly for display preview, while gating all state mutations and persistence on the final validated object at stream completion.
• Backpressure & disconnect safety: Decouple the upstream LLM stream reader from slow UI WebSocket/SSE clients so a disconnected browser tab does not corrupt the server-side agent turn record in the database.