Files
pipelines/docs/model-results.md
T
alan 7b72b8f20f stage: retry pre-admission harness reach failures and widen repair deadline
Treat a failure to reach the harness before any prompt or tool call is
admitted (connecting or creating the session) as a distinct retryable
reach failure, so a transport blip at the first stage no longer ends a
mutating run; keep post-admission readback/poll/agent failures retried
only on an explicit read-only stage and never replay admitted tool work.
Preserve the true cause when the context is already done (join ctx.Err
with the typed transport error) and report a sanitized unknown provider
error name/status instead of a blanket summary.

Make the isolated format-repair deadline proportional to the stage
timeout (floored, capped and clamped to remaining minus a margin) so a
slow-but-repairable result is recovered without ever extending the stage
deadline; the bounded ladder, value guard, deny-all formatter session,
cumulative usage accounting and no-tool-replay rule are unchanged.

Update docs/model-results.md, docs/pipeline-performance-review.md and the
packaged skill references; add stage, opencode and server fixtures.
2026-10-17 14:51:09 +01:00

14 KiB

Model results and scoped tools

Every model result must be one object containing exactly outcome and output. outcome must be a declared edge; output must be an object. Duplicate keys (including nested output keys), missing fields, unknown fields and invalid outcomes are rejected. Free-form output contract prose is not a JSON Schema.

The native stable OpenCode adapter first checks the complete format POST/readback contract in a separate retained, deny-all, no-inference session. Compatible native backends request format.type=json_schema with this envelope and retryCount=0. OpenCode 1.18.35 accepts a format but rejects its stored messages with Expected OutputFormatJsonSchema (even explicit text fails). For that specific incompatibility, requests omit format; strict local validation remains mandatory. Other probe failures, such as authentication or network errors, are not silently downgraded. Returned structured results remain preferred when available, and tool transcripts are preserved. V2's published prompt/generate API does not accept a schema-output field; that adapter retains local validation instead of sending unsupported fields or claiming provider-side schema enforcement.

Output-contract prose describes the map inside output; it never overrides the runtime {outcome, output} envelope. Printed DSML or other tool-call markup is not evidence of a real tool invocation and cannot be accepted as a result. Flattened output fields are rejected, not guessed into a new envelope. Invalid responses are retained even when they are not eligible for syntax repair.

Line-oriented result format

Model and agent stages also accept a line-oriented result format. The response is a markdown-ish document whose first non-empty line is the outcome, followed by sections introduced with # <heading>.

success

# output
{"task": "x"}

# findings
- prose, code fences, quotes and embedded JSON need no escaping

Only the first non-empty line and a # output section are required. The body of # output must be a JSON object; it becomes the stage's output map. Every other heading becomes an evidence field keyed by the normalised heading text (lowercased, spaces to underscores). Extra sections are stored verbatim and are best-effort: an empty heading, an odd heading, or a very large section must never fail the stage. If the outcome is missing, not one of the declared edges, or # output is absent, the stage fails with FINAL RESULT VALIDATION and still retains whatever sections were salvaged.

The parser decides between the legacy JSON envelope and the line-oriented format by precedence. If the response's first non-empty character is { after stripping an optional surrounding markdown fence, it keeps the unchanged JSON path. If a # heading appears before any {, it uses the line-oriented path. This preserves all existing JSON responses, including fenced JSON or short prose prefixes, while allowing new line-format responses to coexist.

The full raw response is written to the stage's artifacts directory as raw-output.md before parsing, so the closest record of what the model produced is always retained and appears in the step's artifact inventory automatically. The parse status is recorded in the step's output map under the reserved key result_parse: parsed (both required parts OK), partial (required parts OK but some extra section unusable), or failed (a required part missing or invalid).

Format recovery is not execution retry

Parser errors retain their diagnostic and byte offset: for example, missing closing brace '}', an unterminated string, or an invalid character. A valid but schema-invalid result is not reformatted to invent a missing field or verdict.

Format recovery has a separate bounded budget, even with retries=0:

  1. A deterministic local pass fixes only the structural error that is derivable from the payload itself, in order: append up to eight missing closing delimiters when the existing response consists of complete JSON tokens; remove a single unambiguous trailing comma before a closing }/]; or insert exactly the one comma/colon the parser expected at its reported byte. Each candidate must pass lossless syntax validation and the full envelope validation. No model call. An ambiguous payload - an unescaped quote or colon inside a string value, or a stray delimiter with no single completion - is never guessed at and falls through.
  2. If the local pass cannot repair it, make at most one fresh, deny-all, retained text-generation request with only the malformed response and precise parsing error. The original implementing/reviewing agent is never re-prompted. The formatter's deadline is derived from the payload size, the stage timeout and the attempt's own remaining budget rather than a fixed short constant, so a large or slower result is not guaranteed to time out; it stays within the stage deadline (clamped to remaining minus a margin), is floored for ordinary payloads, and keeps the 32 KiB input/output bound.
  3. Accept only insertion of missing commas/colons or trailing closing delimiters, with at most 16 insertions. Every existing string, number, literal and structural delimiter must remain verbatim and in order. Validate the full result again. Changed approval, outcome, evidence, reordered data, deleted fields, incomplete strings and fabricated values are rejected.

Every terminal path records an explicit recovery status: repaired (a fully lossless local or formatter recovery), partial (the candidate is valid JSON but still fails envelope validation, so it is never returned as a completed result), or unrecoverable (nothing recovered, or the formatter errored or timed out). The unrecoverable path retains the bounded original payload together with the precise parse error and its byte offset, and the status is included in the FINAL RESULT VALIDATION diagnostic so an operator can see exactly what the model emitted.

If recovery fails, report the original precise error and repair failure; never rerun project edits, commands, or the entire stage to repair final formatting. Configured execution retries remain for retryable text-generation transport errors; an explicit read-only stage may additionally re-emit a malformed final result and retry a post-admission deadline or transport failure up to retries. A failure to reach the harness before any prompt or tool call is admitted (connecting, or creating the session) is a distinct reach failure: nothing was admitted, so it is retried within retries for any stage, mutating or not, with a fresh session and no replayed prompt or tool call. A post-admission readback, active-session poll or agent-execution failure is retried only on an explicit read-only stage. A stage whose tools include edit/shell, or that has no tools block (the legacy read/glob/grep/edit/shell bundle), never has admitted tool work replayed. The deterministic local pass and the fresh deny-all, tool-free formatter session never re-prompt or replay the admitted stage session, so a shell-enabled review stage gains malformed-result recovery without any replay of admitted work. Repair records include the original/candidate response, parsing diagnostic, method, recovery status, byte offset and acceptance flag in the bounded transcript preview. pipelines inspect <run-id> --json and the TUI evidence section expose these records.

OpenCode CLI failures include the HTTP status when supplied, process exit code, and a recognized safe error category (such as session not found or service unavailable). Arbitrary stderr, provider messages and response bodies are not copied into the diagnostic. A failed V2 active-session poll (or native V1 /wait) is not evidence that the agent prompt failed to start, and does not trigger replay of stage work. When the context expires while a request is still in flight, the error keeps both the sanitized transport detail (operation/status/exit) and errors.Is(err, context.DeadlineExceeded), so a deadline that fired during a hung transport call is not replaced by a bare context error; a deadline measured at the attempt boundary with no transport in flight stays the stage overrun it is. Native StructuredOutputError is named directly, rather than reduced to a generic command failure, and an unrecognized provider execution-error name is reported as a bounded, sanitized identifier plus any 4xx/5xx status instead of a blanket summary (a name containing arbitrary bytes stays the generic message). Native HTTP evidence records the actual HTTP operation/status; an exit_code of -1 there is explicitly marked as not a subprocess exit. Cleanup failures preserve the original generation error and active ownership.

For existing native sessions with format-bearing messages, the HTTP readback workaround is read-only CLI opencode export. The exported session ID must match the requested ID. Nothing is rewritten or removed from the database/history. Debug receives the original messages plus a note identifying that recovery path.

Failure diagnostics and confirmed limits

Retained stage failures distinguish two fixed, sanitized categories in the step diagnostic:

  • FINAL RESULT VALIDATION for a strictly invalid final result, with the stage key and the precise parser detail (for example duplicate JSON field "score" or invalid character 'z' after object key:value pair at byte N). It states that completed edits/tool side effects are retained and not undone, and that a full retry repeats work.
  • HARNESS TRANSPORT for request, V2 active-session poll and final-readback failures, including the sanitized malformed or incomplete readback message. A single classifier covers the two transport shapes: a pre-admission reach failure (connecting or creating the session) and a post-admission failure (readback/poll/agent execution). Both share the one HARNESS TRANSPORT label; only the pre-admission shape is retried on a mutating stage, because it replays nothing.

A structured-output validation failure is never reported as a harness transport failure, and vice versa. Neither label includes a provider response body, prompt, credential or arbitrary model content. Duplicate nested fields are rejected by strict parsing rather than merged or last-wins, and only a lossless insertion of missing commas/colons or trailing closers is repaired, so an unknown evidence or verdict value can never become success, and admitted tool work on a mutating or legacy-bundle stage is never replayed (only an explicit read-only stage may re-emit a malformed result within retries).

Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned exactly 262144 bytes of incomplete JSON (invalid agent messages); a reviewer result was rejected for a duplicate nested performance_review field; and an onboarding visual-review result failed with invalid character 'z' after object key:value pair at byte 3968, where unsafe lossless repair was correctly rejected.

A shell-enabled ui_reviewer stage (run 55, pipeline 9 v6, 2026-10-15) failed with invalid character ':' after object key:value pair at byte 7837, where the isolated formatter hit its own 30-second deadline. The malformed payload was retrieved read-only with pipelines inspect 55 --json and reproduced the exact byte offset 7837. It is an unescaped quote/colon inside a string value - an ambiguous case - so the deterministic pass correctly reports it unrecoverable rather than guessing, while the stage-timeout-proportional derived deadline (floored, capped and clamped to the remaining attempt budget) removes the fixed-30-second-deadline failure mode for larger or slower payloads. These are source-level findings covered by mocked internal/stage fixtures. An updated checkout or installed skill is not evidence that a user's running server contains the fix; deploying/restarting is a separate authorized action.

OpenCode V2's published prompt/generate API does not accept a structured-output field, so a prose output_contract does not enforce structured output: V2 results are validated locally by strict parsing only, and no unsupported V2 format field is sent. A review-only fresh execution that could re-derive a failed verdict without replaying the admitted tool-using stage's edits is a future improvement and is not implemented. Two items remain unresolved and must not be reported complete: a macOS fixture path-spelling mismatch (/var/folders versus /private/var/folders are the same real directory) in scripts/test_setup_harness.py, and onboarding UI visual approval.

Optional machine-readable restrictions

tools enables actions per stage. A stage can narrow edit paths and shell commands with an optional permissions block:

```tools
read
glob
grep
edit
shell
```

```permissions
{"edit":["README.md","plugin/*.qml"],"shell":["bash -n install.sh"]}
```

These rules become OpenCode tool permissions, not just prompt advice. Edit patterns are relative to the run directory and intersect any optional run-level --write-paths. Shell entries are exact commands: no wildcards, chaining, redirection or expansion. Empty lists deny an action; an omitted list retains the normal defaults. Null lists, unknown keys, duplicate keys/resources, path traversal, undeclared tools and unrestricted shell alongside restricted edits are rejected. At most 20 resources per action are accepted.

OpenCode checks scanner-produced shell command resources, not necessarily the entire submitted script. A compound script consisting only of individually allowed commands may pass; this is not a raw-script or shell-syntax firewall.

For a deliberately Go-only stage, explicitly declare edit:["*.go"] and the exact allowed verification commands. Without a machine policy, a sentence such as “approved Go files only” is not an enforced permission boundary. Defaults remain language-neutral: this does not restore a global Go-only restriction or require users to enumerate files for ordinary runs. Existing immutable definitions are not rewritten; publish a new version to add intentional restrictions.

Tool permissions are not an OS sandbox. An approved command runs project code with the host user's authority and can have filesystem/network side effects. Do not grant shell access if stronger filesystem isolation is required.