Files
pipelines/docs/model-results.md
T
alan f57457aec9 Make agent-stage retries safe and retain timeout failure evidence
Fix four stage failure-handling and observability defects (A2, A3, A6, A9).

- A2: add safeToReplay(def) as the single definition of retryable post-admission
  work - only an explicit tools block that selects neither edit nor shell. The
  agent/pi WorkStarted guards now break only when the stage is not safe to
  replay, so read-only stages genuinely retry a post-admission timeout while
  mutating and legacy-bundle stages still never replay admitted work.
- A6: re-emit a malformed final result on an explicit read-only stage up to the
  retry budget, preserving format_repairs evidence; mutating stages keep the
  immediate return and their session.
- A3: timed-out stages retain usable evidence. waitForSessionIdle records one
  bounded sanitized OperationEvidence on deadline; the agent deferred cleanup
  does a bounded best-effort latest-assistant readback and settles knowable V2
  usage; RetainSession adds a retained_session pointer to failed step
  transcripts so the session is visible beneath the BoundTranscript threshold.
- A9: a stage outcome routed to #failure now yields a non-empty run diagnostic
  ("stage <name> returned outcome <outcome>") while preferring an existing step
  diagnostic.

Adds focused stage and server tests for retry safety, malformed-result
re-emission, timeout evidence retention and run-level diagnostics; updates the
packaged skill references and docs/cli.md, docs/model-results.md.
2026-10-15 18:27:22 +01:00

156 lines
8.8 KiB
Markdown

# Model results and scoped tools
Every model result must be one object containing exactly `outcome` and `output`.
`outcome` must be a declared edge; `output` must be an object. Duplicate keys
(including nested output keys), missing fields, unknown fields and invalid
outcomes are rejected. Free-form `output` contract prose is not a JSON Schema.
The native stable OpenCode adapter first checks the complete format POST/readback
contract in a separate retained, deny-all, no-inference session. Compatible native
backends request `format.type=json_schema` with this envelope and `retryCount=0`.
OpenCode 1.18.35 accepts a format but rejects its stored messages with
`Expected OutputFormatJsonSchema` (even explicit `text` fails). For that specific
incompatibility, requests omit `format`; strict local validation remains mandatory.
Other probe failures, such as authentication or network errors, are not silently
downgraded. Returned structured results remain preferred when available, and tool
transcripts are preserved. V2's published prompt/generate API does not accept a schema-output
field; that adapter retains local validation instead of sending unsupported
fields or claiming provider-side schema enforcement.
Output-contract prose describes the map **inside** `output`; it never overrides
the runtime `{outcome, output}` envelope. Printed DSML or other tool-call markup
is not evidence of a real tool invocation and cannot be accepted as a result.
Flattened output fields are rejected, not guessed into a new envelope. Invalid
responses are retained even when they are not eligible for syntax repair.
## Format recovery is not execution retry
Parser errors retain their diagnostic and byte offset: for example, `missing
closing brace '}'`, an unterminated string, or an invalid character. A valid but
schema-invalid result is not reformatted to invent a missing field or verdict.
Format recovery has a separate bounded budget, even with `retries=0`:
1. Append up to eight missing closing delimiters when the existing response
consists of complete JSON tokens. Validate the result again. No model call.
2. If necessary, make at most one fresh, deny-all, retained text-generation
request with only the malformed response and precise parsing error. The
original implementing/reviewing agent is never re-prompted. The formatter
has a 30-second deadline within the original stage deadline and a 32 KiB
input/output bound.
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
with at most 16 insertions. Every existing string, number, literal and
structural delimiter must remain verbatim and in order. Validate the full
result again. Changed approval, outcome, evidence, reordered data, deleted
fields, incomplete strings and fabricated values are rejected.
If recovery fails, report the original precise error and repair failure; never
rerun project edits, commands, or the entire stage to repair final formatting.
Configured execution retries remain for retryable text-generation transport
errors; an explicit read-only stage may additionally re-emit a malformed final
result and retry a post-admission deadline or transport failure up to `retries`.
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
read/glob/grep/edit/shell bundle), never has admitted tool work replayed.
Repair records include the original/candidate response, parsing diagnostic,
method and acceptance flag in the bounded transcript preview. `pipelines
inspect <run-id> --json` and the TUI evidence section expose these records.
OpenCode CLI failures include the HTTP status when supplied, process exit code,
and a recognized safe error category (such as session not found or service
unavailable). Arbitrary stderr, provider messages and response bodies are not
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
is not evidence that the agent prompt failed to start, and does not trigger replay
of stage work.
Native `StructuredOutputError` is named directly, rather than reduced to a generic
command failure. Native HTTP evidence records the actual HTTP operation/status;
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
Cleanup failures preserve the original generation error and active ownership.
For existing native sessions with format-bearing messages, the HTTP readback
workaround is read-only CLI `opencode export`. The exported session ID must match
the requested ID. Nothing is rewritten or removed from the database/history.
Debug receives the original messages plus a note identifying that recovery path.
## Failure diagnostics and confirmed limits
Retained stage failures distinguish two fixed, sanitized categories in the step
diagnostic:
- **FINAL RESULT VALIDATION** for a strictly invalid final result, with the stage
key and the precise parser detail (for example `duplicate JSON field "score"` or
`invalid character 'z' after object key:value pair at byte N`). It states that
completed edits/tool side effects are retained and not undone, and that a full
retry repeats work.
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
failures, including the sanitized `malformed or incomplete` readback message.
A structured-output validation failure is never reported as a harness transport
failure, and vice versa. Neither label includes a provider response body, prompt,
credential or arbitrary model content. Duplicate nested fields are rejected by
strict parsing rather than merged or last-wins, and only a lossless insertion of
missing commas/colons or trailing closers is repaired, so an unknown evidence or
verdict value can never become `success`, and admitted tool work on a mutating or
legacy-bundle stage is never replayed (only an explicit read-only stage may
re-emit a malformed result within `retries`).
Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned
exactly 262144 bytes of incomplete JSON (`invalid agent messages`); a reviewer
result was rejected for a duplicate nested `performance_review` field; and an
onboarding visual-review result failed with `invalid character 'z' after object
key:value pair at byte 3968`, where unsafe lossless repair was correctly rejected.
These are source-level findings covered by mocked `internal/stage` fixtures. An
updated checkout or installed skill is **not** evidence that a user's running
server contains the fix; deploying/restarting is a separate authorized action.
OpenCode V2's published prompt/generate API does not accept a structured-output
field, so a prose `output_contract` does **not** enforce structured output: V2
results are validated locally by strict parsing only, and no unsupported V2 format
field is sent. A review-only fresh execution that could re-derive a failed verdict
without replaying the admitted tool-using stage's edits is a future improvement and
is **not implemented**. Two items remain unresolved and must not be reported
complete: a macOS fixture path-spelling mismatch (`/var/folders` versus
`/private/var/folders` are the same real directory) in
`scripts/test_setup_harness.py`, and onboarding UI visual approval.
## Optional machine-readable restrictions
`tools` enables actions per stage. A stage can narrow edit paths and shell
commands with an optional `permissions` block:
````markdown
```tools
read
glob
grep
edit
shell
```
```permissions
{"edit":["README.md","plugin/*.qml"],"shell":["bash -n install.sh"]}
```
````
These rules become OpenCode tool permissions, not just prompt advice. Edit
patterns are relative to the run directory and intersect any optional run-level
`--write-paths`. Shell entries are exact commands: no wildcards, chaining,
redirection or expansion. Empty lists deny an action; an omitted list retains
the normal defaults. Null lists, unknown keys, duplicate keys/resources, path
traversal, undeclared tools and unrestricted shell alongside restricted edits
are rejected. At most 20 resources per action are accepted.
OpenCode checks scanner-produced shell command resources, not necessarily the
entire submitted script. A compound script consisting only of individually
allowed commands may pass; this is not a raw-script or shell-syntax firewall.
For a deliberately Go-only stage, explicitly declare `edit:["*.go"]` and the
exact allowed verification commands. Without a machine policy, a sentence such
as “approved Go files only” is **not** an enforced permission boundary. Defaults
remain language-neutral: this does not restore a global Go-only restriction or
require users to enumerate files for ordinary runs. Existing immutable definitions
are not rewritten; publish a new version to add intentional restrictions.
Tool permissions are not an OS sandbox. An approved command runs project code
with the host user's authority and can have filesystem/network side effects.
Do not grant shell access if stronger filesystem isolation is required.