9f88e4ff71
A model stage whose final result was malformed failed the whole run even when
the work and evidence had survived, because the only recovery was an isolated
formatter call on a fixed 30-second deadline that could time out on its own
(run 55, ui_reviewer of pipeline 9 v6).
Add an ordered, deterministic local repair pass (internal/stage/result_repair.go)
that runs before any model call: (a) append missing closing delimiters via the
existing closeJSONContainers; (b) remove one unambiguous trailing comma before a
closing } or ]; (c) insert exactly the single comma/colon the JSON parser reported
at its byte offset. Every candidate must pass losslessSyntaxRepair(original,
candidate) and decodeModelResult before it is accepted, so no field value is
invented, removed, reordered or rewritten. Ambiguous payloads (an unescaped quote
or colon inside a string value, or a stray delimiter with no single completion)
return not-ok and fall through to the bounded model re-emit rather than guessing.
stageResultText still strips surrounding prose and a single code fence first;
model_prompt.go now also strips prose around a single balanced-but-malformed
object span so the deterministic pass sees just the object.
Replace the fixed 30-second formatter deadline with repairDeadline(payloadBytes,
remaining), which grows monotonically with payload size, has a floor, and is
clamped to the attempt's remaining budget minus a safety margin, so the stage
context remains the outer bound and a large result is not guaranteed to time out.
The one-request-per-attempt, 32 KiB bound and the deny-all, retained, new-session
property of RepairFormat are unchanged for OpenCodeClient and PiClient.
Record an explicit recovery status on every terminal path: repaired, partial or
unrecoverable, plus a byte offset. A candidate that is valid JSON but still fails
envelope validation is marked partial and never returned as a completed result;
the unrecoverable path retains the bounded original payload, the precise parse
error (including its byte offset) and surfaces the status in the FINAL RESULT
VALIDATION diagnostic.
safeToReplay is intentionally unchanged. The deterministic local repair and the
fresh deny-all, no-tools, retained formatter session are independent of the
stage's tool set and never re-prompt or replay the admitted agent session, so a
shell-enabled review stage gains malformed-result recovery without any replay of
admitted work. Risk: no change to mutation safety; the only added cost is one
bounded formatter call, already budgeted. Replayability is not extended for shell
stages.
Docs and packaged skill references (docs/model-results.md,
internal/author/skill/references/{format,debugging,verification}.md) describe the
three-tier recovery, the status words and the derived deadline; the safeToReplay
decision and its risk are recorded there alongside the commit.
Tests add mocked, offline coverage for the deterministic and partial paths, a
repairDeadline monotonicity/floor/clamp table, and a shell-stage regression
proving a recoverable malformed result does not fail the run or replay tool work.
187 lines
11 KiB
Markdown
187 lines
11 KiB
Markdown
# Model results and scoped tools
|
|
|
|
Every model result must be one object containing exactly `outcome` and `output`.
|
|
`outcome` must be a declared edge; `output` must be an object. Duplicate keys
|
|
(including nested output keys), missing fields, unknown fields and invalid
|
|
outcomes are rejected. Free-form `output` contract prose is not a JSON Schema.
|
|
|
|
The native stable OpenCode adapter first checks the complete format POST/readback
|
|
contract in a separate retained, deny-all, no-inference session. Compatible native
|
|
backends request `format.type=json_schema` with this envelope and `retryCount=0`.
|
|
OpenCode 1.18.35 accepts a format but rejects its stored messages with
|
|
`Expected OutputFormatJsonSchema` (even explicit `text` fails). For that specific
|
|
incompatibility, requests omit `format`; strict local validation remains mandatory.
|
|
Other probe failures, such as authentication or network errors, are not silently
|
|
downgraded. Returned structured results remain preferred when available, and tool
|
|
transcripts are preserved. V2's published prompt/generate API does not accept a schema-output
|
|
field; that adapter retains local validation instead of sending unsupported
|
|
fields or claiming provider-side schema enforcement.
|
|
|
|
Output-contract prose describes the map **inside** `output`; it never overrides
|
|
the runtime `{outcome, output}` envelope. Printed DSML or other tool-call markup
|
|
is not evidence of a real tool invocation and cannot be accepted as a result.
|
|
Flattened output fields are rejected, not guessed into a new envelope. Invalid
|
|
responses are retained even when they are not eligible for syntax repair.
|
|
|
|
## Format recovery is not execution retry
|
|
|
|
Parser errors retain their diagnostic and byte offset: for example, `missing
|
|
closing brace '}'`, an unterminated string, or an invalid character. A valid but
|
|
schema-invalid result is not reformatted to invent a missing field or verdict.
|
|
|
|
Format recovery has a separate bounded budget, even with `retries=0`:
|
|
|
|
1. A deterministic local pass fixes only the structural error that is derivable
|
|
from the payload itself, in order: append up to eight missing closing
|
|
delimiters when the existing response consists of complete JSON tokens;
|
|
remove a single unambiguous trailing comma before a closing `}`/`]`; or
|
|
insert exactly the one comma/colon the parser expected at its reported byte.
|
|
Each candidate must pass lossless syntax validation and the full envelope
|
|
validation. No model call. An ambiguous payload - an unescaped quote or colon
|
|
inside a string value, or a stray delimiter with no single completion - is
|
|
never guessed at and falls through.
|
|
2. If the local pass cannot repair it, make at most one fresh, deny-all, retained
|
|
text-generation request with only the malformed response and precise parsing
|
|
error. The original implementing/reviewing agent is never re-prompted. The
|
|
formatter's deadline is derived from the payload size and the attempt's own
|
|
remaining budget rather than a fixed short constant, so a large result is not
|
|
guaranteed to time out; it stays within the stage deadline, is floored for
|
|
ordinary payloads, and keeps the 32 KiB input/output bound.
|
|
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
|
|
with at most 16 insertions. Every existing string, number, literal and
|
|
structural delimiter must remain verbatim and in order. Validate the full
|
|
result again. Changed approval, outcome, evidence, reordered data, deleted
|
|
fields, incomplete strings and fabricated values are rejected.
|
|
|
|
Every terminal path records an explicit recovery status: `repaired` (a fully
|
|
lossless local or formatter recovery), `partial` (the candidate is valid JSON but
|
|
still fails envelope validation, so it is never returned as a completed result),
|
|
or `unrecoverable` (nothing recovered, or the formatter errored or timed out).
|
|
The unrecoverable path retains the bounded original payload together with the
|
|
precise parse error and its byte offset, and the status is included in the
|
|
`FINAL RESULT VALIDATION` diagnostic so an operator can see exactly what the model
|
|
emitted.
|
|
|
|
If recovery fails, report the original precise error and repair failure; never
|
|
rerun project edits, commands, or the entire stage to repair final formatting.
|
|
Configured execution retries remain for retryable text-generation transport
|
|
errors; an explicit read-only stage may additionally re-emit a malformed final
|
|
result and retry a post-admission deadline or transport failure up to `retries`.
|
|
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
|
|
read/glob/grep/edit/shell bundle), never has admitted tool work replayed. The
|
|
deterministic local pass and the fresh deny-all, tool-free formatter session never
|
|
re-prompt or replay the admitted stage session, so a shell-enabled review stage
|
|
gains malformed-result recovery without any replay of admitted work.
|
|
Repair records include the original/candidate response, parsing diagnostic,
|
|
method, recovery status, byte offset and acceptance flag in the bounded transcript
|
|
preview. `pipelines inspect <run-id> --json` and the TUI evidence section expose
|
|
these records.
|
|
|
|
OpenCode CLI failures include the HTTP status when supplied, process exit code,
|
|
and a recognized safe error category (such as session not found or service
|
|
unavailable). Arbitrary stderr, provider messages and response bodies are not
|
|
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
|
|
is not evidence that the agent prompt failed to start, and does not trigger replay
|
|
of stage work.
|
|
Native `StructuredOutputError` is named directly, rather than reduced to a generic
|
|
command failure. Native HTTP evidence records the actual HTTP operation/status;
|
|
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
|
|
Cleanup failures preserve the original generation error and active ownership.
|
|
|
|
For existing native sessions with format-bearing messages, the HTTP readback
|
|
workaround is read-only CLI `opencode export`. The exported session ID must match
|
|
the requested ID. Nothing is rewritten or removed from the database/history.
|
|
Debug receives the original messages plus a note identifying that recovery path.
|
|
|
|
## Failure diagnostics and confirmed limits
|
|
|
|
Retained stage failures distinguish two fixed, sanitized categories in the step
|
|
diagnostic:
|
|
|
|
- **FINAL RESULT VALIDATION** for a strictly invalid final result, with the stage
|
|
key and the precise parser detail (for example `duplicate JSON field "score"` or
|
|
`invalid character 'z' after object key:value pair at byte N`). It states that
|
|
completed edits/tool side effects are retained and not undone, and that a full
|
|
retry repeats work.
|
|
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
|
|
failures, including the sanitized `malformed or incomplete` readback message.
|
|
|
|
A structured-output validation failure is never reported as a harness transport
|
|
failure, and vice versa. Neither label includes a provider response body, prompt,
|
|
credential or arbitrary model content. Duplicate nested fields are rejected by
|
|
strict parsing rather than merged or last-wins, and only a lossless insertion of
|
|
missing commas/colons or trailing closers is repaired, so an unknown evidence or
|
|
verdict value can never become `success`, and admitted tool work on a mutating or
|
|
legacy-bundle stage is never replayed (only an explicit read-only stage may
|
|
re-emit a malformed result within `retries`).
|
|
|
|
Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned
|
|
exactly 262144 bytes of incomplete JSON (`invalid agent messages`); a reviewer
|
|
result was rejected for a duplicate nested `performance_review` field; and an
|
|
onboarding visual-review result failed with `invalid character 'z' after object
|
|
key:value pair at byte 3968`, where unsafe lossless repair was correctly rejected.
|
|
|
|
A shell-enabled `ui_reviewer` stage (run 55, pipeline 9 v6, 2026-10-15) failed with
|
|
`invalid character ':' after object key:value pair at byte 7837`, where the
|
|
isolated formatter hit its own 30-second deadline. The malformed payload was
|
|
retrieved read-only with `pipelines inspect 55 --json` and reproduced the exact
|
|
byte offset 7837. It is an unescaped quote/colon inside a string value - an
|
|
ambiguous case - so the deterministic pass correctly reports it unrecoverable
|
|
rather than guessing, while the new derived deadline removes the fixed-deadline
|
|
failure mode for larger payloads. These are source-level findings covered by mocked
|
|
`internal/stage` fixtures. An updated checkout or installed skill is **not**
|
|
evidence that a user's running server contains the fix; deploying/restarting is a
|
|
separate authorized action.
|
|
|
|
OpenCode V2's published prompt/generate API does not accept a structured-output
|
|
field, so a prose `output_contract` does **not** enforce structured output: V2
|
|
results are validated locally by strict parsing only, and no unsupported V2 format
|
|
field is sent. A review-only fresh execution that could re-derive a failed verdict
|
|
without replaying the admitted tool-using stage's edits is a future improvement and
|
|
is **not implemented**. Two items remain unresolved and must not be reported
|
|
complete: a macOS fixture path-spelling mismatch (`/var/folders` versus
|
|
`/private/var/folders` are the same real directory) in
|
|
`scripts/test_setup_harness.py`, and onboarding UI visual approval.
|
|
|
|
## Optional machine-readable restrictions
|
|
|
|
`tools` enables actions per stage. A stage can narrow edit paths and shell
|
|
commands with an optional `permissions` block:
|
|
|
|
````markdown
|
|
```tools
|
|
read
|
|
glob
|
|
grep
|
|
edit
|
|
shell
|
|
```
|
|
|
|
```permissions
|
|
{"edit":["README.md","plugin/*.qml"],"shell":["bash -n install.sh"]}
|
|
```
|
|
````
|
|
|
|
These rules become OpenCode tool permissions, not just prompt advice. Edit
|
|
patterns are relative to the run directory and intersect any optional run-level
|
|
`--write-paths`. Shell entries are exact commands: no wildcards, chaining,
|
|
redirection or expansion. Empty lists deny an action; an omitted list retains
|
|
the normal defaults. Null lists, unknown keys, duplicate keys/resources, path
|
|
traversal, undeclared tools and unrestricted shell alongside restricted edits
|
|
are rejected. At most 20 resources per action are accepted.
|
|
|
|
OpenCode checks scanner-produced shell command resources, not necessarily the
|
|
entire submitted script. A compound script consisting only of individually
|
|
allowed commands may pass; this is not a raw-script or shell-syntax firewall.
|
|
|
|
For a deliberately Go-only stage, explicitly declare `edit:["*.go"]` and the
|
|
exact allowed verification commands. Without a machine policy, a sentence such
|
|
as “approved Go files only” is **not** an enforced permission boundary. Defaults
|
|
remain language-neutral: this does not restore a global Go-only restriction or
|
|
require users to enumerate files for ordinary runs. Existing immutable definitions
|
|
are not rewritten; publish a new version to add intentional restrictions.
|
|
|
|
Tool permissions are not an OS sandbox. An approved command runs project code
|
|
with the host user's authority and can have filesystem/network side effects.
|
|
Do not grant shell access if stronger filesystem isolation is required.
|