f57457aec9
Fix four stage failure-handling and observability defects (A2, A3, A6, A9).
- A2: add safeToReplay(def) as the single definition of retryable post-admission
work - only an explicit tools block that selects neither edit nor shell. The
agent/pi WorkStarted guards now break only when the stage is not safe to
replay, so read-only stages genuinely retry a post-admission timeout while
mutating and legacy-bundle stages still never replay admitted work.
- A6: re-emit a malformed final result on an explicit read-only stage up to the
retry budget, preserving format_repairs evidence; mutating stages keep the
immediate return and their session.
- A3: timed-out stages retain usable evidence. waitForSessionIdle records one
bounded sanitized OperationEvidence on deadline; the agent deferred cleanup
does a bounded best-effort latest-assistant readback and settles knowable V2
usage; RetainSession adds a retained_session pointer to failed step
transcripts so the session is visible beneath the BoundTranscript threshold.
- A9: a stage outcome routed to #failure now yields a non-empty run diagnostic
("stage <name> returned outcome <outcome>") while preferring an existing step
diagnostic.
Adds focused stage and server tests for retry safety, malformed-result
re-emission, timeout evidence retention and run-level diagnostics; updates the
packaged skill references and docs/cli.md, docs/model-results.md.
156 lines
8.8 KiB
Markdown
156 lines
8.8 KiB
Markdown
# Model results and scoped tools
|
|
|
|
Every model result must be one object containing exactly `outcome` and `output`.
|
|
`outcome` must be a declared edge; `output` must be an object. Duplicate keys
|
|
(including nested output keys), missing fields, unknown fields and invalid
|
|
outcomes are rejected. Free-form `output` contract prose is not a JSON Schema.
|
|
|
|
The native stable OpenCode adapter first checks the complete format POST/readback
|
|
contract in a separate retained, deny-all, no-inference session. Compatible native
|
|
backends request `format.type=json_schema` with this envelope and `retryCount=0`.
|
|
OpenCode 1.18.35 accepts a format but rejects its stored messages with
|
|
`Expected OutputFormatJsonSchema` (even explicit `text` fails). For that specific
|
|
incompatibility, requests omit `format`; strict local validation remains mandatory.
|
|
Other probe failures, such as authentication or network errors, are not silently
|
|
downgraded. Returned structured results remain preferred when available, and tool
|
|
transcripts are preserved. V2's published prompt/generate API does not accept a schema-output
|
|
field; that adapter retains local validation instead of sending unsupported
|
|
fields or claiming provider-side schema enforcement.
|
|
|
|
Output-contract prose describes the map **inside** `output`; it never overrides
|
|
the runtime `{outcome, output}` envelope. Printed DSML or other tool-call markup
|
|
is not evidence of a real tool invocation and cannot be accepted as a result.
|
|
Flattened output fields are rejected, not guessed into a new envelope. Invalid
|
|
responses are retained even when they are not eligible for syntax repair.
|
|
|
|
## Format recovery is not execution retry
|
|
|
|
Parser errors retain their diagnostic and byte offset: for example, `missing
|
|
closing brace '}'`, an unterminated string, or an invalid character. A valid but
|
|
schema-invalid result is not reformatted to invent a missing field or verdict.
|
|
|
|
Format recovery has a separate bounded budget, even with `retries=0`:
|
|
|
|
1. Append up to eight missing closing delimiters when the existing response
|
|
consists of complete JSON tokens. Validate the result again. No model call.
|
|
2. If necessary, make at most one fresh, deny-all, retained text-generation
|
|
request with only the malformed response and precise parsing error. The
|
|
original implementing/reviewing agent is never re-prompted. The formatter
|
|
has a 30-second deadline within the original stage deadline and a 32 KiB
|
|
input/output bound.
|
|
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
|
|
with at most 16 insertions. Every existing string, number, literal and
|
|
structural delimiter must remain verbatim and in order. Validate the full
|
|
result again. Changed approval, outcome, evidence, reordered data, deleted
|
|
fields, incomplete strings and fabricated values are rejected.
|
|
|
|
If recovery fails, report the original precise error and repair failure; never
|
|
rerun project edits, commands, or the entire stage to repair final formatting.
|
|
Configured execution retries remain for retryable text-generation transport
|
|
errors; an explicit read-only stage may additionally re-emit a malformed final
|
|
result and retry a post-admission deadline or transport failure up to `retries`.
|
|
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
|
|
read/glob/grep/edit/shell bundle), never has admitted tool work replayed.
|
|
Repair records include the original/candidate response, parsing diagnostic,
|
|
method and acceptance flag in the bounded transcript preview. `pipelines
|
|
inspect <run-id> --json` and the TUI evidence section expose these records.
|
|
|
|
OpenCode CLI failures include the HTTP status when supplied, process exit code,
|
|
and a recognized safe error category (such as session not found or service
|
|
unavailable). Arbitrary stderr, provider messages and response bodies are not
|
|
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
|
|
is not evidence that the agent prompt failed to start, and does not trigger replay
|
|
of stage work.
|
|
Native `StructuredOutputError` is named directly, rather than reduced to a generic
|
|
command failure. Native HTTP evidence records the actual HTTP operation/status;
|
|
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
|
|
Cleanup failures preserve the original generation error and active ownership.
|
|
|
|
For existing native sessions with format-bearing messages, the HTTP readback
|
|
workaround is read-only CLI `opencode export`. The exported session ID must match
|
|
the requested ID. Nothing is rewritten or removed from the database/history.
|
|
Debug receives the original messages plus a note identifying that recovery path.
|
|
|
|
## Failure diagnostics and confirmed limits
|
|
|
|
Retained stage failures distinguish two fixed, sanitized categories in the step
|
|
diagnostic:
|
|
|
|
- **FINAL RESULT VALIDATION** for a strictly invalid final result, with the stage
|
|
key and the precise parser detail (for example `duplicate JSON field "score"` or
|
|
`invalid character 'z' after object key:value pair at byte N`). It states that
|
|
completed edits/tool side effects are retained and not undone, and that a full
|
|
retry repeats work.
|
|
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
|
|
failures, including the sanitized `malformed or incomplete` readback message.
|
|
|
|
A structured-output validation failure is never reported as a harness transport
|
|
failure, and vice versa. Neither label includes a provider response body, prompt,
|
|
credential or arbitrary model content. Duplicate nested fields are rejected by
|
|
strict parsing rather than merged or last-wins, and only a lossless insertion of
|
|
missing commas/colons or trailing closers is repaired, so an unknown evidence or
|
|
verdict value can never become `success`, and admitted tool work on a mutating or
|
|
legacy-bundle stage is never replayed (only an explicit read-only stage may
|
|
re-emit a malformed result within `retries`).
|
|
|
|
Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned
|
|
exactly 262144 bytes of incomplete JSON (`invalid agent messages`); a reviewer
|
|
result was rejected for a duplicate nested `performance_review` field; and an
|
|
onboarding visual-review result failed with `invalid character 'z' after object
|
|
key:value pair at byte 3968`, where unsafe lossless repair was correctly rejected.
|
|
These are source-level findings covered by mocked `internal/stage` fixtures. An
|
|
updated checkout or installed skill is **not** evidence that a user's running
|
|
server contains the fix; deploying/restarting is a separate authorized action.
|
|
|
|
OpenCode V2's published prompt/generate API does not accept a structured-output
|
|
field, so a prose `output_contract` does **not** enforce structured output: V2
|
|
results are validated locally by strict parsing only, and no unsupported V2 format
|
|
field is sent. A review-only fresh execution that could re-derive a failed verdict
|
|
without replaying the admitted tool-using stage's edits is a future improvement and
|
|
is **not implemented**. Two items remain unresolved and must not be reported
|
|
complete: a macOS fixture path-spelling mismatch (`/var/folders` versus
|
|
`/private/var/folders` are the same real directory) in
|
|
`scripts/test_setup_harness.py`, and onboarding UI visual approval.
|
|
|
|
## Optional machine-readable restrictions
|
|
|
|
`tools` enables actions per stage. A stage can narrow edit paths and shell
|
|
commands with an optional `permissions` block:
|
|
|
|
````markdown
|
|
```tools
|
|
read
|
|
glob
|
|
grep
|
|
edit
|
|
shell
|
|
```
|
|
|
|
```permissions
|
|
{"edit":["README.md","plugin/*.qml"],"shell":["bash -n install.sh"]}
|
|
```
|
|
````
|
|
|
|
These rules become OpenCode tool permissions, not just prompt advice. Edit
|
|
patterns are relative to the run directory and intersect any optional run-level
|
|
`--write-paths`. Shell entries are exact commands: no wildcards, chaining,
|
|
redirection or expansion. Empty lists deny an action; an omitted list retains
|
|
the normal defaults. Null lists, unknown keys, duplicate keys/resources, path
|
|
traversal, undeclared tools and unrestricted shell alongside restricted edits
|
|
are rejected. At most 20 resources per action are accepted.
|
|
|
|
OpenCode checks scanner-produced shell command resources, not necessarily the
|
|
entire submitted script. A compound script consisting only of individually
|
|
allowed commands may pass; this is not a raw-script or shell-syntax firewall.
|
|
|
|
For a deliberately Go-only stage, explicitly declare `edit:["*.go"]` and the
|
|
exact allowed verification commands. Without a machine policy, a sentence such
|
|
as “approved Go files only” is **not** an enforced permission boundary. Defaults
|
|
remain language-neutral: this does not restore a global Go-only restriction or
|
|
require users to enumerate files for ordinary runs. Existing immutable definitions
|
|
are not rewritten; publish a new version to add intentional restrictions.
|
|
|
|
Tool permissions are not an OS sandbox. An approved command runs project code
|
|
with the host user's authority and can have filesystem/network side effects.
|
|
Do not grant shell access if stronger filesystem isolation is required.
|