226 lines
13 KiB
Markdown
226 lines
13 KiB
Markdown
# Model results and scoped tools
|
|
|
|
Every model result must be one object containing exactly `outcome` and `output`.
|
|
`outcome` must be a declared edge; `output` must be an object. Duplicate keys
|
|
(including nested output keys), missing fields, unknown fields and invalid
|
|
outcomes are rejected. Free-form `output` contract prose is not a JSON Schema.
|
|
|
|
The native stable OpenCode adapter first checks the complete format POST/readback
|
|
contract in a separate retained, deny-all, no-inference session. Compatible native
|
|
backends request `format.type=json_schema` with this envelope and `retryCount=0`.
|
|
OpenCode 1.18.35 accepts a format but rejects its stored messages with
|
|
`Expected OutputFormatJsonSchema` (even explicit `text` fails). For that specific
|
|
incompatibility, requests omit `format`; strict local validation remains mandatory.
|
|
Other probe failures, such as authentication or network errors, are not silently
|
|
downgraded. Returned structured results remain preferred when available, and tool
|
|
transcripts are preserved. V2's published prompt/generate API does not accept a schema-output
|
|
field; that adapter retains local validation instead of sending unsupported
|
|
fields or claiming provider-side schema enforcement.
|
|
|
|
Output-contract prose describes the map **inside** `output`; it never overrides
|
|
the runtime `{outcome, output}` envelope. Printed DSML or other tool-call markup
|
|
is not evidence of a real tool invocation and cannot be accepted as a result.
|
|
Flattened output fields are rejected, not guessed into a new envelope. Invalid
|
|
responses are retained even when they are not eligible for syntax repair.
|
|
|
|
## Line-oriented result format
|
|
|
|
Model and agent stages also accept a line-oriented result format. The response is
|
|
a markdown-ish document whose first non-empty line is the outcome, followed by
|
|
sections introduced with `# <heading>`.
|
|
|
|
```
|
|
success
|
|
|
|
# output
|
|
{"task": "x"}
|
|
|
|
# findings
|
|
- prose, code fences, quotes and embedded JSON need no escaping
|
|
```
|
|
|
|
Only the first non-empty line and a `# output` section are required. The body of
|
|
`# output` must be a JSON object; it becomes the stage's `output` map. Every other
|
|
heading becomes an evidence field keyed by the normalised heading text (lowercased,
|
|
spaces to underscores). Extra sections are stored verbatim and are best-effort: an
|
|
empty heading, an odd heading, or a very large section must never fail the stage.
|
|
If the outcome is missing, not one of the declared edges, or `# output` is absent,
|
|
the stage fails with `FINAL RESULT VALIDATION` and still retains whatever sections
|
|
were salvaged.
|
|
|
|
The parser decides between the legacy JSON envelope and the line-oriented format by
|
|
precedence. If the response's first non-empty character is `{` after stripping an
|
|
optional surrounding markdown fence, it keeps the unchanged JSON path. If a `# `
|
|
heading appears before any `{`, it uses the line-oriented path. This preserves all
|
|
existing JSON responses, including fenced JSON or short prose prefixes, while
|
|
allowing new line-format responses to coexist.
|
|
|
|
The full raw response is written to the stage's artifacts directory as
|
|
`raw-output.md` before parsing, so the closest record of what the model produced is
|
|
always retained and appears in the step's artifact inventory automatically. The
|
|
parse status is recorded in the step's output map under the reserved key
|
|
`result_parse`: `parsed` (both required parts OK), `partial` (required parts OK but
|
|
some extra section unusable), or `failed` (a required part missing or invalid).
|
|
|
|
## Format recovery is not execution retry
|
|
|
|
Parser errors retain their diagnostic and byte offset: for example, `missing
|
|
closing brace '}'`, an unterminated string, or an invalid character. A valid but
|
|
schema-invalid result is not reformatted to invent a missing field or verdict.
|
|
|
|
Format recovery has a separate bounded budget, even with `retries=0`:
|
|
|
|
1. A deterministic local pass fixes only the structural error that is derivable
|
|
from the payload itself, in order: append up to eight missing closing
|
|
delimiters when the existing response consists of complete JSON tokens;
|
|
remove a single unambiguous trailing comma before a closing `}`/`]`; or
|
|
insert exactly the one comma/colon the parser expected at its reported byte.
|
|
Each candidate must pass lossless syntax validation and the full envelope
|
|
validation. No model call. An ambiguous payload - an unescaped quote or colon
|
|
inside a string value, or a stray delimiter with no single completion - is
|
|
never guessed at and falls through.
|
|
2. If the local pass cannot repair it, make at most one fresh, deny-all, retained
|
|
text-generation request with only the malformed response and precise parsing
|
|
error. The original implementing/reviewing agent is never re-prompted. The
|
|
formatter's deadline is derived from the payload size and the attempt's own
|
|
remaining budget rather than a fixed short constant, so a large result is not
|
|
guaranteed to time out; it stays within the stage deadline, is floored for
|
|
ordinary payloads, and keeps the 32 KiB input/output bound.
|
|
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
|
|
with at most 16 insertions. Every existing string, number, literal and
|
|
structural delimiter must remain verbatim and in order. Validate the full
|
|
result again. Changed approval, outcome, evidence, reordered data, deleted
|
|
fields, incomplete strings and fabricated values are rejected.
|
|
|
|
Every terminal path records an explicit recovery status: `repaired` (a fully
|
|
lossless local or formatter recovery), `partial` (the candidate is valid JSON but
|
|
still fails envelope validation, so it is never returned as a completed result),
|
|
or `unrecoverable` (nothing recovered, or the formatter errored or timed out).
|
|
The unrecoverable path retains the bounded original payload together with the
|
|
precise parse error and its byte offset, and the status is included in the
|
|
`FINAL RESULT VALIDATION` diagnostic so an operator can see exactly what the model
|
|
emitted.
|
|
|
|
If recovery fails, report the original precise error and repair failure; never
|
|
rerun project edits, commands, or the entire stage to repair final formatting.
|
|
Configured execution retries remain for retryable text-generation transport
|
|
errors; an explicit read-only stage may additionally re-emit a malformed final
|
|
result and retry a post-admission deadline or transport failure up to `retries`.
|
|
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
|
|
read/glob/grep/edit/shell bundle), never has admitted tool work replayed. The
|
|
deterministic local pass and the fresh deny-all, tool-free formatter session never
|
|
re-prompt or replay the admitted stage session, so a shell-enabled review stage
|
|
gains malformed-result recovery without any replay of admitted work.
|
|
Repair records include the original/candidate response, parsing diagnostic,
|
|
method, recovery status, byte offset and acceptance flag in the bounded transcript
|
|
preview. `pipelines inspect <run-id> --json` and the TUI evidence section expose
|
|
these records.
|
|
|
|
OpenCode CLI failures include the HTTP status when supplied, process exit code,
|
|
and a recognized safe error category (such as session not found or service
|
|
unavailable). Arbitrary stderr, provider messages and response bodies are not
|
|
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
|
|
is not evidence that the agent prompt failed to start, and does not trigger replay
|
|
of stage work.
|
|
Native `StructuredOutputError` is named directly, rather than reduced to a generic
|
|
command failure. Native HTTP evidence records the actual HTTP operation/status;
|
|
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
|
|
Cleanup failures preserve the original generation error and active ownership.
|
|
|
|
For existing native sessions with format-bearing messages, the HTTP readback
|
|
workaround is read-only CLI `opencode export`. The exported session ID must match
|
|
the requested ID. Nothing is rewritten or removed from the database/history.
|
|
Debug receives the original messages plus a note identifying that recovery path.
|
|
|
|
## Failure diagnostics and confirmed limits
|
|
|
|
Retained stage failures distinguish two fixed, sanitized categories in the step
|
|
diagnostic:
|
|
|
|
- **FINAL RESULT VALIDATION** for a strictly invalid final result, with the stage
|
|
key and the precise parser detail (for example `duplicate JSON field "score"` or
|
|
`invalid character 'z' after object key:value pair at byte N`). It states that
|
|
completed edits/tool side effects are retained and not undone, and that a full
|
|
retry repeats work.
|
|
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
|
|
failures, including the sanitized `malformed or incomplete` readback message.
|
|
|
|
A structured-output validation failure is never reported as a harness transport
|
|
failure, and vice versa. Neither label includes a provider response body, prompt,
|
|
credential or arbitrary model content. Duplicate nested fields are rejected by
|
|
strict parsing rather than merged or last-wins, and only a lossless insertion of
|
|
missing commas/colons or trailing closers is repaired, so an unknown evidence or
|
|
verdict value can never become `success`, and admitted tool work on a mutating or
|
|
legacy-bundle stage is never replayed (only an explicit read-only stage may
|
|
re-emit a malformed result within `retries`).
|
|
|
|
Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned
|
|
exactly 262144 bytes of incomplete JSON (`invalid agent messages`); a reviewer
|
|
result was rejected for a duplicate nested `performance_review` field; and an
|
|
onboarding visual-review result failed with `invalid character 'z' after object
|
|
key:value pair at byte 3968`, where unsafe lossless repair was correctly rejected.
|
|
|
|
A shell-enabled `ui_reviewer` stage (run 55, pipeline 9 v6, 2026-10-15) failed with
|
|
`invalid character ':' after object key:value pair at byte 7837`, where the
|
|
isolated formatter hit its own 30-second deadline. The malformed payload was
|
|
retrieved read-only with `pipelines inspect 55 --json` and reproduced the exact
|
|
byte offset 7837. It is an unescaped quote/colon inside a string value - an
|
|
ambiguous case - so the deterministic pass correctly reports it unrecoverable
|
|
rather than guessing, while the new derived deadline removes the fixed-deadline
|
|
failure mode for larger payloads. These are source-level findings covered by mocked
|
|
`internal/stage` fixtures. An updated checkout or installed skill is **not**
|
|
evidence that a user's running server contains the fix; deploying/restarting is a
|
|
separate authorized action.
|
|
|
|
OpenCode V2's published prompt/generate API does not accept a structured-output
|
|
field, so a prose `output_contract` does **not** enforce structured output: V2
|
|
results are validated locally by strict parsing only, and no unsupported V2 format
|
|
field is sent. A review-only fresh execution that could re-derive a failed verdict
|
|
without replaying the admitted tool-using stage's edits is a future improvement and
|
|
is **not implemented**. Two items remain unresolved and must not be reported
|
|
complete: a macOS fixture path-spelling mismatch (`/var/folders` versus
|
|
`/private/var/folders` are the same real directory) in
|
|
`scripts/test_setup_harness.py`, and onboarding UI visual approval.
|
|
|
|
## Optional machine-readable restrictions
|
|
|
|
`tools` enables actions per stage. A stage can narrow edit paths and shell
|
|
commands with an optional `permissions` block:
|
|
|
|
````markdown
|
|
```tools
|
|
read
|
|
glob
|
|
grep
|
|
edit
|
|
shell
|
|
```
|
|
|
|
```permissions
|
|
{"edit":["README.md","plugin/*.qml"],"shell":["bash -n install.sh"]}
|
|
```
|
|
````
|
|
|
|
These rules become OpenCode tool permissions, not just prompt advice. Edit
|
|
patterns are relative to the run directory and intersect any optional run-level
|
|
`--write-paths`. Shell entries are exact commands: no wildcards, chaining,
|
|
redirection or expansion. Empty lists deny an action; an omitted list retains
|
|
the normal defaults. Null lists, unknown keys, duplicate keys/resources, path
|
|
traversal, undeclared tools and unrestricted shell alongside restricted edits
|
|
are rejected. At most 20 resources per action are accepted.
|
|
|
|
OpenCode checks scanner-produced shell command resources, not necessarily the
|
|
entire submitted script. A compound script consisting only of individually
|
|
allowed commands may pass; this is not a raw-script or shell-syntax firewall.
|
|
|
|
For a deliberately Go-only stage, explicitly declare `edit:["*.go"]` and the
|
|
exact allowed verification commands. Without a machine policy, a sentence such
|
|
as “approved Go files only” is **not** an enforced permission boundary. Defaults
|
|
remain language-neutral: this does not restore a global Go-only restriction or
|
|
require users to enumerate files for ordinary runs. Existing immutable definitions
|
|
are not rewritten; publish a new version to add intentional restrictions.
|
|
|
|
Tool permissions are not an OS sandbox. An approved command runs project code
|
|
with the host user's authority and can have filesystem/network side effects.
|
|
Do not grant shell access if stronger filesystem isolation is required.
|