Files
pipelines/docs/model-results.md
T
alan 88dfc051e5 Model stages: outcome-first output shape and one in-session outcome repair
The prompt now says output.md must start with # outcome and the outcome
word alone on the next line, then # handoff, then detail. Runs 241/243
followed the old "start with # handoff" wording and dropped the outcome.

When a completed turn lacks a valid outcome, the OpenCode/agent and Pi
clients send one short follow-up in the same session asking only for the
outcome line. An outcome-only reply goes at the top of the original report;
a still-invalid reply fails with the same validation error as before. The
follow-up's tokens accrue to the session usage. No outcome is inferred.

Missing tools/evidence now name blocked when declared, else error. The
parser tolerates a trailing period on the outcome word.
2026-11-04 17:20:58 +00:00

169 lines
7.6 KiB
Markdown

# Model turns, results and scoped tools
Model stages are file-based. Each visit is given its capped essential
`memory.md` inline, may read `input.md` and `history.md` on demand, and returns the complete markdown for `output.md`. Only the outcome section is
interpreted; the rest is free-form text for the next stage and the operator.
## Turn files
Every model visit gets its own directory under the run's artifact root:
```
<artifacts>/<run>/stages/01-implementer/input.md
<artifacts>/<run>/stages/01-implementer/memory.md
<artifacts>/<run>/stages/01-implementer/history.md
<artifacts>/<run>/stages/01-implementer/output.md
<artifacts>/<run>/stages/01-implementer/private/
<artifacts>/<run>/public/
```
- `input.md` is the run prompt on the first visit. Later visits keep the run
prompt under a `# Run task` heading (its own headings demoted two levels) and
add the previous visit's body under a `# Previous turn output` heading, so
every stage sees what the run is for even when an earlier stage summarised.
- `memory.md` is essential memory, hard-capped at 4 KB (`turn.EssentialCap`)
and inlined in every stage's prompt: the run task, the previous stage's
handoff (its `# handoff` section, else the head of its output) with outcome and
public/ additions, a one-line-per-stage outcome ledger and the server-captured
changed files. Every truncation names the file holding the full text.
- `history.md` is one `## <stage> · visit <n>` section per completed turn: the
turn body followed by a system-written, sorted list of the files that turn
added to `public/`. It is what `memory.md` used to be, and it is never inlined.
- `private/` belongs to one visit and is never handed to a later stage.
- `public/` is shared by every stage in the run.
The server writes `input.md`, `memory.md` and `history.md` before the model runs and
`output.md` after it. All three are inventoried as step artifacts, together with
`private/` and the files in `public/`. See
[stage artifacts](stage-artifacts.md) for capture limits and URLs.
Every stage gets `memory.md` inline. A stage with `read` is given the paths of
`input.md` and `history.md` to read only when it needs more detail. A stage
without `read` cannot, so the server also inlines `input.md` (capped at 32 KB);
`history.md` is never inlined. Long runs used to resend the whole history on
every model call (19 KB average, 92 KB maximum in measured runs); the essential
file bounds that per-call cost.
## Output contract
`output.md` must contain exactly one outcome heading. The heading may be any ATX
level and is followed by the first non-blank line, which must be exactly one of
the stage's outgoing edge names:
```md
## outcome
success
# Result
The rest of the output is free-form markdown.
```
Matching ignores case, surrounding whitespace and one trailing period, and returns the canonical edge
spelling. Headings and words inside fenced code blocks are ignored, so a code
sample cannot add a false outcome. Missing, duplicate, empty or undeclared
outcomes fail the stage. The parser never guesses, repairs the text, or looks for
any other section.
When a turn completes but its final message has no valid outcome, the harness
client (OpenCode or Pi) sends one short follow-up in the same session asking
only for the `# outcome` line and listing the allowed outcomes, without redoing
work. An outcome-only reply is placed at the top of the original output (any
invalid outcome section is replaced); a full reply replaces it. The outcome is
always the model's explicit answer, never inferred from prose. Its tokens count
toward the stage usage. If the reply is still invalid, the stage fails with the
same validation error as before.
Everything after the outcome value line is retained as the turn body under the
single `markdown` key of the internal output map. No JSON envelope, output
schema, field selection, `output_contract`, or format-repair pass exists for
model stages. Prompts should ask for free-form headings, never JSON fields.
Human and ntfy stages are unchanged and do not use turn files or the outcome
heading.
## Prompt shape
The server builds the prompt from the stage's own instructions plus runtime
facts:
- the paths to `memory.md`, `input.md` and `output.md` (or their contents for a
text-only stage);
- the stage prompt verbatim;
- the required shape, outcome first (`# outcome` and one of the allowed
outcomes, then `# handoff`, then any detail), with a worked example;
- the execution time and timezone;
- a granted resource target, when one exists;
- the writable `private/` and shared `public/` artifact directories, when they
exist;
- the authoritative permitted-tools line and the optional permitted shell
commands.
The model returns the complete `output.md` markdown as its final message. The
server writes that file and parses only the outcome.
## Failure diagnostics and retries
Retained stage failures use two fixed, sanitized categories:
- **FINAL RESULT VALIDATION** for an invalid turn output: missing, duplicate,
empty or undeclared outcome, or an unwritable `output.md`. Completed edits and
tool side effects are retained; a full retry repeats work.
- **HARNESS TRANSPORT** for request, active-session poll and final-readback
failures. A pre-admission reach failure (connecting or creating the session)
replays nothing and is retried on any stage. A post-admission deadline or
transport failure is retried only on an explicit read-only stage. A stage with
`edit`/`shell`, or with no tools block (the legacy read/glob/grep/edit/shell
bundle), never replays admitted tool work.
A validation failure is never reported as harness transport, and vice versa.
Neither diagnostic includes a provider response body, prompt, credential or
arbitrary model content.
## Optional machine-readable restrictions
`tools` enables actions per stage. A stage can narrow edit paths and shell
commands with an optional `permissions` block:
````markdown
```tools
read
glob
grep
edit
shell
```
```permissions
{"edit":["README.md","plugin/*.qml"],"shell":["bash -n install.sh"]}
```
````
These rules become OpenCode tool permissions, not just prompt advice. Edit
patterns are relative to the run directory and intersect any optional run-level
`--write-paths`. Shell entries are exact commands: no wildcards, chaining,
redirection or expansion. Empty lists deny an action; an omitted list retains
the normal defaults. Null lists, unknown keys, duplicate keys/resources, path
traversal, undeclared tools and unrestricted shell alongside restricted edits
are rejected. At most 20 resources per action are accepted.
Artifact access is separate and server-derived: a visit may write only its own
`private/` directory and the run-level `public/` directory. Those are the only
paths added to the permission rules and named in the prompt. The turn directory
itself is server-owned, so the model never writes `input.md`, `memory.md` or
`output.md` directly.
OpenCode checks scanner-produced shell command resources, not necessarily the
entire submitted script. A compound script consisting only of individually
allowed commands may pass; this is not a raw-script or shell-syntax firewall.
For a deliberately Go-only stage, explicitly declare `edit:["*.go"]` and the
exact allowed verification commands. Without a machine policy, a sentence such
as "approved Go files only" is **not** an enforced permission boundary. Defaults
remain language-neutral. Existing immutable definitions are not rewritten;
publish a new version to add intentional restrictions.
Tool permissions are not an OS sandbox. An approved command runs project code
with the host user's authority and can have filesystem/network side effects. Do
not grant shell access if stronger filesystem isolation is required.