Files
pipelines/docs/model-results.md
alan 88dfc051e5 Model stages: outcome-first output shape and one in-session outcome repair
The prompt now says output.md must start with # outcome and the outcome
word alone on the next line, then # handoff, then detail. Runs 241/243
followed the old "start with # handoff" wording and dropped the outcome.

When a completed turn lacks a valid outcome, the OpenCode/agent and Pi
clients send one short follow-up in the same session asking only for the
outcome line. An outcome-only reply goes at the top of the original report;
a still-invalid reply fails with the same validation error as before. The
follow-up's tokens accrue to the session usage. No outcome is inferred.

Missing tools/evidence now name blocked when declared, else error. The
parser tolerates a trailing period on the outcome word.
2026-11-04 17:20:58 +00:00

7.6 KiB

Model turns, results and scoped tools

Model stages are file-based. Each visit is given its capped essential memory.md inline, may read input.md and history.md on demand, and returns the complete markdown for output.md. Only the outcome section is interpreted; the rest is free-form text for the next stage and the operator.

Turn files

Every model visit gets its own directory under the run's artifact root:

<artifacts>/<run>/stages/01-implementer/input.md
<artifacts>/<run>/stages/01-implementer/memory.md
<artifacts>/<run>/stages/01-implementer/history.md
<artifacts>/<run>/stages/01-implementer/output.md
<artifacts>/<run>/stages/01-implementer/private/
<artifacts>/<run>/public/
  • input.md is the run prompt on the first visit. Later visits keep the run prompt under a # Run task heading (its own headings demoted two levels) and add the previous visit's body under a # Previous turn output heading, so every stage sees what the run is for even when an earlier stage summarised.
  • memory.md is essential memory, hard-capped at 4 KB (turn.EssentialCap) and inlined in every stage's prompt: the run task, the previous stage's handoff (its # handoff section, else the head of its output) with outcome and public/ additions, a one-line-per-stage outcome ledger and the server-captured changed files. Every truncation names the file holding the full text.
  • history.md is one ## <stage> · visit <n> section per completed turn: the turn body followed by a system-written, sorted list of the files that turn added to public/. It is what memory.md used to be, and it is never inlined.
  • private/ belongs to one visit and is never handed to a later stage.
  • public/ is shared by every stage in the run.

The server writes input.md, memory.md and history.md before the model runs and output.md after it. All three are inventoried as step artifacts, together with private/ and the files in public/. See stage artifacts for capture limits and URLs.

Every stage gets memory.md inline. A stage with read is given the paths of input.md and history.md to read only when it needs more detail. A stage without read cannot, so the server also inlines input.md (capped at 32 KB); history.md is never inlined. Long runs used to resend the whole history on every model call (19 KB average, 92 KB maximum in measured runs); the essential file bounds that per-call cost.

Output contract

output.md must contain exactly one outcome heading. The heading may be any ATX level and is followed by the first non-blank line, which must be exactly one of the stage's outgoing edge names:

## outcome
success

# Result

The rest of the output is free-form markdown.

Matching ignores case, surrounding whitespace and one trailing period, and returns the canonical edge spelling. Headings and words inside fenced code blocks are ignored, so a code sample cannot add a false outcome. Missing, duplicate, empty or undeclared outcomes fail the stage. The parser never guesses, repairs the text, or looks for any other section.

When a turn completes but its final message has no valid outcome, the harness client (OpenCode or Pi) sends one short follow-up in the same session asking only for the # outcome line and listing the allowed outcomes, without redoing work. An outcome-only reply is placed at the top of the original output (any invalid outcome section is replaced); a full reply replaces it. The outcome is always the model's explicit answer, never inferred from prose. Its tokens count toward the stage usage. If the reply is still invalid, the stage fails with the same validation error as before.

Everything after the outcome value line is retained as the turn body under the single markdown key of the internal output map. No JSON envelope, output schema, field selection, output_contract, or format-repair pass exists for model stages. Prompts should ask for free-form headings, never JSON fields.

Human and ntfy stages are unchanged and do not use turn files or the outcome heading.

Prompt shape

The server builds the prompt from the stage's own instructions plus runtime facts:

  • the paths to memory.md, input.md and output.md (or their contents for a text-only stage);
  • the stage prompt verbatim;
  • the required shape, outcome first (# outcome and one of the allowed outcomes, then # handoff, then any detail), with a worked example;
  • the execution time and timezone;
  • a granted resource target, when one exists;
  • the writable private/ and shared public/ artifact directories, when they exist;
  • the authoritative permitted-tools line and the optional permitted shell commands.

The model returns the complete output.md markdown as its final message. The server writes that file and parses only the outcome.

Failure diagnostics and retries

Retained stage failures use two fixed, sanitized categories:

  • FINAL RESULT VALIDATION for an invalid turn output: missing, duplicate, empty or undeclared outcome, or an unwritable output.md. Completed edits and tool side effects are retained; a full retry repeats work.
  • HARNESS TRANSPORT for request, active-session poll and final-readback failures. A pre-admission reach failure (connecting or creating the session) replays nothing and is retried on any stage. A post-admission deadline or transport failure is retried only on an explicit read-only stage. A stage with edit/shell, or with no tools block (the legacy read/glob/grep/edit/shell bundle), never replays admitted tool work.

A validation failure is never reported as harness transport, and vice versa. Neither diagnostic includes a provider response body, prompt, credential or arbitrary model content.

Optional machine-readable restrictions

tools enables actions per stage. A stage can narrow edit paths and shell commands with an optional permissions block:

```tools
read
glob
grep
edit
shell
```

```permissions
{"edit":["README.md","plugin/*.qml"],"shell":["bash -n install.sh"]}
```

These rules become OpenCode tool permissions, not just prompt advice. Edit patterns are relative to the run directory and intersect any optional run-level --write-paths. Shell entries are exact commands: no wildcards, chaining, redirection or expansion. Empty lists deny an action; an omitted list retains the normal defaults. Null lists, unknown keys, duplicate keys/resources, path traversal, undeclared tools and unrestricted shell alongside restricted edits are rejected. At most 20 resources per action are accepted.

Artifact access is separate and server-derived: a visit may write only its own private/ directory and the run-level public/ directory. Those are the only paths added to the permission rules and named in the prompt. The turn directory itself is server-owned, so the model never writes input.md, memory.md or output.md directly.

OpenCode checks scanner-produced shell command resources, not necessarily the entire submitted script. A compound script consisting only of individually allowed commands may pass; this is not a raw-script or shell-syntax firewall.

For a deliberately Go-only stage, explicitly declare edit:["*.go"] and the exact allowed verification commands. Without a machine policy, a sentence such as "approved Go files only" is not an enforced permission boundary. Defaults remain language-neutral. Existing immutable definitions are not rewritten; publish a new version to add intentional restrictions.

Tool permissions are not an OS sandbox. An approved command runs project code with the host user's authority and can have filesystem/network side effects. Do not grant shell access if stronger filesystem isolation is required.