stage turns: replace JSON results with file-based markdown turns

Each model visit now gets <run>/stages/<seq>-<stage>/ with input.md,
memory.md, output.md, private/ and a shared run-level public/. The model
reads the two input files, follows the stage's instructions verbatim and
returns complete markdown; only the # outcome section is parsed.

- add internal/turn layout, render and output parsing
- write input.md/memory.md before a visit and output.md from the final
  message; inventory all turn files as stage artifacts
- accumulate completed turns and system-written public file additions in
  memory, including across retries and resume
- remove the JSON envelope, strict field validation, line format and the
  repair ladder for model stages
- republish pipeline authoring prompts and docs in the new style

Checks: go build ./..., go vet ./..., go test -count=1 ./..., gofmt -l .
This commit is contained in:
2026-10-19 22:27:44 +01:00
parent b2fd614ba4
commit e37b864c40
66 changed files with 2199 additions and 3028 deletions
+60 -20
View File
@@ -1,34 +1,55 @@
# Stage turns
Stage turns are file-based. Each visit gets its own directory under the run:
Stage turns are file-based. Each model-stage visit gets its own directory under
the run's artifact root:
```
<run>/stages/01-implementer/input.md
<run>/stages/01-implementer/memory.md
<run>/stages/01-implementer/output.md
<run>/stages/01-implementer/private/
<run>/public/
<artifacts>/<run>/stages/01-implementer/input.md
<artifacts>/<run>/stages/01-implementer/memory.md
<artifacts>/<run>/stages/01-implementer/output.md
<artifacts>/<run>/stages/01-implementer/private/
<artifacts>/<run>/public/
```
`internal/turn.Layout` derives those paths deterministically (the sequence is
zero-padded to two digits). `Paths.Ensure` creates the turn, private and public
directories idempotently; it never deletes or rewrites existing files.
The first turn's `input.md` is the run prompt. A later turn's input is the
previous turn's output under a `# Previous turn output` heading. Memory is
plain markdown: one `## <stage> · visit <n>` section per completed turn, with
that turn's output verbatim followed by a system-written list of files added to
`public/`. List entries are relative to the run directory, sorted, and unique.
Input, memory and output are stored per turn, and private artifacts stay in the
turn's `private/` directory while shared artifacts go to the run-level
`public/` directory.
`input.md` is the run prompt on the first visit and the previous visit's output
body (without the outcome section) under a `# Previous turn output` heading
afterwards. `memory.md` is plain
markdown: one `## <stage> · visit <n>` section per completed turn, with that
turn's body followed by a system-written list of files it added to `public/`.
List entries are relative to the run directory, sorted, and unique.
The server writes `input.md` and `memory.md` before the model runs. The model
returns the complete markdown for `output.md`; the server writes that file from
the final message and then parses it. This is deliberate rather than a model
file write: a text-only stage (an explicit empty `tools` block) has no file
tool, and every harness then shares one result path. The prompt says the server
writes the file, and the artifact permission rules keep `input.md`, `memory.md`
and `output.md` server-owned. A text-only stage also has no file-reading tool,
so the server embeds both input files in its prompt instead. The turn files are
inventoried as step artifacts alongside `private/` (visible only to its own
visit) and `public/` (shared by every stage and listed with a `public/` prefix).
A selected-stage retry appends the complete merged retry input, including the
operator's `--input-file` values, to the selected visit's `input.md` under a
`# Retry input` heading. The step row alone is not the model's contract.
The public-file list is rebuilt from each step's persisted artifact inventory on
resume, so a restarted server does not print `_(none)_` for a turn that really
added files. A retry checkpoint also records each prefix turn's exact additions;
the new run copies those files from the source attempt's `public/` into its own
artifact root before the first visit, so memory names files the retry can read.
Files written by a failed visit are not carried or attributed to a later turn.
## Outcome-only output contract
A turn's `output.md` must contain exactly one outcome section and may contain any
markdown after it. The outcome section is an ATX heading named `outcome` (any
level) followed by the first non-blank line, which must be exactly one of the
stage's allowed outcome words:
markdown before or after it. The outcome section is an ATX heading named
`outcome` (any level) followed by the first non-blank line, which must be exactly
one of the stage's allowed outcome words:
```md
## outcome
@@ -42,6 +63,25 @@ The rest of the output is the body, preserved verbatim.
Matching trims whitespace and compares case-insensitively, returning the
canonical allowed spelling. Text before the heading is ignored. Headings and
words inside fenced code blocks (triple backticks or tildes) are ignored, so a
diff or code sample cannot add a false outcome. Missing, duplicate, empty, or unknown
outcomes are explicit errors; the parser never guesses, repairs the text, or
looks for any other section.
diff or code sample cannot add a false outcome. Missing, duplicate, empty, or
unknown outcomes are explicit errors; the parser never guesses, repairs the
text, or looks for any other section.
The server does not validate any other heading. The stage's body is retained
under the single `markdown` key of the internal output map for the existing UI,
and there is no JSON envelope, schema validation, or format-repair ladder.
## Authoring prompts
A stage prompt states the task plainly and asks for the complete `output.md`
markdown. The server adds the file paths, the model's permitted tools, the
allowed outcome words, and the requirement that the only interpreted section is
`# outcome`. Prompts should not ask for JSON fields: free-form headings are for
the next stage and the operator, not a server contract.
`memory` and `output` blocks are retired for model stages. An `output` block is
tolerated and ignored so an exported old definition can be edited in place, but
it neither selects nor validates fields. A `memory` block must be removed: it is
now an unknown model-stage block and fails validation. Use `prompt`, `tools`,
`edges`, `limits`, `configs` (and the type-specific
`permissions`/`checks`/`verification` blocks) to describe a stage.