Files
pipelines/docs/stage-artifacts.md
T
alan e37b864c40 stage turns: replace JSON results with file-based markdown turns
Each model visit now gets <run>/stages/<seq>-<stage>/ with input.md,
memory.md, output.md, private/ and a shared run-level public/. The model
reads the two input files, follows the stage's instructions verbatim and
returns complete markdown; only the # outcome section is parsed.

- add internal/turn layout, render and output parsing
- write input.md/memory.md before a visit and output.md from the final
  message; inventory all turn files as stage artifacts
- accumulate completed turns and system-written public file additions in
  memory, including across retries and resume
- remove the JSON envelope, strict field validation, line format and the
  repair ladder for model stages
- republish pipeline authoring prompts and docs in the new style

Checks: go build ./..., go vet ./..., go test -count=1 ./..., gofmt -l .
2026-10-19 22:27:44 +01:00

6.5 KiB

Stage artifacts

A stage that produces images, screenshots, reports or logs - typically written into an ignored or untracked directory - would otherwise leave nothing retrievable per step. Every model visit therefore gets a file-based turn directory, and the files it contains are recorded on the step and served over HTTP.

Where they live

The server creates this layout under the run's artifact root:

<artifacts root>/<run>/stages/<seq>-<stage>/input.md
<artifacts root>/<run>/stages/<seq>-<stage>/memory.md
<artifacts root>/<run>/stages/<seq>-<stage>/output.md
<artifacts root>/<run>/stages/<seq>-<stage>/private/
<artifacts root>/<run>/public/

The sequence is zero-padded to two digits. The artifacts root defaults to the directory holding the database (<data directory>/artifacts) and can be pointed elsewhere with the server's ArtifactsRoot option. Only model stages get a turn directory.

  • input.md, memory.md and output.md are server-written turn files.
  • private/ belongs to this visit. It is writable by the stage and is never handed to a later stage.
  • public/ is run-level and shared by every stage. Files are writable and readable by each visit.

The layout and ownership rules live in internal/turn.Layout. A retry is a new run with its own artifact root, so it never overwrites the failed attempt's turn files.

What the stage is told

The prompt names the turn directory, the writable private/ directory and the shared public/ directory, and says that the server owns input.md, memory.md and output.md. Those are the exact paths the tool permission rules grant. A Pi stage also receives PIPELINES_ARTIFACTS_DIR; OpenCode V2 tools run inside the shared serve process, so its prompt is the reliable carrier.

Turn-local artifacts go in private/. Reports, logs and screenshots that later stages should see go in public/. There are no technology, language or file-type rules.

What is recorded

After the stage finishes and before the step is stored, the server makes one bounded inventory of the turn directory and the run's public directory. It records, per file, its name, size_bytes, content_type (from the extension, with a content sniff of at most the first 512 bytes as a fallback) and a server-relative url. Public entries carry a public/ name prefix. The same fields appear under each step's artifacts in pipelines inspect RUN_ID --json and in the TUI.

input.md, memory.md and output.md are the primary artifacts of every model visit. private/ files are listed without a prefix; shared files are listed as public/... on every visit that can see them.

  • Limits: 200 files per step, 8 MiB per file and 64 MiB total. Over-cap files are skipped and the skipped/truncated counts are recorded with the step.
  • Symlinks in any component and non-regular files are excluded, and the walk is bounded by the file cap so a large directory is never fully enumerated.
  • Artifacts are evidence, not a gate. A missing, unreadable, symlinked, oversized or empty directory records nothing and never fails the run or changes its outcome. The database retains the recorded list, not file copies.

Fetching bytes

GET /artifacts/<run>/<step>/<name> on the server that serves /healthz and /connect returns the file. public/<name> maps to the run-level public directory; any other name maps to that sequence's turn directory. Runs recorded before the file-based protocol landed remain readable from the older <run>/<step>/ layout.

Any regular file contained in that visit's turn directory or the run's public/ directory is reachable, including a file added after the bounded inventory or skipped by its size/file caps; the URL is not a lookup in the recorded list. Path traversal, absolute paths, symlink escapes and unknown runs/steps are refused. Responses carry the correct Content-Type and Content-Length; HEAD returns headers without a body; and Range requests return 206 Partial Content with the exact slice (or 416 when unsatisfiable), so images and video can stream. http.ServeContent streams from the opened file, so a request never loads the whole file into memory.

Session export

When a model stage that ran against an OpenCode harness finishes, the server also exports the session the harness actually produced and writes it into the same turn directory as session.html. It is an ordinary artifact: the bounded inventory lists it and GET /artifacts/<run>/<step>/session.html serves it with no special casing.

  • Source: the read-only native export path. The native v1 CLI exposes it as opencode export <id>; the installed v2 CLI exposes it as opencode session export <id>. Both reuse the same invocation helper and session-ID/messages validation as the transcript readback. The file is the harness's real session, never a reconstruction.
  • Bounds: the source read is capped at 8 MiB and the rendered file at 2 MiB. When either is cut, the file carries a visible truncation banner near the start and the end and the step records truncated: true; a truncated export is never described as whole.
  • Evidence: the step records session_export (session_id, backend, source, html, html_bytes, messages, truncated, diagnostic), so pipelines inspect RUN_ID --json shows what was written without reading the file.
  • Isolation: exporting never changes the stage's status, outcome, target or result validation, never waits for the session, never mutates, resumes or interrupts it, and a failure records a sanitized diagnostic and continues. An existing session.html is never overwritten and the skip reason is recorded.
  • Pi limitation: a Pi stage has no native session export path, so the step records source: "unavailable" with an explicit diagnostic and writes no file.

A run recorded before the HTML-only change may still list session.md among its stored artifacts; the dated evidence is kept on disk rather than deleted.

Removal with run history

Run artifacts live under <root>/<run> and last as long as the run does. pipelines retention report/apply (see CLI and agent authoring) is the only thing that removes them, and it does so only for a run the operator explicitly selected for removal. Removing a run removes its whole artifact directory; protected and retained runs keep theirs, and their recorded artifact URLs keep serving. A symlink at <root>/<run> is removed as a link without touching its target, and a regular file at that path is refused rather than deleted. Nothing removes artifacts automatically.