Rewrite format.md around turn files, the output.md order (# outcome, edge word, # handoff, detail), checks, configs, human stages and the land stage delivery gate (sh -c in a fresh detached worktree, clean checkouts incl. untracked, 64 KB output cap, opencode only). Align review guidance across practices, verification, orchestration and the feedback-loop example. Fix loop-limit semantics in control.md against runner.go, budget_tokens sizing, read for large inputs, blocked-else-error and one stage naming set. Move dated verification logs to docs/verification-history.md and brokered chat, TUI keys and harness internals to docs/operator-reference.md; label operator-only references in SKILL.md.
18 KiB
System, storage and run evidence
Pipelines turns a reusable process into sequential stages with explicit handoffs, outcome routing and retained evidence. A Go CLI/TUI connects to an execution server; the server owns execution and SQLite history, OpenCode owns model/tool sessions, and optional ntfy delivery supplies notifications. It is technology-neutral, not a parallel/distributed task engine or a guarantee that model reports are correct.
Lifecycle and location
- Drafts are editable Markdown plus JSON metadata. Publishing validates and creates immutable versions; drafts remain retained. Keep context create-only/base-version guards. The four generic examples (one-stage, two-stages, feedback-loop, human-decision) are the template set; a brand-new database is seeded with them once, and an existing or already-seeded database never is. Templates can be created directly with an editable prefilled name or customized first. Editing a bundled repository template changes no installed or published definition; only explicit authoring then submission creates a new immutable version.
- A run pins its exact definition version, initial input and resolved directory. The client invocation directory is the default, but files/tools execute on the server. Docker needs that directory mounted at the same absolute path. A remote client does not upload its repository or make host credentials available remotely.
- Host and Docker OpenCode credentials are separate. Preserve setup-selected model inheritance; do not select a provider just because its ID appears in an example.
- Runtime caps are 30 stage steps and five visits to a stage. Runs in different
resolved workspaces execute concurrently; only a second run in a workspace an
active run is already changing is refused, naming the conflicting run and
advising you to wait or use a different workspace (for example another
checkout). Agent-session quiescence is scoped to the run: a run fails only when
one of its own sessions cannot be confirmed idle, so one run's active agent never
fails another's concurrent run. An active session left over by a restart with no
owning active run is an orphan and still refuses new work until interrupted.
Reusing that active run's request key reattaches idempotently.
type=humandecision stages exist and pause the run (seereferences/format.md); there are no parallel branches. Optional worktrees, full backup/restore, automatic database sync and independent verifier stage types are future ideas, not existing capabilities. Do not invent them in definitions. - Services are native or Docker-based. With an existing Docker installation do not start a competing native server. Server APIs are unauthenticated: trusted networks only; never recommend public exposure. Tool scope is not a universal OS sandbox.
Where data lives
SQLite retains pipelines/immutable versions, runs, recorded stage visits, inputs/outputs, status/errors, audit transcripts, session ownership/references, reported per-stage model usage (tokens/cost), retry links, schedules/state, notification receipts and file changes. Not everything is in that database: drafts/settings are client files; complete OpenCode sessions and credentials use OpenCode storage; repository artifacts are files; ntfy has its own storage. A pipeline ZIP is not a full backup of these components. Never merge or sync live SQLite files using generic file synchronization. Do not delete databases, sessions, volumes or credentials to repair a workflow.
Inspect results first
Use the authoritative CLI_PREFIX from the prompt:
CLI_PREFIX search QUERY [--type KIND] [--limit N] --json: one read-only query across pipelines, versions, runs, steps, field text and artifacts. QUERY is free text, a bare status word (queued/running/succeeded/failed/canceled/interrupted) selecting those runs, or a lone date expression (today,yesterday,this week— Monday 00:00 local — orYYYY-MM-DD) selecting the runs created in that window; anything else stays free text.--typeis the only narrowing mechanism and takes one ofpipeline,version,run,step,field,artifact; an unknown kind is an error. Ranking is exact name (3) > name substring (2) > body text (1), equal tiers by recency, and every hit reportsscore,matched_fieldandrecency. The cap defaults to 50 and clamps to 200; every kind scans at most 2000 rows and field scanning stops at 2000 steps / 1 MiB and reportstruncated/scanned_steps/scanned_bytes/cap/scan_bound_bytes. Targets are stable:pipeline:<id>,pipeline:<id>:v<version>,run:<id>,run:<id>:<sequence>,run:<id>:<sequence>:<field>, or an artifact URL.CLI_PREFIX runs --json: run history (optional--pipeline NAMEscope). Each run carries a bounded, single-linetitlesaying what it was for, set explicitly withCLI_PREFIX run ... --title "..."or generated from its prompt by the shared summariser on the configured default harness. It is never empty: when generation is unavailable or times out the server derives the prompt's first line, andtitle_source(model,fallback, orexplicit) plustitle_modelrecord how it was produced, so a diagnosis can tell them apart. Humanrunsandinspectshow the title beside the pipeline name and version, so two runs of one pipeline are distinguishable;CLI_PREFIX title RUN_ID "..."renames a run after the fact without changing its input or steps. The stored prompt is unchanged, and listing/renaming never calls a model.CLI_PREFIX inspect RUN_ID --json: selected run, visits and retained evidence. Each model step carries the usage its harness reported (usage.tokens_total,tokens_in/tokens_outonly whensplit_known,cost_usdonly whencost_known, plus the model the step actually ran with); the run carries the totalledusageover those stages, so a mixed-model run shows what each stage cost. An unreported cost or split is shown as unavailable, never as zero and never inferred from output length or transcript text; a historical or pre-upgrade run simply omitsusage. Exact stage outputs are authoritative; a transcript over the 32 KiB retained bound is a valid, credential-redacted preview envelope (with preserved stage limits, failure/format diagnostics and the retained session pointer), not the complete tool history.CLI_PREFIX decide RUN_ID [STAGE] --option KEY [--feedback TEXT] [--json]: answer a runwaitingon atype=humanstage (references/format.md).CLI_PREFIX run ... --json --wait --wait-timeout DURATION:--waitpolls until terminal and--wait-timeoutbounds only the client wait after admission, independent of per-request--timeout. A reached wait limit leaves the admitted run active; inspect it read-only withinspect RUN_ID --json, or keep waiting by rerunning--waitand reusing the original--request-key. Never start a duplicate run, and never expect an automatic replay or cancel of admitted work.CLI_PREFIX show NAME --version N --json: exact pinned definition, not whichever version happens to be latest now.CLI_PREFIX status --json: connection details; candidate addresses are not proof they are reachable from another device.CLI_PREFIX ping --json:pong.build.version,build.commit(full Git revision) andbuild.dirtyidentify the compiled server, not the client or its runtime checkout. A dirty build includes-dirty;commit: "unknown"ordirty: nullmeans metadata was unavailable. Older servers omitbuild. Never infer a deployed fix from source/skill contents. Native Go builds use embedded VCS metadata; Docker setup stamps build arguments. For an explicitly requested Docker rebuild,python3 scripts/build_metadata.pyresolves the current releases and stampsdocker compose buildfrom the checkout with the immutable digest; it does not restart/deploy. A plaindocker compose buildwithout that resolved digest cannot build the image. Plainpingalso displays server identity;CLI_PREFIX version --json(alias--version) identifies the client offline. Include both in sanitized bug reports. Two dirty builds at the same commit can differ. Docker setup verifies its started server identity against the build arguments before declaring setup complete.
Pipelines is a single-user first-version preview for supervised work, not a
multi-user service. Recommend a disposable repository for initial trials. Tool
permissions and same-path home mounts do not isolate valuable files. Never collect
raw credential/session databases or publish private inspect output as diagnostics.
In a source checkout, python3 scripts/smoke-first-install.py checks a disposable
source-built test binaries, draft/publication/transfer and restart persistence without
inference, notifications or deployment. It is not a live-provider or phone test.
That is a developer test fixture, not the native installer. End-user installation
runs natively through ./install by default (scripts/setup-native.py, with
--native equivalent); the Docker server + ntfy installation is explicit with
./install --docker.
Retained stage file changes
Each newly executed visit compares a bounded, read-only workspace snapshot just
before execution with one after execution/cleanup. SQLite retains unified patches
and metadata in steps[].file_changes; snapshots themselves are transient memory.
The stage's File changes section highlights added/removed lines and hunk headers.
- The baseline is stage-start content, not HEAD/index. Pre-existing edits and earlier stages' changes are not attributed to this visit. Added/deleted/modified files, modes and binary-change metadata are recorded; renames are delete/add.
- Git scope is tracked and nonignored untracked files within the run directory; outside Git, regular workspace files. Symlinks/special files, Git internals, node_modules and common environment/key/credential/database paths are excluded. This is path-based protection, not secret-content scanning. Patches may still contain sensitive text; never publish them without approval.
- Limits: 10,000 entries, 1 MiB/file and 32 MiB/snapshot of retained text,
256 MiB/hashed file, five-second snapshot context, 5,000 combined text lines,
64 KiB/patch, 128 KiB serialized file evidence/visit. Text over the 1 MiB or
32 MiB budget is still compared by hash, and in Git a path the snapshot could
not compare is listed when its
git statusentry changes; such entries carryno_patch: true(andtruncated) with no patch, and the visit ispartial.no_patch_reasonsays why (binary,file_size,budget,lines,evidence_size,git_status,baseline). In Git only dirty/untracked files spend the text budget; clean files' start content is read back from their verified stage-start index blob, so stage commits do not move the baseline. Missing/skipped/truncated captures must be described as partial/unavailable;inspectadds a run-levelwarningsentry naming every partial/unavailable visit. - Failed/canceled stages retain changes when execution returns; unconfirmed agent cleanup is partial. Crashes may have no after-capture. Historical visits say not recorded—never infer no edits from absent evidence.
- These are observed workspace changes, not exclusive authorship. Concurrent edits, changes outside scope, ignored new files and changes reverted before capture can be absent or mixed in. Capture failure does not change the declared run verdict.
Server skip fallback (skipped edge)
A non-human stage that declares a skipped edge is completed by the server, with
no model call, and routed along skipped only when every earlier visit of the
run recorded file_changes with status complete, none recorded any changed file,
and none captured an image artifact. The step's verdict reason starts
"Server fallback skip (no model run):" with reason_source: fallback. Any changed
file (no path or extension is treated as "not UI"), any partial/unavailable or
missing capture (crash, historical row, human decision, checkpoint-retry prefix)
runs the model, which may still answer skipped itself. auto_skip=false in the
stage's configs disables the fallback for that stage. Never describe a fallback
skip as a model judgement, and never treat it as evidence that nothing changed.
Do not add model output fields, new stages or extra model calls merely to obtain file diffs: current capture is server-owned. Do not promise full files/rollback.
Retained stage artifacts
Turn files (input.md, memory.md, history.md, output.md, private/) and
the run-level public/ are described in references/format.md. They live under
<data>/artifacts/<run>/stages/<seq>-<stage>/; the root is configurable and
defaults next to the database. The server owns input.md, memory.md,
history.md and output.md.
After the visit the server makes one bounded inventory of the turn and public
directories and records each file's name, size, content type (extension, sniff
fallback) and a relative url; these appear under each step's artifacts in
inspect RUN_ID --json and the TUI's A artifacts view, which groups them by
stage and visit, marks images and shows the recorded count,
capture-pending/not-recorded wording and partial-capture notes. Public entries
carry a public/ prefix. To see what a late stage was given, read its
memory.md; history.md is the full record it could consult.
GET /artifacts/<run>/<step>/<name> serves the bytes (correct Content-Type and
Content-Length, HEAD, and Range/206/416 via http.ServeContent), refusing
traversal, absolute paths, symlink escapes and unknown runs/steps. A pre-turn
<run>/<step>/ path stays readable for runs recorded before the file protocol
landed.
- Limits: 200 files, 8 MiB/file, 64 MiB total per visit; over-cap files are skipped and counted; symlinks and non-regular files are excluded.
- Artifacts are read-only evidence, not a gate: absent, unreadable, symlinked or oversized directories record nothing and never change the run verdict. The database keeps the recorded list, not file copies.
- Do not invent artifact files or claim a stage produced one without the recorded evidence; the server observes the directory after the stage.
Session export
- When an OpenCode-harness model stage finishes, the server exports its real
session read-only (v1
opencode export <id>, v2opencode session export <id>) and writes exactly one file,session.html, into the same turn artifacts directory; it lists and serves like any other artifact. - The
session.mdexport is retired: nothing writes it any more and it is never presented. A run recorded before the HTML-only change still listssession.mdamong its stored artifacts (dated evidence kept on disk, not deleted); every listing — app, TUI, web spike — hides it rather than showing it beside the html. - The source read is capped at 8 MiB and the rendered file at 8 MiB, matching the
ordinary per-artifact file bound; a cut source or a cut rendered file carries a
visible truncation banner at both ends and
session_export.truncated: true, and a truncated export is never called whole. - The step records
session_export(session_id,backend,source, the html file name,html_bytes,messages,truncated,diagnostic) ininspect RUN_ID --json. A record written before the change still carriedmarkdownandmarkdown_bytes; readers ignore those fields. - A failed export, or a Pi stage with no native export path, records a diagnostic
(Pi:
source: "unavailable") and never fails the run or changes the stage result. An existingsession.htmlis never overwritten.
Operator-only retention
Run history and its artifacts are never pruned automatically. The operator-only
pipelines retention report (strictly read-only) and pipelines retention apply
(removal after explicit approval) are the only commands that remove them; no
server startup, scheduler, list_runs, stats, runs or inspect path calls
retention, and there is no background goroutine, timer or startup hook. A policy
selects by rolling UTC age (--older-than), newest-N count (--keep), or their
intersection. Runs that are queued/running, waiting on a person, running a cleanup
agent, or referenced by a retained run's retry provenance are protected and
reported with a reason; their artifacts are kept. Removing a run removes its
<root>/<run> artifact directory and reclaims database space with one VACUUM
and one WAL checkpoint, reporting measured bytes. Pipeline definitions and their
immutable versions are never touched. The commands are local, require an existing
--db, and need no running server.
Debug and retry
Debug opens a new conversation with retained failure context, exact definition and session evidence; it does not resume implementation or grant permission to edit. Inspect first, identify evidence-supported causes, and say what remains unknown. Server-side session evidence is the latest bounded, credential-redacted evidence for up to the 3 most recent session-bearing steps; OpenCode V2 selects the latest assistant message with the same bounded readback as the final result, so a small final result is visible behind large prior tool content. Inspect the retained session itself when a preview is bounded or omitted; never assume a complete tool history from a preview. Retry is allowed for a failed run after cleanup stops: a fresh run/session starts at entry with the original input/directory/pinned version/edit scope. Previous side effects remain and can happen again. Never describe retry or cancellation as rollback, or resume/replay admitted tool work just to recover formatting.
Retained model-stage failures carry a fixed sanitized category. FINAL RESULT VALIDATION means output.md had a missing, duplicate, empty or undeclared
outcome (references/format.md); HARNESS TRANSPORT covers request, session-poll
and readback failures. Completed edits and tool side effects are retained, not
undone, so a full retry repeats work. A failure before any prompt is admitted is
retried within retries for any stage; a post-admission deadline or transport
failure is retried only on an explicit read-only stage; a stage with edit/shell
(or no tools block) is never replayed. Neither category exposes provider/model
bodies or credentials.