1278 lines
79 KiB
Markdown
1278 lines
79 KiB
Markdown
# CLI and agent authoring
|
||
|
||
In the TUI, publishing is labelled **Create pipeline** for a new definition and
|
||
**Update pipeline** for a new immutable version. The CLI `submit` command and
|
||
`:submit` shortcut remain compatible. Templates offer description/graph panes:
|
||
Tab expands the focused pane, `n/p` selects a template, and `hjkl` pans the focused
|
||
graph. Create opens one confirmation with an editable, prefilled template name;
|
||
Enter creates the pipeline without a separate naming or editor step. Duplicate
|
||
names are rejected rather than updating an existing pipeline.
|
||
|
||
`pipelines help` lists commands; `pipelines help edit` (or `edit --help`) shows
|
||
command-specific flags and examples. Installed commands work from any directory.
|
||
The installed command is a compiled Go binary. Optional autostart uses the OS user
|
||
service manager directly. Legacy Python-managed installations require an explicit
|
||
`PIPELINES_PREFIX`; no configuration is inferred from the executable's location.
|
||
For source development, use `go run ./cmd/pipelines help`; start the native server
|
||
separately. Set `PIPELINES_URL` or pass `--url` to select another server.
|
||
|
||
## Dashboard and status
|
||
|
||
`pipelines --url ws://HOST:PORT/connect ping --json` returns `pong` with the
|
||
running server's compiled `build.version`, full Git `build.commit`, and
|
||
`build.dirty`. Uncommitted builds have a `-dirty` suffix. Missing metadata is
|
||
explicit (`commit: "unknown"`, `dirty: null`); older servers omit `build`.
|
||
This identifies the server binary, not the CLI or files mounted into the container.
|
||
Plain `ping` also prints the server version/commit and, beside the local client
|
||
build, the client/server relationship. `pipelines version --json`
|
||
(or `pipelines --version`) identifies the local client without connecting to a
|
||
server. Include both identities when reporting a problem; they can differ.
|
||
|
||
`ping`, `connection`, `status` and every server command now check that relationship.
|
||
The server stamps its `build` and integer `protocol` onto each response (the
|
||
unsolicited welcome envelope too). The client compares this build against the
|
||
server: a same-protocol build mismatch prints exactly one `warning:` on stderr
|
||
(stdout JSON stays parseable) and a server speaking a newer protocol fails the
|
||
command non-zero before it acts on the frame. If a response carries a top-level
|
||
field this client does not model — for example an `artifacts` field from a newer
|
||
server — the client prints a one-time `server sent unrecognized top-level field(s)
|
||
...` notice instead of silently appearing to have no data. Rebuild or reinstall the
|
||
CLI so it matches the server; the installer also warns when a different `pipelines`
|
||
resolves earlier on PATH than the freshly installed binary.
|
||
|
||
|
||
Home's `r` opens **all** runs, not only failures or the newest 50. Inside a pipeline,
|
||
`r` and `:runs` show only that pipeline's runs across its versions. In any run list,
|
||
press `/` for fuzzy search, type to filter, and Enter to return to the results.
|
||
Search includes pipeline name, version, status, outcome, stage and diagnostic;
|
||
`#123` searches run-ID prefixes. Esc while searching clears the filter without
|
||
closing the list. The filter survives inspection/back and live list refreshes.
|
||
Home's `/` instead opens the omni popup, which searches pipelines, versions, runs,
|
||
steps/visits, artifact names, and the text of stage inputs, outputs and evidence.
|
||
Each row carries a short type badge (`pipeline`, `run`, `step`, `artifact` or
|
||
`match`); a text match also shows a snippet with the matched part bracketed so it
|
||
is legible without colour. A bare status word (`queued`, `running`, `succeeded`,
|
||
`failed`, `canceled`, `interrupted`) or a date (`today`, `yesterday`, `this week`,
|
||
or `YYYY-MM-DD` in local time) selects runs. Enter opens the matched pipeline,
|
||
run, stage visit, artifact or field. The same query is available non-interactively
|
||
as `pipelines search QUERY [--type KIND] [--limit N] [--json]`; `--type` is the
|
||
only kind-narrowing mechanism.
|
||
Wide terminals
|
||
show runs and pipelines side by side, with stacked panels on narrow terminals.
|
||
Home run cards prioritize reported results, status, completed visits and duration;
|
||
active cards also show the current stage. Start/finish times remain in run details.
|
||
The usage summary is aggregated over all stored history, including retry attempts:
|
||
total/active runs, outcomes, success rate, average completed duration, pipeline
|
||
usage and a zero-filled 14-day daily activity chart (UTC). Success rate is
|
||
succeeded / (succeeded + failed); active, canceled and interrupted runs are excluded.
|
||
The average includes all runs with a finish timestamp. It also totals the reported
|
||
model usage across history: `tokens_total`, an in/out split only when every
|
||
token-reporting stage reported one, and the summed cost, plus counts of stages
|
||
that did or did not report a cost. A stage whose provider reported no cost is
|
||
counted as cost-unknown and never contributes a fabricated or zero cost, and no
|
||
cost/token count is inferred from output length or transcript text. The totals are
|
||
the sum of the per-run totals `inspect` shows. Only usage rows whose recorded
|
||
step still exists are counted, so rows orphaned by an old history wipe are ignored
|
||
(not deleted). `pipelines stats --json` exposes these aggregates.
|
||
|
||
Use `:status` on any TUI page to open server and ntfy information. Scroll with
|
||
arrows or `j/k`, PgUp/PgDn; `r` refreshes and Esc closes without changing the page.
|
||
`pipelines status [--json]` returns the same server data, including hostname,
|
||
start/uptime, the connected server endpoint/port, active interface IPs, ntfy
|
||
publishing health, public phone URL/topic/port and candidate URLs. It does not
|
||
start services or send a notification. The ntfy health probe has a two-second
|
||
deadline; it checks the publishing server, not phone reachability.
|
||
|
||
IPs belong to the **server computer or container**, not necessarily the client.
|
||
Candidate IP/port combinations are not reachability checks: loopback bindings,
|
||
firewalls, container port mapping and TLS can prevent access. ntfy candidates
|
||
are only derived when a configured endpoint identifies this computer; remote
|
||
ntfy hosts are not misrepresented as local. Publishing URLs and credentials are
|
||
never returned. Missing public phone configuration is labeled explicitly.
|
||
`pipelines service status` remains the separate OS autostart status command.
|
||
|
||
## Search
|
||
|
||
`pipelines search QUERY [--type KIND] [--limit N] [--debug-files] [--json]` runs one read-only
|
||
query across the whole system: pipeline names and their versions, run identity,
|
||
status and dates, stage and visit identifiers, artifact names and types, and the
|
||
text of stage inputs, outputs, evidence and transcripts. It never reads
|
||
artifact contents, never makes a per-result follow-up request, and asks for no
|
||
confirmation.
|
||
Artifact hits are deliverables only: runner-internal debugging files (see
|
||
[Stage artifacts](stage-artifacts.md#deliverables-and-debugging-files)) match
|
||
only with `--debug-files` (`include_debug` on the request).
|
||
|
||
- Free text matches case-insensitively. A bare status word (`queued`, `running`,
|
||
`succeeded`, `failed`, `canceled`, `interrupted`) selects those runs, and a lone
|
||
date expression (`today`, `yesterday`, `this week` — Monday 00:00 local — or a
|
||
literal `YYYY-MM-DD`) selects the runs created in that window; any other text,
|
||
including a status word inside a sentence, stays free text.
|
||
- `--type` is the only narrowing mechanism and takes one kind: `pipeline`,
|
||
`version`, `run`, `step`, `field` or `artifact`. An unknown kind is an error,
|
||
never a silently ignored filter.
|
||
- Order: results are newest first by `recency` (pipeline/version creation,
|
||
run creation, step start), then numeric id and target, before the cap applies,
|
||
so the cap drops the oldest hits; each kind's row scan also walks newest
|
||
first. `score` still records the match tier (exact name 3, name substring 2,
|
||
body text 1). Every result carries `score`, `matched_field` and `recency` so a
|
||
client can re-rank.
|
||
- Bounds: the result cap defaults to 50 and clamps to 200 (`0` = default);
|
||
every kind scans at most 2000 rows, and field-text scanning stops at 2000 steps
|
||
or 1 MiB decoded. `truncated` plus `scanned_steps`, `scanned_bytes`, `cap` and
|
||
`scan_bound_bytes` report when a bound applied.
|
||
- Targets are stable and navigable: `pipeline:<id>`, `pipeline:<id>:v<version>`,
|
||
`run:<id>`, `run:<id>:<sequence>`, `run:<id>:<sequence>:<field>`, or an
|
||
artifact URL served by the artifact endpoint.
|
||
- Chats: a `search` request with `kind: "chat"` (protocol only; the all-kind
|
||
search never includes chats, so older clients see no change) matches the
|
||
operator's chat sessions. Every listed chat's title and directory are
|
||
matched from the registry; for up to 24 of the most recently used chats not
|
||
matched there, the newest 100 messages' text is read through the chat broker,
|
||
four at a time, within a one-second budget. A transcript still unread when the
|
||
budget expires is left out and the reply says `truncated`; `scanned_chats`
|
||
reports how many transcripts were read. Each chat yields at most one hit
|
||
(`target` `chat:<session id>`, `matched_field` `title`, `directory` or
|
||
`message`) whose `chat` payload carries the title, directory, harness and,
|
||
for a message match, the message id, role and a snippet. Hits are ordered by
|
||
the chat's last use, newest first, whatever field matched, before the cap
|
||
applies. The server answers it
|
||
off the connection's read loop, so it never delays other replies; a newer chat
|
||
search on the same connection cancels the previous one. The app sends it after
|
||
its run search and shows the hits as a Chats group.
|
||
|
||
```sh
|
||
pipelines search "release notes" --json
|
||
pipelines search failed --json
|
||
pipelines search 2026-10-16 --type run --json
|
||
pipelines search "acceptance check" --type field --limit 10
|
||
```
|
||
|
||
## Draft-first harness workflow
|
||
|
||
```sh
|
||
pipelines new # the configured harness asks what to build and names it
|
||
pipelines edit # fuzzy pipeline selection
|
||
pipelines edit "bug fix" # ask the harness what to change
|
||
pipelines edit "bug fix" "Add a review step"
|
||
pipelines new "release notes" "Summarize changes and notify my phone"
|
||
pipelines new "Add a feature" --template feedback-loop
|
||
```
|
||
|
||
`new` opens the configured harness (Pi or OpenCode; host by default, or
|
||
Docker-linked when a checkout is linked) immediately when its name/request is
|
||
omitted. The skill asks only for missing
|
||
decisions and records the chosen name in draft metadata. `edit` accepts an exact
|
||
name or ID, with names taking precedence over numeric IDs. Both retain private
|
||
drafts and launch the harness once with a short `@pipelines` reference and
|
||
context-file path. There is no preliminary harness API call, temporary service,
|
||
session creation, or instruction preload. The prompt's CLI prefix is
|
||
Docker-aware: a Docker-linked profile embeds the in-container `pipelines`
|
||
executable and, for this Compose service's own loopback endpoint, scoped
|
||
transport aliases. Loopback is recognized only for parsed IP literals and the
|
||
exact `localhost` name (case-insensitive), so a remote DNS name such as
|
||
`127.remote.example` stays remote with no Docker probe. The single bounded
|
||
`docker compose port` probe must return a complete `host:port` binding that
|
||
covers the requested loopback address and port (wildcard or matching family);
|
||
malformed or non-matching bindings fail clearly, and a loopback `wss` endpoint
|
||
is rejected as having no supported local TLS mapping rather than downgraded to
|
||
plaintext. nvim remains a client-side editor.
|
||
`./install` installs/updates the global skill and format reference for OpenCode,
|
||
Claude Code (`~/.claude/skills`) and Codex (`$CODEX_HOME/skills`, default
|
||
`~/.codex/skills`). Plain
|
||
`go install` users run `pipelines skill install` separately. Shared login/config
|
||
and user-modified skills are preserved; harness permissions remain in effect.
|
||
|
||
The skill encourages editing only the draft, running server validation, summarizing
|
||
the changes, and leaving submission to the user. This is an instruction, not a
|
||
security sandbox: an agent with CLI/shell access can submit when explicitly asked.
|
||
Every submission, including agent-issued `submit`/`rpc`, still uses server validation.
|
||
|
||
## Drafts in the TUI
|
||
|
||
The server runs one active run per resolved workspace, including overlapping
|
||
parent/descendant directories (and symlink aliases). Separate sibling worktrees
|
||
remain independent. A new request key in a
|
||
different workspace is admitted immediately and executes concurrently; a second
|
||
run in a workspace an active run is already changing is refused, naming the
|
||
conflicting run and advising you to wait or use a different workspace (for
|
||
example another checkout of the repository). Reusing the original request key
|
||
reattaches idempotently.
|
||
|
||
`cancel ID` / `cancel_run` records `cancel_requested: true` before signaling the
|
||
executor. The reply can still have `status: running` (or `queued`) and no
|
||
`finished_at`: this means **stopping**, not stopped. The workspace stays reserved
|
||
until execution returns and owned-session cleanup is confirmed; late success
|
||
cannot override cancellation. A parked human run has no executor and cancels
|
||
immediately. Repeating Stop while stopping is safe.
|
||
|
||
If cleanup cannot be confirmed, the run stays nonterminal with
|
||
`cleanup_unconfirmed: true`, retains partial step evidence, and reserves its
|
||
workspace. The web client labels this explicitly and hides duplicate Stop.
|
||
Successful startup recovery interrupts and confirms recorded sessions before
|
||
settling the run; without a recoverable session pointer, uncertainty and the
|
||
reservation survive restart rather than inventing stop confirmation. Recovery
|
||
does not resume a canceled run. A database-sidecar advisory server lock is held
|
||
through runner shutdown, so a second server on that database cannot recover or
|
||
interrupt the first one's live runs. Keep the sidecar file in place; its existence
|
||
is not ownership (the kernel lock is). This is a local-filesystem lock, not a
|
||
distributed lease; hard-link database aliases and filesystems without reliable
|
||
advisory locking are unsupported.
|
||
|
||
Use `:templates` for the four manual-only examples, read in order: one stage,
|
||
two stages, a feedback loop and a human decision (`examples/README.md`). A
|
||
brand-new database already holds them as ordinary pipelines; the server seeds
|
||
them once and never again, so removing them or upgrading never brings them back.
|
||
**Submit template** (`s` or Enter) asks for a
|
||
name without opening an editor. Enter saves a retained draft, then a separate
|
||
confirmation publishes it as a new pipeline. Esc before naming creates nothing;
|
||
Esc at publication keeps the draft. **Customize** (`e`) opens the editor picker
|
||
instead. Both paths inherit the configured default model and preserve existing
|
||
pipelines/history. Only the stages that make the requested change (`implement`,
|
||
`act`) may edit; the others read, and the loop's reviewer also runs read-only
|
||
checks. No example commits, pushes or deploys. A genuine operator gate is a
|
||
`type=human` decision stage, never a model stage pretending to approve. Editing a
|
||
bundled template source changes no installed or published definition; a template
|
||
becomes an immutable version only after explicit authoring and publication.
|
||
|
||
Home's `n` / `:new` and **Customize** (`e`) on a template first ask how to author the new
|
||
pipeline: **nvim** opens a retained Markdown draft file, or a configured harness
|
||
(Pi/OpenCode) opens the collaborative session with `@pipelines`. Tab/Shift+Tab or
|
||
arrows select; typing fuzzy-searches the choices; Enter launches and Esc cancels
|
||
without creating a draft. The choice overrides `VISUAL`/`EDITOR`. New drafts preserve
|
||
the create-only publication guard and still require explicit submission unless
|
||
you already published them from the harness. The configured harness can ask for a
|
||
missing name; for an unnamed nvim draft, `pipelines submit <draft.md> --name NAME`
|
||
names and publishes it. CLI `pipelines new` retains its existing direct-harness
|
||
behavior; `--no-open` creates the file without launching an editor.
|
||
|
||
Press `d` on the dashboard or use `:drafts` to browse the same retained drafts as
|
||
`pipelines drafts`. Use `n/p` to select, `j/k` to scroll source, `e`/Enter to reopen
|
||
the existing draft, `v` for server validation, and `s` for submission confirmation.
|
||
`r` refreshes the list. Drafts can be validated, edited, and submitted to whichever
|
||
server you are connected to. Same-server submissions preserve create-only/version
|
||
guards. A different destination imports the draft as a new pipeline, ignoring
|
||
the origin's version number and refusing an existing destination name; the
|
||
confirmation shows the destination, and opening, validating or submitting a draft
|
||
whose recorded server differs from the configured one prints a notice naming both
|
||
URLs so it is never silently retargeted. Retained source and origin metadata stay
|
||
unchanged. For an edit draft, CLI `submit --expected-version N` can explicitly
|
||
approve an update against version N on the destination (or `--force` can explicitly
|
||
skip that version check). For unnamed drafts, submission asks you to
|
||
type a pipeline name directly in the TUI. Enter saves the name in retained metadata
|
||
and opens publication confirmation; it does not publish yet. Esc keeps the draft
|
||
without saving unfinished name input. Draft selection/editor completion never publishes it.
|
||
On editor exit, the connected TUI checks the authoritative server before asking
|
||
to submit. If the draft source is already published (including a newer version
|
||
after the draft's base that was subsequently superseded), it skips confirmation
|
||
and reloads the latest pipeline/page and Home list. Local-only changes still need
|
||
explicit submission. A failed check retains the draft and says publication could
|
||
not be verified; it never assumes publication from an editor's exit code. Original
|
||
draft conflict guards and retained draft files remain unchanged.
|
||
New/edit drafts open the configured default harness; `--harness opencode|pi|nvim`
|
||
selects another configured harness or nvim. TUI pickers show only configured
|
||
harnesses, default first, plus nvim. See [harnesses and runtime control](harnesses-and-control.md)
|
||
for fuzzy model setup, separate defaults, shared memory and runner-owned limits.
|
||
|
||
For agents or scripts that should not open a terminal:
|
||
|
||
```sh
|
||
pipelines new "release notes" --no-open
|
||
pipelines edit "bug fix" --json
|
||
pipelines drafts --json
|
||
pipelines validate /path/to/pipeline-123.md --json
|
||
pipelines submit /path/to/pipeline-123.md --json
|
||
```
|
||
|
||
`validate` and `submit` apply the authoring lint to new drafts (stored versions
|
||
keep loading and running unchanged). Errors reject the draft and are listed
|
||
together, one `stage KEY: RULE: message` per line: a stage that can modify the
|
||
workspace without a `checks` block (`none: <reason>` or, on a `delivery=true`
|
||
stage, the commands), a prompt naming an undeclared outcome ("report skipped"),
|
||
a "the server lands it" claim with no `delivery=true` stage, a loop edge without
|
||
`limits`, and a check guard that passes without running (`|| true`,
|
||
`[ ! -x tool ] || ...`). Warnings print as `warning: ...` on stderr (the
|
||
`warnings` array with `--json`) and do not block: runner boilerplate in a prompt,
|
||
a prompt over 2000 bytes, an instruction to read the full history every visit,
|
||
and back-quoted non-outcome words after return/report. The skill's
|
||
`references/practices.md` explains each rule.
|
||
|
||
`--json` never opens an editor or picker; provide a name explicitly. It returns
|
||
draft metadata including `path`, `name`, `server`, `create_only`, and `base_version`.
|
||
`--no-open` prints the path and review commands. `$PIPELINES_DRAFT_DIR` overrides
|
||
the shared cache directory. CLI and TUI drafts use private Markdown plus metadata
|
||
sidecars; `submit FILE` can publish either. External files require `--name NAME`.
|
||
|
||
New drafts use create-only submissions and cannot overwrite existing names.
|
||
Edits carry a latest-version guard checked atomically by the server. On conflict,
|
||
inspect/export the latest version and reconcile the draft. `--force` explicitly
|
||
bypasses the version guard only, **never** syntax, graph, model, tool, schedule,
|
||
or notification validation. Original versions/sources remain immutable.
|
||
Submitting activates included schedules, so review them before publishing.
|
||
|
||
## CLI/TUI capability map
|
||
|
||
| TUI action | Agent-capable CLI |
|
||
| --- | --- |
|
||
| Browse/search pipelines | `list --json`, `show NAME --json` |
|
||
| Search everything | `search QUERY --json` (see [Search](#search)) |
|
||
| Original source and history | `export NAME --version N`, `versions NAME --json` |
|
||
| Browse templates/source | `templates --json`, `template KEY --source` |
|
||
| Create or edit a definition | `new`, `edit`, `drafts` |
|
||
| Validate and publish | `validate FILE`, `submit FILE` |
|
||
| Start a run | `run NAME PROMPT [--title TITLE] [--model P/M] [--stage-model S=P/M] --yes --json` |
|
||
| Run history/inspection | `runs --json`, `inspect ID --json` |
|
||
| Rename a run | `title ID "TITLE" --json` |
|
||
| Decide a waiting human stage | `decide ID [STAGE] --option KEY [--feedback TEXT] --json` |
|
||
| Cancel | `cancel ID --json` |
|
||
| Connection and phone settings | `connection --json`, `ping --json` |
|
||
| Server/ntfy status popup | `status --json` |
|
||
| Usage statistics/charts | `stats --json` |
|
||
| Run-history retention | `retention report --keep N --json`, `retention apply --keep N --yes` |
|
||
| Retire an unused pipeline | `archive NAME --yes`, `unarchive NAME`, `list --archived` |
|
||
| Legacy operational diagnostics | managed `logs`, `doctor` |
|
||
|
||
The TUI does not spawn CLI processes for each request. Both front ends share
|
||
`internal/client`'s command transport and `internal/draft`'s persisted drafts;
|
||
the server owns all validation, persistence, execution, and schedules. `rpc`
|
||
exposes the full protocol as an escape hatch, so no server command is TUI-only:
|
||
|
||
```sh
|
||
printf '%s' '{"type":"get_pipeline","pipeline_id":1,"version":1}' | pipelines rpc -
|
||
```
|
||
|
||
Existing protocol commands: `ping`, `list_pipelines` (optional `include_archived`),
|
||
`get_pipeline`, `archive_pipeline`, `unarchive_pipeline`,
|
||
`list_templates`, `validate`, `submit`, `run_pipeline`, `list_runs`, `get_run`,
|
||
`cancel_run`, `retry_run`, `decide_run`, `set_run_title`, `get_run_evidence`,
|
||
`get_status`, `search`. `search` takes a `search` request object (`query`, with an
|
||
optional `kind` and `limit`) and returns a `search_results` response whose ranked
|
||
`results` each carry a stable `target` the client can open plus the typed payload
|
||
and matched field. `list_runs` also returns all-history `usage`
|
||
aggregates. `attention_acks` reads and `ack_attention` adds to the operator's
|
||
shared in-app acknowledgements (dismissed Needs-you entries and chat read
|
||
watermarks; see [ntfy.md](ntfy.md#clearing-notifications-across-devices)); other
|
||
connections receive an `attention_acks_changed` push. `subscribe_notifications`
|
||
(optional `after` cursor) streams the same notifications the server publishes to
|
||
ntfy as `notification` pushes, `clear_notifications` (`ids`) clears them on every
|
||
device, and `notifications_cleared` reports clears; `ntfy_enabled` in the
|
||
notification settings turns the ntfy publish off without stopping the stream (see
|
||
[ntfy.md](ntfy.md#app-notification-stream-no-ntfy-app-needed)).
|
||
|
||
Every run summary (`list_runs` rows, `get_run`, subscribe snapshots and terminal
|
||
`run_event` frames) carries `size_bytes`: bytes on disk for the run (artifact tree
|
||
with turn documents and session exports, server-created Pi session directories,
|
||
plus an estimate of its database rows). It is cached in `pipeline_runs.size_bytes`,
|
||
invalidated by triggers when the run changes, measured on completion, and
|
||
backfilled in the background; 0 means not measured yet.
|
||
|
||
`list_runs` rows (and `get_run`) also carry `usage`, the run's totalled reported
|
||
model usage, in the same shape as the run detail's total: `tokens_total`,
|
||
`stages` and `tokens_known_stages` (the total is partial when fewer stages
|
||
reported tokens than recorded usage), the `tokens_in`/`tokens_out` split with
|
||
`split_known`, the `tokens_reasoning`/`cache_read`/`cache_write` components with
|
||
`cache_known` (true only when every token-reporting stage recorded them;
|
||
`tokens_total` includes cache reads, which dominate for a caching harness), and
|
||
`cost_usd` with `cost_known_stages`/`cost_unknown_stages`. The list computes it
|
||
with one grouped SQL aggregate per batch of 100 runs over the stored per-stage
|
||
usage rows joined to their recorded steps, never a step load or transcript parse.
|
||
It is omitted for a run with no recorded model usage; an older server omits it on
|
||
every row, and clients then show nothing. Subscribe snapshots and `run_event`
|
||
frames do not carry it, so a running run's total refreshes with the next list.
|
||
|
||
`delete_runs` removes finished runs to free space:
|
||
|
||
```json
|
||
{"type":"delete_runs","id":"1","run_ids":[12,13,99]}
|
||
{"type":"runs_deleted","id":"1","runs_deleted":{"deleted":[12],"refused":[{"run_id":13,"reason":"run is running; cancel it or wait for it to finish"},{"run_id":99,"reason":"run not found"}],"freed_bytes":48213}}
|
||
{"type":"runs_removed","run_ids":[12]}
|
||
```
|
||
|
||
Only terminal runs (succeeded, failed, canceled, interrupted) are deleted; active
|
||
runs, unconfirmed cleanup, a running session export, a run an unfinished retry
|
||
depends on, and unknown ids are refused individually. Deletion removes every
|
||
row scoped to the run in one transaction plus `<artifacts>/<run>` and owned Pi
|
||
session directories; user workspaces are never touched. Subscribed connections
|
||
whose scope covers the runs receive the `runs_removed` push. `pipelines runs rm
|
||
<id...>` is the CLI form. See
|
||
`internal/protocol/message.go` for JSON envelopes. Mutating RPCs
|
||
have real effects; they are not previews. TUI-only animation, scrolling, and
|
||
transient client logs are presentation, not separate server capabilities.
|
||
|
||
## Archiving pipelines
|
||
|
||
There is no pipeline deletion: an unused pipeline is archived instead.
|
||
|
||
```sh
|
||
pipelines archive "old flow" --yes # hide it; history is kept
|
||
pipelines list --archived # only archived ones (--all lists both)
|
||
pipelines unarchive "old flow"
|
||
```
|
||
|
||
Archiving sets `archived_at` on the pipeline and nothing else. Its versions,
|
||
runs, artifacts, `inspect`, `runs --pipeline` and search history keep working and
|
||
keep showing its name. While archived:
|
||
|
||
- `list_pipelines` omits it unless the request sets `include_archived`; then the
|
||
summary carries `archived_at` (RFC 3339). The TUI and the app's main list hide it;
|
||
the app's Archived filter shows and unarchives it.
|
||
- `run_pipeline` and `retry_run` are refused: `pipeline "X" is archived; unarchive
|
||
it first`.
|
||
- Its name stays reserved: `new`, `submit` (with or without `--create-only`) and
|
||
import are refused with the same message rather than creating a second pipeline.
|
||
- Archiving is refused while an enabled schedule points at it, naming each
|
||
schedule; pause or repoint them first. Enabling a schedule for an archived
|
||
pipeline is refused too. Refusing is deliberate rather than pausing schedules
|
||
implicitly: the operator sees exactly which timers stop, and unarchiving never
|
||
needs to guess which schedules to turn back on.
|
||
|
||
| Command | Payload | Reply |
|
||
| --- | --- | --- |
|
||
| `archive_pipeline` | `pipeline_id` or `name`, optional `expected_version` | `pipeline_archived` with `detail` |
|
||
| `unarchive_pipeline` | `pipeline_id` or `name`, optional `expected_version` | `pipeline_unarchived` with `detail` |
|
||
|
||
`name` matches exactly first and then as a numeric ID, as in the CLI. A positive
|
||
`expected_version` refuses the change when the latest version differs (the CLI
|
||
sends the version it resolved). Repeating a request already in effect succeeds
|
||
without change. Like schedule saves, these are lifecycle controls and are not
|
||
gated by authoring being disabled.
|
||
|
||
## Saved run schedules
|
||
|
||
Schedules are independent pipeline/version + run-information + timer records,
|
||
not workflow Markdown. See [saved schedules](schedules.md) for the web flow,
|
||
JSON examples, clock semantics and legacy migration.
|
||
|
||
| Command | Payload | Reply |
|
||
| --- | --- | --- |
|
||
| `list_schedules` | optional `pipeline_id` | `schedules` array |
|
||
| `save_schedule` | full `schedule` record, optional creation `request_key` | `schedule` |
|
||
| `delete_schedule` | `schedule_id`, `expected_revision` | `schedule_deleted` |
|
||
|
||
The record contains `name`, `pipeline_id`, positive pinned `version`, `enabled`,
|
||
`input` (nonempty string `prompt`), existing absolute server `workspace`, `rule`,
|
||
and optional `model`, `stage_models`, `notify_mode`, `title`, `output_format`,
|
||
`write_paths` and legacy `allow_tools`. Updates require the current `revision`.
|
||
`next_due`, creation and update timestamps are server-owned. These are execution
|
||
controls and do not enable workflow publication on an execution-only server.
|
||
|
||
Pipeline summaries carry `scheduled`, `schedule_count`, `enabled_schedule_count`.
|
||
Run summaries and live frames carry actual timer provenance: `scheduled`,
|
||
optional `schedule_id` and `schedule_name`. A manual/API run of a scheduled
|
||
pipeline is not automatically scheduled. Historical provenance survives deletion.
|
||
|
||
## Chat sessions
|
||
|
||
Structured tools use `chat_answer` with
|
||
`chat_answer: {session_id, form_id, answers, action?}`. Optional `action` is
|
||
`accept`, `decline` or `cancel` only when advertised by the pending form's
|
||
`actions`; ordinary question replies can omit it. Answers retain JSON types.
|
||
OpenCode cancellation calls the existing form DELETE API. Codex MCP elicitation
|
||
returns the corresponding action/content; other Codex questions accept answers
|
||
only. Permission frames may provide `always_label: "Allow for session"` to avoid
|
||
misrepresenting a session-scoped grant as a permanent policy change.
|
||
|
||
Chat streams may send `chat_event.kind="resync"` to request an authoritative
|
||
history refresh without inventing text deltas or ending a queued prompt. Native
|
||
step markers may include `step.turn_id`; a terminal marker for an older turn
|
||
must not clear a newer turn's Stop control or release its queue slot.
|
||
|
||
Native context compaction is reported as `chat_event.kind="compaction"`, with
|
||
`compaction: {status: "running" | "completed" | "failed", id?: string}`.
|
||
Optional `native_turn_id` scopes the activity to its real harness turn; ignore
|
||
activity for an older turn when a newer native turn is known.
|
||
This is session activity and may have no `message_id`: clients must not invent
|
||
an assistant message or tool call for it. Show **Compacting context…** while
|
||
running; completion clears that phase but does not itself end the model turn.
|
||
A new subscriber receives any retained active compaction. Clear stale activity
|
||
before reconnecting, then preserve fresh replayed activity while history loads.
|
||
This reports native observations only; it does not request or trigger compaction.
|
||
|
||
Usage frames (`chat_event.kind="usage"`) may carry full native snapshots with
|
||
`usage.scope="session"`, `total_tokens`, `context_tokens`, `context_window`,
|
||
`cost_usd`, and `output_tokens_per_second`. Optional fields are unknown when
|
||
absent, not zero; reported zero values are retained explicitly. A new snapshot
|
||
replaces the previous one—never add repeated/replayed totals together.
|
||
`rate_basis="generation"` uses a native generation interval;
|
||
`rate_basis="turn"` denotes an end-to-end average that may include tools/waits.
|
||
Throughput must not be inferred from output text length.
|
||
|
||
Total tokens count processed usage across model calls; they are not the current
|
||
context size. Context usage/window and USD costs are shown only when available
|
||
from the harness. In particular, a subscription without reported USD cost is not
|
||
free, and the browser does not estimate a bill from a price table. Usage frames
|
||
are session metadata: no assistant/tool message is created and no queued prompt
|
||
is admitted or completed by a usage update. Subscriptions replay retained usage;
|
||
native readback refreshes it where supported.
|
||
|
||
`usage.account` is the account's subscription quota as the harness reported it,
|
||
absent when unknown:
|
||
`{"harness","source","plan","as_of","windows":[{"id","label","used_percent","resets_at"}],"unsupported"}`.
|
||
`used_percent` is 0–100 (it can exceed 100); `as_of` and `resets_at` are epoch
|
||
milliseconds. Claude Code fills it from its `rate_limit_event` stream lines
|
||
(5-hour and weekly windows; plan from the initialize `account`), shared by every
|
||
chat on the same account; API-key logins never report one. Codex reads
|
||
`account/rateLimits/read` once per connection (again when a chat opens after
|
||
five minutes) and merges `account/rateLimits/updated` notifications.
|
||
OpenCode has no quota API, so it sends `{"harness":"OpenCode","unsupported":true}`.
|
||
A usage frame with only `account` updates the quota and leaves token figures as
|
||
they were. No quota is estimated and no model call is made to obtain one.
|
||
|
||
The brokered OpenCode chat exposes the operator's own sessions, separate from the
|
||
sessions pipeline runs create. The protocol types are:
|
||
|
||
- `chat_sessions` lists the operator's own sessions only, ordered
|
||
most-recently-used first. Each carries its `directory`, its `tag` and its
|
||
recency. A session with
|
||
no recorded tag is never listed, and no pipeline stage session is ever listed:
|
||
stage sessions are still recorded in `run_steps`/`run_session_owners`
|
||
for the run that owns them. The list is fail-closed: with no registry and no
|
||
store it returns an empty list, never the endpoint's whole session list. A
|
||
session is never marked `busy`: any number of clients may watch and write one.
|
||
Each row carries `activity` (`turn_running`, `background_tasks`, `seq`): the
|
||
server's live view of the session. An older server omits it.
|
||
- `chat_activity` is pushed to every open connection, not only the chat's
|
||
watchers, when a session's turn starts or ends or its count of running
|
||
background tasks changes (shells, background commands, Claude Code background
|
||
Bash and subagents, Codex command executions). The body is `chat_activity`
|
||
(`session_id`, `turn_running`, `background_tasks`, `seq`). It is sent on those
|
||
transitions only, never per output chunk. `seq` increases with every change
|
||
and keeps increasing across a restart, so a client keeps the snapshot (list
|
||
row or push) with the larger `seq`. Background tasks are counted only while
|
||
the server observes the session: after a chat's last viewer leaves, the
|
||
server keeps observing it while its tasks run (bounded like an unattended
|
||
turn: at most 16 such chats and 6 hours). Once released it reports none
|
||
rather than a stale count.
|
||
- `chat_automations` reads provider-owned scheduled work for one operator chat.
|
||
Its body is `chat_query: {session_id}` and the reply is
|
||
`chat_automations: {session_id, provider, status, active_count?, observed_at_ms?, reason?}`.
|
||
`status` is `supported` or `unavailable`; a supported empty result has
|
||
`active_count: 0`, while an unavailable result omits the count (unknown is
|
||
never represented as zero). OpenCode uses its documented
|
||
`GET /api/experimental/schedule?sessionID=...` read and counts only records
|
||
whose native `sessionID` matches and whose status is `active`. The older
|
||
OpenCode serve surface and Codex report their exact readback limitation.
|
||
Claude Code is supported only after its native Stop-hook reports a bounded
|
||
`session_crons` snapshot; `observed_at_ms` is when that last snapshot arrived,
|
||
not a live query, and both one-shot and recurring entries count. Before that,
|
||
or when the host is stale/legacy or metadata is malformed, Claude reports
|
||
`unavailable` rather than guessing from transcript history or CronCreate.
|
||
The request is ownership-checked against the operator session registry,
|
||
bounded and read-only; it never starts or resumes a model turn and does not
|
||
inspect transcripts or count project/global work.
|
||
- `chat_subscribe` attaches a connection to one session's shared live stream (body
|
||
`session_id`) and replays the session's still-pending approval requests, so every
|
||
client that watches a session sees every frame any client's turn produces — the
|
||
other devices' prompts and the turn's streaming output — and can answer a shared
|
||
approval. Attaching is idempotent per connection; the subscription is released
|
||
when the connection ends. The reply carries `chat_queue` (`session_id`,
|
||
`prompts`): the prompts the server still holds for the session, each with
|
||
`operation_id`, `message_id`, `native_message_id`, `text`, `client`,
|
||
`attachments` and `queued_at`, so a reloaded client draws them as queued.
|
||
- `chat_prompt` may come from any client. A prompt whose session already has a turn
|
||
in flight is handed to that turn at once when the harness accepts mid-turn input
|
||
(Claude Code reads it from stdin at its next step; Codex takes it with
|
||
`turn/steer` pinned to the active turn) and is acknowledged with
|
||
`chat_steered: true`. Otherwise
|
||
(OpenCode, or a turn that ended meanwhile) it is written to the session's durable
|
||
server-side queue (bounded, currently 8) and acknowledged with
|
||
`chat_queued: true`; the agent never receives two interleaved turns. Queued
|
||
prompts belong to the session, not the connection: they are delivered in order
|
||
when the turn ends even with no client attached, and after a server restart once
|
||
the harness is idle. A retry with the same `operation_id` is acknowledged again,
|
||
never queued or delivered twice. The prompt is echoed to the other watching
|
||
clients as a `chat_event` of kind `user`, carried with the sender's `client`
|
||
label (a device name), while the sender's own transcript keeps its local copy;
|
||
a queued prompt's user event is published again as it is delivered.
|
||
- `chat_queue` is pushed to every watching client whenever a session's queue
|
||
changes, with the current `prompts` and the change: `delivered`, `removed`, or
|
||
`failed` with `error` (the harness refused it; it is not retried).
|
||
- `chat_unqueue` (body `chat_unqueue`: `session_id`, `operation_id`) removes one
|
||
prompt still waiting in the queue and replies with the updated `chat_queue`. A
|
||
prompt already handed to the harness is refused as already sent. Closing a chat
|
||
drops its queue.
|
||
- `chat_reply` is exactly-once: the first answer to a pending request wins, is
|
||
recorded with the answering client's label and is pushed back to every watching
|
||
client as the same `chat_permission` frame with `resolved: true`, `decision` and
|
||
`resolved_by`, so each card clears and shows the outcome. A later answer is
|
||
refused as already answered (naming the client that won), never silently ignored
|
||
and never delivered to the agent twice.
|
||
- `chat_create` takes a `title` (default "New chat") and a `directory`
|
||
(default `~`, the operator's home; `~`, an omitted or empty value and a `~/...`
|
||
value all resolve to the home directory). A relative or missing directory is
|
||
rejected before any OpenCode call. The created session is recorded with the
|
||
stored tag `operator` and returned as the response's `chat_session`.
|
||
An optional `ephemeral: true` creates the session for automated verification:
|
||
it is created, bootstrapped and usable exactly like any other chat and can still
|
||
be closed and reopened, but its registry row is hidden, so `chat_sessions` never
|
||
lists it and a check that drives the real UI cannot leave a stray "New chat" in
|
||
the operator's list. The flag only removes a chat from the shared list, so it
|
||
cannot reveal or alter another session.
|
||
- `chat_directories` returns the operator's home first, then the directories of
|
||
previous operator sessions most-recently-used first, deduplicated and bounded
|
||
(currently 8). It is a dedicated request so the create dialog can load the
|
||
choices without first listing sessions; `chat_sessions` stays a pure session
|
||
list.
|
||
- `list_directory` backs the app's "Choose folder" picker. The request body is
|
||
`list_directory: {"path": "..."}`; `path` is optional, absolute or starting
|
||
with `~` (`~` or `~/sub`), and defaults to the server user's home. The reply
|
||
is `directory_listing: {"path", "home", "parent", "entries", "truncated"}`:
|
||
`path` is the absolute, cleaned directory (expressed under `home`), `parent` is
|
||
`null` at the home root, and each entry is
|
||
`{"name", "path", "is_dir", "is_git_repo", "hidden", "size", "modified"}` — directories first,
|
||
then files, each group sorted case-insensitively. `size` is file bytes (omitted
|
||
for directories); `modified` is the entry mtime in RFC3339 UTC. Both are omitted
|
||
when entry metadata is unavailable. Listings never include file contents;
|
||
`is_git_repo` marks a directory containing
|
||
`.git`; `hidden` marks a dot-name. Listing is confined to the home subtree:
|
||
the path is cleaned and its symlinks resolved first, so `..`, an absolute path
|
||
outside home, or a symlink that escapes home is refused with an `error` frame
|
||
("... is outside the home directory ..."); a symlink entry that escapes home is
|
||
listed only as a plain name. Kernel pseudo filesystems (proc, sysfs, devfs and
|
||
similar) are refused, permission problems are reported as
|
||
`permission denied: <path>`, and a reply carries at most 2000 entries with
|
||
`truncated: true` when more exist.
|
||
- `server_stats` returns `server_stats_result` with a `server_stats` snapshot of
|
||
host, CPU, memory, local storage, Pipelines data, process and GPU information.
|
||
Sampling is bounded to about three seconds; unavailable sections carry an
|
||
`error` or `reason`. GPU utilization is optional (macOS reports the device
|
||
name only). See [Server stats](notes/server-stats.md).
|
||
- `chat_adopt` takes a `session_id` that already exists on the configured
|
||
endpoint and records it in the operator session registry, using the endpoint's
|
||
own directory and title (never a guess or the server's working directory). The
|
||
response echoes the adopted session as `chat_session` and, like `chat_create`,
|
||
its registry row in `chat_sessions`, so `chat_sessions` then lists it and
|
||
`chat_prompt` can continue it. A nil payload or an invalid/blank id is refused
|
||
before any endpoint call; an id the endpoint does not report is refused naming
|
||
the id and records nothing; an endpoint record with an empty directory is
|
||
refused rather than substituting the home or working directory. The CLI
|
||
equivalent is `pipelines chat-adopt <session-id>` (`--json` for the full
|
||
response; otherwise a short `Adopted <id> · <directory>` line).
|
||
|
||
### Choosing the endpoint
|
||
|
||
By default chat brokers from the single `opencode serve` instance the server
|
||
spawns for itself, unchanged. To broker from the operator's own OpenCode service
|
||
instead, set the opt-in `-opencode-endpoint` server flag or the
|
||
`PIPELINES_OPENCODE_ENDPOINT` environment variable. Precedence is explicit flag,
|
||
then explicit environment, then the endpoint recorded on a previous run; with
|
||
none set the spawned default is used. The reference must be an http(s) URL with a
|
||
host and no credentials, query, fragment or path. It is validated once at
|
||
startup: the backend is detected from the service's own `/api/info` (V2) or
|
||
`/global/health` (native) contract, and a streamable `text/event-stream` event
|
||
path is required. A malformed reference, an unreachable host, a reachable
|
||
non-OpenCode service, or a healthy service with no SSE each disable chat with an
|
||
error naming the endpoint — never a silent fallback to another instance. An
|
||
endpoint that authenticates accepts an optional password through the private file
|
||
`PIPELINES_OPENCODE_ENDPOINT_PASSWORD_FILE` (a regular 0600 file; never an
|
||
inline secret), and `PIPELINES_OPENCODE_ENDPOINT_USERNAME` overrides the detected
|
||
default user. The validated choice is recorded so a restart with no flag or
|
||
environment keeps brokering from the same place; an unreachable recorded endpoint
|
||
disables chat with a clear message and no fallback. The endpoint choice is
|
||
preserved by `pipelines service enable` autostart (the variables, with the
|
||
password file as a reference).
|
||
|
||
The stored tag is exactly `operator`; the display label `Operator` is derived from
|
||
it, so there is no second tag vocabulary. Chat uses the server's one OpenCode
|
||
transport. A session's stream is fanned out to every client watching it, prompts
|
||
are serialised per session (queued behind the in-flight turn rather than
|
||
rejected), and approvals are shared and exactly-once; the retained-state bounds,
|
||
the resync-on-reconnect path, the exclusion of a run's own session and
|
||
per-connection write serialisation are unchanged.
|
||
|
||
The operator-session registry, filter, directory defaulting and the list bound are
|
||
covered by the `pipeline`, `opencode` and `server` unit tests. The live end-to-end
|
||
check is opt-in (`PIPELINES_TEST_OPENCODE_CHAT_LIVE=1`); it runs against a scratch
|
||
store and never touches the deployed server. Android rendering of the tag is out
|
||
of scope for the server change; the wire carries `tag: "operator"`.
|
||
|
||
## Scripted runs
|
||
|
||
Every run records a short, single-line `title` saying what it was for. `pipelines
|
||
run --title "..."` sets it explicitly; otherwise the server generates it from the
|
||
prompt with the configured default harness. It is never empty: when generation is
|
||
unavailable or times out, the server derives the prompt's first line, and
|
||
`title_source` (`model`, `fallback`, or `explicit` for an operator-set title) plus
|
||
`title_model` record how it was produced so a diagnosis can tell them apart.
|
||
`pipelines runs` shows the title beside the pipeline name and version, and
|
||
`pipelines inspect` returns the title, name and version, so two runs of one
|
||
pipeline are distinguishable. Rename after the fact with `pipelines title
|
||
<run-id> "<title>"`; the run's input, steps and execution are unchanged. The
|
||
stored prompt is unchanged, and listing/renaming never calls a model.
|
||
|
||
### Model overrides at launch
|
||
|
||
A pipeline version pins one model per stage. Without republishing, a run can
|
||
choose different models at launch:
|
||
|
||
```sh
|
||
pipelines run "bug fix" "Fix the layout" --json --yes \
|
||
--model openai/gpt-5 \
|
||
--stage-model review=openai/o4-mini
|
||
```
|
||
|
||
- `--model PROVIDER/MODEL` overrides the model for **every** model stage of that
|
||
run.
|
||
- `--stage-model STAGE=PROVIDER/MODEL` overrides **one** stage and is repeatable,
|
||
so any number of stages can be overridden. A repeated key for the same stage is
|
||
last-wins.
|
||
- Precedence: a per-stage override beats the whole-run override, which beats the
|
||
model pinned in the pipeline version.
|
||
|
||
Overrides apply to that run only. The pipeline definition and its immutable
|
||
version are never modified, `show <name> --version N` still returns the pinned
|
||
source and graph, and the run records the `model` and `stage_models` it was
|
||
launched with so it stays reproducible; `retry` reuses them. At launch every
|
||
value must be a `provider/model` pair of the configured form and every stage key
|
||
must exist in the pinned version — an unknown key fails naming the valid keys and
|
||
no run is created. A per-stage key that names a non-model stage is refused
|
||
because that stage never runs a model; `--model` on a pipeline with no model stage
|
||
is accepted and inert. The effective model reaches the stage and its
|
||
repair/formatter path from the same resolved definition, and each step's evidence
|
||
records the provider/model it actually ran with (`inspect <id> --json` exposes
|
||
per-step `provider`/`model`, and the run's graph shows the effective pair). Each
|
||
model step also records the usage the harness reported, carried from the same
|
||
accounting the stage already used for its budget: `usage.tokens_total`, the
|
||
`tokens_in`/`tokens_out` split when the harness reported it (`split_known`),
|
||
`cost_usd` when reported, the `tokens_reasoning`/`cache_read`/`cache_write`
|
||
components when the harness reported the component split (`cache_known`; rows
|
||
recorded before these were kept lack it, so their cache share is unknown), and
|
||
the explicit `tokens_known`/`cost_known`/`split_known` markers. Some subscription
|
||
providers report `cost_known` with no positive `cost_usd`; clients show that cost
|
||
as not reported, never as $0. The run detail totals those stages as `usage`, so a
|
||
mixed-model run shows each stage's model, tokens and cost and the run total. A
|
||
stage whose provider reported no cost shows its tokens with `cost_known: false`
|
||
(never a fabricated or zero cost), and a harness that reports only a total (Pi)
|
||
shows `tokens_total` with the split marked unavailable rather than presenting the
|
||
total as input tokens. The
|
||
CLI is the deliverable; a TUI picker for choosing overrides is a documented
|
||
follow-up, not half-built.
|
||
|
||
The run detail (`inspect <id> --json`, `get_run`) also carries an additive
|
||
top-level `warnings` string list, derived at read time from the steps; `inspect`
|
||
repeats each entry on stderr as `warning: ...`. Today it names every visit whose
|
||
retained file-change capture was `partial` or `unavailable`, because such a
|
||
capture is not a complete workspace diff and disables the server skip fallback
|
||
(see [stage-file-diffs.md](stage-file-diffs.md)). Older peers ignore the field.
|
||
|
||
|
||
On either the runs list or run inspector, press `P` (or use `:pipeline`) to open
|
||
the selected run's exact pipeline version. Navigation does not cancel the run.
|
||
|
||
Initial input includes `pwd`, the resolved run directory (normally the directory
|
||
you invoked Pipelines from), alongside the task and any custom input fields.
|
||
It is recorded by the server and shown in the run's Input section.
|
||
|
||
Press `r` or use `:retry` on a failed run to restart it from the first stage.
|
||
CLI equivalent: `pipelines retry <run-id>` (optional `--yes`, `--json`, `--wait`,
|
||
`--wait-timeout`, and `--request-key`). The new attempt retains the initial input,
|
||
directory, pinned pipeline version, and edit scope, and records `retry_of`. Original
|
||
history is unchanged. Legacy runs gain `pwd` from their recorded workspace on retry.
|
||
Retry replays the entire workflow; earlier edits or notifications are not undone
|
||
and may happen again. A missing original directory blocks retry rather than
|
||
silently using your current directory.
|
||
|
||
Retry is hidden and rejected while a run is executing or agent cleanup remains
|
||
active, even if a terminal status was already persisted. Run RPCs report
|
||
`executing` alongside status so clients can distinguish cleanup from completion.
|
||
|
||
A pipeline may contain a `type=human` decision stage (see the authoring skill's
|
||
`references/format.md`). Its step becomes `waiting`, the run reports that it is
|
||
waiting on a person, and no later stage runs. `runs`/`inspect --json` show the
|
||
`waiting` status, the stage and its recorded instructions and options. Answer it
|
||
with `pipelines decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]`
|
||
or, in the run inspector, press `c` to open the option picker, choose with
|
||
`↑/↓`, type optional feedback with `Tab`, and confirm with Enter. The choice and
|
||
feedback are recorded as the step's evidence and the run resumes along
|
||
`edges[KEY]`. An unknown option is refused with the valid list, a decision to a
|
||
stage that is not waiting is refused, and a repeated decision for an
|
||
already-decided visit does not double-apply. The wait is persisted in the run
|
||
rows, so it survives a server restart and `decide` still works afterwards;
|
||
`cancel` and `retry` keep working while waiting and after a decision, and a
|
||
human stage is never killed by a stage timeout or `max_visits`.
|
||
|
||
The run inspector's upper half shows the entire graph from the run's immutable
|
||
pinned version using the shared renderer and viewport. Executed stages and taken
|
||
routes retain their colors; unexecuted stages, unchosen routes and unreached
|
||
terminals are dimmed. Queued runs show the full graph before the first visit.
|
||
Repeated visits remain individually inspectable in the stage sidebar, while the
|
||
graph highlights every audited route through a loop. On small terminals details
|
||
take precedence. Older servers without a graph fall back to a matching loaded
|
||
definition or the recorded execution path, never a different pipeline/version. Press
|
||
`g` to focus the graph, arrows to pan, and `j/k` or PgUp/PgDn to scroll. Press `g`
|
||
or Tab to return to details. When not focused, the graph follows the latest visit
|
||
or selected stage. Existing polling refreshes the graph once per second, and
|
||
running durations show elapsed time rather than “in progress.”
|
||
|
||
The server can also push run changes. A client sends `subscribe` scoped to a
|
||
workspace (and optionally to explicit run ids); the server first returns the
|
||
current state of the matching runs, then a compact `run_event` delta on each
|
||
transition — which run changed, its status, outcome, current stage, completed-step
|
||
count and timestamps, and, when a single step changed, that step's sequence, stage,
|
||
status, outcome, target and diagnostic. It is a delta, never the whole run, and
|
||
rapid changes are coalesced to at most a few frames per second per run.
|
||
`unsubscribe` or disconnecting stops the frames. This lets a client drop the
|
||
once-per-second poll: the Android app consumes these pushes instead, while the
|
||
TUI still polls in this release.
|
||
|
||
Run-list summaries and subscribe snapshots additionally carry compact
|
||
`graph_evidence: {complete: true, visits: [...]}`. Each visit contains only
|
||
`sequence`, `stage`, `status`, `outcome` and `target`: the last visit of each
|
||
stage and each distinct taken route, ordered by sequence. Repeated loops do not
|
||
grow the summary indefinitely, and private step inputs, outputs, transcripts
|
||
and diagnostics are excluded from this projection. Live `run_event.graph_visits`
|
||
preserves changed visits across coalescing (unlike the single latest `step`).
|
||
Clients merge these by sequence and retain last-stage/distinct-route evidence.
|
||
Missing evidence from an older peer means unknown, not that every stage shares
|
||
the overall run result. These fields are additive; older peers ignore them.
|
||
|
||
In the run inspector, `n`/`p` select the next/previous recorded execution visit,
|
||
including repeated visits in loops, from any pane. Selection stops at the last
|
||
visit or overview; it does not wrap. The selected stage has a full-row highlight;
|
||
the focused stage/details pane has a highlighted title and pointer. In details,
|
||
the section containing the top visible row is highlighted and named in the pane
|
||
title. `Tab` cycles stages → details → graph (Shift+Tab reverses); `g` switches
|
||
graph focus. Each graph stage title shows live progress from its latest execution
|
||
visit. New run pages open with the graph focused and using roughly three quarters
|
||
of the available height. Tab expands the focused pane and contracts the others;
|
||
the stage list also widens when focused, and shrinks to a sidebar for details.
|
||
Pending stages are dimmed. The graph is centered horizontally/vertically
|
||
when it fits; the box-and-arrow diagram is retained at every viewport size,
|
||
never replaced with a stage list to fit. Oversized diagrams remain
|
||
pannable with `h/j/k/l` or arrows while the graph is focused; `c` recenters the
|
||
selected/current stage. `P` opens the
|
||
pinned pipeline (lowercase `p` remains an alias only on the runs list).
|
||
|
||
The pipeline page also gives most height to its focused graph or stage-details
|
||
pane; Tab switches focus and resizes the two panes. Focused titles use the same
|
||
background highlight as the run page.
|
||
Graph boxes are sized to their contents rather than the longest label anywhere
|
||
in the graph. Fork columns share only their own required width, route tracks are
|
||
adjacent, and branch spacing follows stacked boxes rather than tall trunk/merge
|
||
nodes. All stages, outcome labels, ports, and terminal arrows remain visible in
|
||
the full pannable diagram. Incoming route IDs are shown on the left and outgoing
|
||
outcome labels on the right of shared rows, without separate input/output
|
||
sections or divider rows. This reduces height while allowing enough width to
|
||
keep both labels readable. Failure-bound error exits remain at the bottom.
|
||
Pipeline and run graph panes both center horizontally and vertically when the
|
||
diagram fits, and clamp scroll/pan offsets on resize or graph changes.
|
||
|
||
Press `a` to select the active/latest retained configured-harness agent session and
|
||
open its tool evidence. Session ID is available during execution; tool transcripts
|
||
appear when persisted (currently after generation). This does not attach a live
|
||
harness client or stream its messages. Supporting live attachment consistently
|
||
across the shared and privately owned harness backends is deliberately deferred.
|
||
|
||
## Stage turn files
|
||
|
||
Each model-stage visit receives a file-based turn under the run's artifact
|
||
directory. `pipelines inspect <run-id> --json` and the run inspector list these
|
||
files as that step's artifacts:
|
||
|
||
```
|
||
<artifacts>/<run>/stages/01-implementer/input.md
|
||
<artifacts>/<run>/stages/01-implementer/memory.md
|
||
<artifacts>/<run>/stages/01-implementer/history.md
|
||
<artifacts>/<run>/stages/01-implementer/output.md
|
||
<artifacts>/<run>/stages/01-implementer/private/
|
||
<artifacts>/<run>/public/
|
||
```
|
||
|
||
`input.md` is the run prompt on the first visit; later visits carry the run
|
||
prompt under `# Run task` followed by the previous visit's output body (without
|
||
its outcome section) under `# Previous turn output`. `memory.md` is the capped (4 KB) essential memory inlined in every prompt: task,
|
||
previous stage's `# handoff`, outcome ledger and changed files. `history.md` is
|
||
the accumulated markdown of completed turns plus the files each turn added to
|
||
the shared `public/` directory, read on demand. A stage without `read` also
|
||
receives `input.md` in its prompt (capped), because it has no file-reading tool. The model returns the complete
|
||
`output.md` markdown; only its `# outcome` section is interpreted by the server.
|
||
Everything else is free-form and preserved under the step's `markdown` output.
|
||
`private/` belongs to its own visit, while `public/` is shared by every stage in
|
||
the run. Artifacts are served from `/artifacts/<run>/<step>/<name>`; shared files
|
||
appear with a `public/` prefix.
|
||
|
||
### Run notification modes and receipts
|
||
|
||
`--notify auto|on|off` records `notify_mode` on the run (also returned in
|
||
summaries and `run_event` frames). Auto retains stage policies and the
|
||
run-level waiting alert; its failure or success is silent unless the server's
|
||
`run_finished` setting ("Notify when any run ends", off by default) is on. On
|
||
also alerts when the run fails or succeeds, without requiring a stage hook. Off
|
||
suppresses stage, run-level system and browser tray alerts.
|
||
|
||
`inspect --json` / `get_run` includes top-level `notifications` receipts with
|
||
`sequence`, `event`, `channel`, `status`, `attempts`, and optional `diagnostic`
|
||
and `message_id`. Status is pending, sending, sent, failed, or covered (already
|
||
published by the stage). Run-level system alerts persist and retry at most three
|
||
times, recovering pending work after restart. A publisher acknowledgement is not
|
||
proof of reception; a crash before saving it can duplicate a message. See
|
||
[`notes/audit-notifications.md`](notes/audit-notifications.md) for limits.
|
||
|
||
## Investigating runs with a configured harness
|
||
|
||
On the runs list or inspector, press `D` or use `:debug`; CLI equivalent:
|
||
`pipelines debug <run-id>`. This opens a fresh conversation in the configured
|
||
harness (Pi or OpenCode), not the implementing session. It supplies exact
|
||
commands for `inspect <run-id> --json` (including stage history, exact outputs,
|
||
diagnostics, notifications, session IDs and bounded transcript previews),
|
||
`show <pipeline-id> --version N --json`, and `runs --pipeline <pipeline-id>
|
||
--json`. The commands reuse the same Docker-aware CLI prefix as authoring, so a
|
||
Docker-linked profile runs them in-container against the original logical
|
||
endpoint. All retrieval remains available through the CLI and can be repeated
|
||
while execution is active. No new model API or live attachment integration is
|
||
required. The injected server-side session evidence is the latest bounded,
|
||
credential-redacted evidence for up to the 3 most recent session-bearing steps;
|
||
when a preview is bounded or omitted, inspect the retained session itself rather
|
||
than assuming a complete tool history.
|
||
|
||
`debug --json` returns the context and prompt without launching the harness;
|
||
`--no-open` prints the prompt. `--harness opencode|pi|nvim` selects a configured
|
||
harness or nvim; `--opencode` selects the host executable. Debugging
|
||
works for active and terminal runs and does not automatically retry, cancel,
|
||
or resume them. If the recorded directory is unavailable, the harness launches
|
||
from your current directory and retrieves historical evidence through the CLI.
|
||
|
||
## Editor choice and draft cleanup
|
||
|
||
On the pipeline detail page, `e` / `:edit` opens an editor picker: **nvim** or
|
||
the configured harness (Pi/OpenCode, default first). Type to fuzzy-search;
|
||
Tab/Shift+Tab (or arrows) choose; Enter opens;
|
||
Esc cancels without creating a draft. Explicit choice overrides VISUAL/EDITOR.
|
||
Both choices edit a retained copy with the normal version guard, never the
|
||
published definition. Submission remains separate and explicit.
|
||
|
||
In the draft browser, `x` asks to delete the selected draft; `y` confirms and
|
||
Esc or `n` keeps it. The CLI equivalent, `pipelines draft-delete <draft.md>`, is
|
||
an explicit deletion command and needs no server connection; `--json` reports
|
||
the result. It asks for confirmation; `--yes` explicitly approves scripted deletion.
|
||
Only managed drafts inside the configured draft cache are accepted.
|
||
Source and metadata symlinks/directories and external Markdown files are
|
||
rejected. Published definitions and run history are not deleted. Cleanup can
|
||
also remove orphaned metadata when a draft source was already removed.
|
||
|
||
```sh
|
||
cd ~/repos/project
|
||
pipelines run "bug fix" "Fix the layout" --json
|
||
pipelines run "bug fix" "Fix the layout" --json --wait --wait-timeout 10m
|
||
pipelines run "bug fix" "Fix the layout" --json --request-key my-unique-task
|
||
```
|
||
|
||
Runs default to the client's current directory and latest accepted version.
|
||
The server resolves that directory without a repository allowlist. `--version` chooses
|
||
an immutable version in JSON mode. Starting a run enables its stage-declared tools
|
||
without a second approval. Editing supports all file types and new files within
|
||
the run directory; no file list is required. Shell commands use the repository's
|
||
actual tooling. `--write-paths` is an optional narrower scope, not a prerequisite.
|
||
`--yes` skips action confirmation for explicitly approved automation;
|
||
`--allow-tools` remains a compatibility flag and is not required.
|
||
`--wait` polls until terminal status, returns nonzero on failure/cancellation,
|
||
and does not cancel a run when waiting times out. `--wait-timeout` bounds only the
|
||
client wait after admission and is independent of the per-request `--timeout`; a
|
||
reached wait limit leaves the admitted run active and never cancels or replays it.
|
||
When it is reached, inspect read-only with `pipelines inspect <run-id> --json`, or
|
||
keep waiting by rerunning `--wait` and reusing the original `--request-key` rather
|
||
than starting a duplicate run. Reuse `--request-key` to prevent duplicate work.
|
||
A supplied pipeline/task opens an action-confirmation popup before
|
||
starting with a live inspector (unless `--yes` was supplied). When a task is omitted,
|
||
Enter inserts a newline and Ctrl+S opens the confirmation for the composed task.
|
||
There is no file-list question or typed RUN confirmation. Shell runs with your
|
||
user's authority: the run directory and tool policy are not an OS sandbox.
|
||
|
||
Run, retry, cancellation, CLI publication, and draft deletion require confirmation.
|
||
CLI prompts go to stderr and default to No; use `--yes` for non-interactive approval.
|
||
The TUI's run/retry/cancel popup accepts `y`; Enter does not implicitly approve,
|
||
and `n`/Esc cancels without sending the action. Retry restarts with new sessions
|
||
and original input/pwd, never the failed conversation; side effects may repeat.
|
||
Publication retains its explicit submission confirmation and version conflict guards.
|
||
Quit with `:quit`, not `q`; Ctrl+C remains an interrupt.
|
||
|
||
JSON results go to stdout, errors to stderr (JSON error envelopes when requested),
|
||
and failures exit nonzero. Export/template `--source` deliberately prints raw
|
||
Markdown. `--timeout` bounds each server request; `--wait-timeout` bounds the
|
||
optional client wait independently and, when reached, leaves admitted work active.
|
||
Neither changes the stage execution timeout.
|
||
|
||
### Stage limits
|
||
|
||
Model-stage `configs` support these optional settings:
|
||
|
||
```text
|
||
retries=2
|
||
timeout=1h
|
||
budget_usd=1.50
|
||
budget_tokens=50000
|
||
```
|
||
|
||
Retries default to **2** (three total attempts); timeout defaults to **one hour
|
||
per attempt**. Omitted budgets are unlimited; configured budgets must be positive.
|
||
Each budget covers all attempts, format-only repairs, and repeated visits to that
|
||
stage in one run. A new run starts new accounting. Tokens include input, output,
|
||
reasoning, and cache reads/writes. USD amounts use OpenCode's reported cost.
|
||
Missing or invalid required accounting stops the stage rather than counting zero.
|
||
Usage is polled during execution and read again afterward; an in-flight request
|
||
may exceed a limit before usage arrives. These are not provider-side hard caps.
|
||
|
||
Automatic retries cover retryable transport failures and per-attempt timeouts.
|
||
An explicit read-only stage (a tools block listing neither `edit` nor `shell`)
|
||
may also retry a post-admission deadline or transport failure and re-emit a
|
||
malformed final result up to `retries`; a stage whose tools include `edit`/`shell`
|
||
or that has no tools block never replays admitted tool work or invalid results.
|
||
Budget failures always stop. Cleanup must be confirmed before another attempt.
|
||
Existing explicit `retries=0` settings remain unchanged. The old 30-minute run deadline no longer truncates
|
||
stage timeouts; the 30-step and five-visits-per-stage limits still apply.
|
||
Native ntfy stages also accept `timeout` (default `1h`), but never automatically
|
||
retry potentially delivered notifications and do not accept model budgets.
|
||
|
||
Pinned definitions contain the configured limits. Run transcripts retain limits,
|
||
attempt counts and reported budget usage, also shown in the TUI's Evidence pane.
|
||
The same reported usage is recorded on the run as typed per-step evidence and
|
||
totalled for the run, so `inspect <id> --json` exposes tokens in/out, the total,
|
||
the cost when reported and the model each stage ran with. A stage revisited within
|
||
a run shares one budget, so each step records only that visit's increment of it
|
||
and the run total is not the cumulative budget counted again. Missing or unknown cost
|
||
or split is reported as unavailable (never zero or inferred), and a historical or
|
||
asked-for run recorded before this evidence existed simply omits `usage`.
|
||
Exact stage outputs are authoritative; a transcript over the 32 KiB retained
|
||
bound is stored as a valid, credential-redacted preview envelope that preserves
|
||
available stage limits and failure/format diagnostics and names the retained
|
||
session instead of slicing to malformed JSON.
|
||
|
||
### Result-first views
|
||
|
||
Home shows retained result previews and completed visits alongside each run and
|
||
pipeline. Pipeline pages show the latest result for the **selected version**,
|
||
not a result from a different published definition. Run opens focused on results,
|
||
with output before input and the graph below. Failed or ongoing runs preserve
|
||
partial output and completed stage reports without labeling them final success.
|
||
These previews quote recorded output; they do not independently verify model claims.
|
||
Tab and Shift+Tab toggle **Details ↔ Graph** only. The visit selector lives inside
|
||
Details; `n`/`p` still navigate recorded visits, including repeated stages, and `0`
|
||
returns to the overview. `j`/`k` scroll the report; the focused graph uses `hjkl`
|
||
for panning and `c` to center. Secondary progress/input/evidence/execution sections
|
||
are collapsed initially. Small inactive graph previews collapse instead of showing
|
||
severed boxes. See [UI/UX review](ui-ux-review.md) for screen-by-screen feedback.
|
||
|
||
## Run retention
|
||
|
||
Nothing prunes run history automatically. `pipelines retention report` and
|
||
`pipelines retention apply` are the only commands that remove run history or run
|
||
artifacts, they are local and operator-controlled, and they never run as a side
|
||
effect of any other command, server startup, `newRunner`, the scheduler,
|
||
`list_runs`, `stats`, `runs` or `inspect`. Pipeline definitions and their
|
||
immutable versions are never touched: retention is about run history and the
|
||
artifacts runs produced.
|
||
|
||
```sh
|
||
# Read-only: show exactly what a policy would remove.
|
||
pipelines retention report --db ~/.local/share/pipelines/data/pipelines.db --keep 200 --json
|
||
|
||
# Explicit removal of that same set (artifacts go with their run).
|
||
pipelines retention apply --db ~/.local/share/pipelines/data/pipelines.db --keep 200 --yes --json
|
||
```
|
||
|
||
Both commands accept `--older-than DURATION`, `--keep N`, `--db FILE`,
|
||
`--artifacts DIR` and `--json`; `apply` also accepts `--yes`. At least one policy
|
||
flag is required; an unknown subcommand, a negative count and a non-positive age
|
||
are refused. `--db` must name an existing database and is never created. With only
|
||
`--db`, the artifacts root defaults to `artifacts/` beside the database, matching
|
||
`cmd/pipelines-server/main.go`; `--artifacts` overrides it. The commands are
|
||
dispatched before server URL resolution, so they work without a running server
|
||
(for example `--url not-a-server`) and never require, restart or redeploy the
|
||
deployed server.
|
||
|
||
### Policy
|
||
|
||
- `--older-than` selects runs created **strictly before** `now - DURATION`. The
|
||
age is a rolling UTC duration, not a calendar-day boundary. It accepts Go
|
||
durations plus a count of days or weeks (`720h`, `30d`, `12w`).
|
||
- `--keep N` retains the newest `N` runs by run id and selects ranks `N+1` and
|
||
beyond. `--keep 0` selects every eligible run; a count larger than the history
|
||
selects none.
|
||
- Both together are the conservative **intersection**: a run is selected only
|
||
when it is both older than the age bound and outside the newest `N`.
|
||
|
||
### Protected runs
|
||
|
||
A policy-selected run is nonetheless never removed when it:
|
||
|
||
- is `queued` or `running`;
|
||
- is `waiting` on a person (a `waiting` run or a `waiting` step);
|
||
- has any step with `agent_active=1` (cleanup still active); or
|
||
- is referenced by a run that will remain through retry provenance
|
||
(`run_retries.original_run_id`), propagated transitively along a retry chain.
|
||
If both an original and its retry are selected, no retained link remains and
|
||
both may be removed.
|
||
|
||
Protected runs are reported with their reason and their artifacts are never
|
||
counted as reclaimable. Free-text run ids in prompts are not treated as
|
||
dependencies; retry provenance is the only structured cross-run dependency the
|
||
store records.
|
||
|
||
If a selected run becomes protected between the report and the delete — a new
|
||
retry reference, a waiting/active state or new agent activity — `apply` aborts
|
||
with the run ids and reasons, deletes no rows or artifacts, and un-stages
|
||
anything already renamed. A removal never half-completes.
|
||
|
||
### What apply removes and reclaims
|
||
|
||
`apply` stages only the would-remove run directories into one
|
||
`artifacts/.retention-staging-<timestamp>-<pid>/` directory (a same-filesystem
|
||
rename), deletes the rows for those runs in one immediate write transaction
|
||
(`step_notifications`, `step_file_changes`, `step_control`, `step_artifacts`,
|
||
`step_verdicts`, `step_session_exports`, `step_usage`, `run_session_owners`,
|
||
`run_steps`, `run_retries`, then
|
||
`pipeline_runs`, batched at 500 ids), commits, and removes the staging directory.
|
||
A symlink at `<root>/<id>` is removed as a link without touching its target; a
|
||
regular file at `<root>/<id>` is refused. A symlinked artifacts root is followed
|
||
consistently: the report sizes it and apply stages through the same directory, so
|
||
the artifacts really go. All regular files under a removed run directory are
|
||
counted, including files beyond the server's 200-file/8-MiB/64-MiB inventory
|
||
caps, which bound the recorded inventory, not disk ownership.
|
||
|
||
The database is then compacted with one `VACUUM` and one
|
||
`PRAGMA wal_checkpoint(TRUNCATE)`, retried at most three short times on
|
||
`SQLITE_BUSY`. `database_bytes_reclaimed` and `space_reclaimed_bytes` are
|
||
measured, not estimated. If `VACUUM` fails the command returns an error, sets
|
||
`vacuum_error` and claims no reclaimed database pages while still reporting the
|
||
artifact bytes it did remove; a later `apply` with nothing left to remove can
|
||
retry the `VACUUM`. If staging cleanup fails the report still gives the removed
|
||
run count and totals, names the staging path and the remaining bytes/files, and
|
||
does not count the leftovers as reclaimed.
|
||
|
||
### Measured example (this machine)
|
||
|
||
Reported before, applied, reported after, against the live database while its
|
||
server kept running (server PID 99492 unchanged throughout; the server was also
|
||
creating new runs concurrently, which is why the artifact `du` moved
|
||
independently):
|
||
|
||
```text
|
||
report --keep 200: total 207 · would remove 7 · protected 0 · retained 200
|
||
artifacts 418 files · 279668627 bytes (266.7 MiB)
|
||
database 27897856 bytes · free 0 · reclaimable 279668627
|
||
apply --keep 200: removed 7 runs · 418 files · 266.7 MiB
|
||
vacuumed · database reclaimed 1.5 MiB · now 25.1 MiB · free 0
|
||
total space reclaimed 268.2 MiB
|
||
report --keep 200: total 200 · would remove 0 · retained 200
|
||
artifacts 0 files · 0 bytes
|
||
database 26353664 bytes · free 0
|
||
pipelines 19 · pipeline_versions 45 (unchanged)
|
||
```
|
||
|
||
Independent observation: `du -sk` of the artifacts root fell from 2815316 to
|
||
2572456 KiB (the running server added artifacts during the same window), the
|
||
database file from 27897856 to 26353664 bytes, the WAL was truncated to zero and
|
||
the logs stayed at 18404 KiB. A retained run's artifact still served (HTTP 200,
|
||
`/artifacts/100/1/00_run94_artifacts.png`) while a removed run's URL returned 404
|
||
(`/artifacts/83/1/...`). The measured numbers above come from that
|
||
before/apply/after output, not an estimate.
|
||
|
||
## Verification scope
|
||
|
||
CLI integration tests exercise draft/validate/submit/history/conflict handling and
|
||
scripted run/inspection/cancellation through a real local WebSocket server (the
|
||
run test uses a controlled executor, not a model). The isolated installed CLI
|
||
also exercised the actual OpenCode API's session creation and skill activation,
|
||
while capturing interactive launch arguments to avoid a model turn. Full
|
||
model-driven authoring and Linux execution are **not tested**.
|
||
|
||
## Per-run report contract
|
||
|
||
`pipelines run NAME PROMPT --output-format html` requires a server-written
|
||
`public/report.html` before run success; `--output-format markdown` requires
|
||
`public/report.md`. Omit the option for historical behavior. The `run_pipeline`
|
||
wire request and returned run record carry `output_format` (`html` or `markdown`),
|
||
and retries inherit it. Request-key replays must retain the same format.
|
||
Reports use nonempty terminal text output and are bounded to 2 MiB; missing,
|
||
empty, unsafe or unverifiable files fail the run. This checks structure, not
|
||
factual correctness. See [audit notes](notes/audit-deliverables.md).
|
||
|
||
`pipelines repair-session-export RUN_ID SEQUENCE [--json]` asks the runtime to
|
||
read back that settled visit's exact recorded native session/backend. It never
|
||
executes a stage or changes a definition. It writes a versioned repair artifact
|
||
and appends evidence without replacing original files or outcomes. The protocol
|
||
command is `repair_session_export` with `run_id` and `export_sequence`; the reply
|
||
is a run detail containing each visit's `session_export_repairs`. CLI JSON emits
|
||
the repair record with explicit completeness `status`, and incomplete repair
|
||
exits nonzero. Recovery still depends on upstream session availability.
|
||
|
||
Incomplete native stage-session exports also recover automatically after the
|
||
run settles: up to three durable runtime attempts, with backoff and restart-safe
|
||
leases. `inspect RUN_ID --json` includes each visit's `session_export_recovery`
|
||
status/attempt count/diagnostic and preserved `session_export_repairs`. Exhausted
|
||
automatic recovery remains a warning; manual versioned repair is still available.
|
||
A run parked on an export handoff never waits beyond that: once its job is
|
||
exhausted (or the wait exceeds 15 minutes with the job pending or past its
|
||
lease), the visit keeps its partial/unavailable evidence and diagnostic, the run
|
||
continues to its next stage or terminal, and its workspace is released. The
|
||
server applies this at startup and every 30 seconds, so a run parked before an
|
||
upgrade recovers after the next restart. A native session larger than the 8 MiB
|
||
export bound (typically inline image payloads read by a tool) is always partial.
|
||
|
||
### Storage overview and retention (application RPC)
|
||
|
||
The authenticated application uses `get_storage` to read a `storage` reply.
|
||
`storage.total_bytes` sums logical regular-file lengths, deduplicated by inode;
|
||
it is not allocated filesystem space or the size of the whole disk. Categories
|
||
cover database/journals, run artifacts/exports, chat attachments, server-owned
|
||
Pi session files, other data under the database directory, linked worktrees,
|
||
and known project workspaces. Symlink targets are excluded. Unknown external
|
||
harness histories and repositories without a recorded workspace are outside
|
||
this inventory. `measurement` is `logical_file_bytes`; `partial`, category
|
||
errors and warnings explicitly mark incomplete scans. Repository identities
|
||
are remembered so removing run history cannot make sibling worktrees disappear
|
||
from the next measurement.
|
||
|
||
`set_storage_policy` requires `storage_policy`:
|
||
|
||
```json
|
||
{"enabled":false,"max_age_days":30,"max_bytes":10737418240,"include_worktrees":false}
|
||
```
|
||
|
||
Zero disables the corresponding limit. Days must be 0–36500 and bytes 0–2^50.
|
||
Automatic cleanup requires at least one limit. It is disabled by default,
|
||
persisted server-side, and runs hourly while enabled; neither startup nor saving
|
||
a policy performs cleanup. Age expiry and size pressure are independent triggers:
|
||
expired eligible data is selected, then oldest eligible data until the size target
|
||
is met or only protected data remains. Run age is completion time (creation only
|
||
when no completion time exists); worktree age is the latest measured modification.
|
||
Database pages are not vacuumed on the live service, so a size target is best effort.
|
||
|
||
`preview_storage_cleanup` optionally accepts that policy and returns
|
||
`storage_cleanup_preview` with `storage_cleanup`: policy, total/reclaimable bytes,
|
||
target status, candidate/protected counts and items. Each item identifies its kind,
|
||
id, path, measured bytes, reason and protected flag. A preview performs no cleanup.
|
||
The result contains an opaque `preview_token` valid for ten minutes.
|
||
`apply_storage_cleanup` requires the same `storage_policy` and top-level
|
||
`storage_cleanup_token`; it returns `storage_cleanup_applied` with removed count,
|
||
measured reduction and any refusal/errors. Tokens are one-use; expired or changed
|
||
policies require another preview. Apply never adds candidates that appeared after
|
||
the confirmed preview, and revalidates every candidate.
|
||
|
||
Cleanup removes eligible completed run records and their server-owned files via
|
||
the existing transactional deletion path, never definitions or chat history.
|
||
Queued/running/waiting, unconfirmed-cleanup, active/exporting or retry-dependent
|
||
runs remain protected. Worktrees are opt-in and must be verified linked trees,
|
||
clean, unlocked and fully merged into local main. Primary/main checkouts, any
|
||
recorded operator-chat directory, active-run directory, live server-data roots,
|
||
unmerged work, and trees containing another registered checkout are retained.
|
||
Branches are never deleted and Git removal never uses force. New chat/workspace
|
||
admission is serialized with the short final removal gate.
|
||
|
||
Storage work runs beside the WebSocket read loop with one bounded worker; busy
|
||
requests fail promptly instead of blocking chat prompts or permissions. A pass
|
||
has a 20-second context, bounded Git output/time and file traversal, and at most
|
||
100 deletion candidates. Partial scans cannot claim that the storage target has
|
||
been reached. Cleanup refusals and limits remain visible rather than discarding
|
||
protected data to satisfy a quota.
|
||
|
||
|
||
### LAN server discovery
|
||
|
||
`{"type":"discover_servers"}` returns `discovered_servers` with a
|
||
`discovered_servers` array of `{name,endpoint,auth,version}` records. The server
|
||
browses `_pipelines._tcp.local` on its own LAN, with a bounded DNS-SD query; it
|
||
does not scan arbitrary ports. An unavailable browse returns a protocol error.
|
||
The authenticated web front door exposes the same result at `GET /api/discovery`,
|
||
including before the browser has opened its application websocket.
|