36fad7d130
Combines lane-go's typed omni search, launch-time model overrides, TUI artifacts view, run-state websocket push, and the "reflect the change where it belongs" committer rule with the Templates gallery, chat/omni-search and other reviewed work already on main. The 12 files both sides touched keep both intents composed rather than dropping either side: the omni search keeps the typed search contract (CLI `pipelines search` and the TUI popup with type badges and marked snippets) alongside the run-list quick filter; cli.md and ui-ux-review.md document both the search rework and the artifacts view; orchestration.md, harnesses.md and operations.md keep the model-override and reflect-where-it-belongs guidance; and the protocol/CLI/server/TUI code keeps the new search, run-event, override and artifacts paths alongside the existing run, title and help surfaces. Checks before committing: - go build ./... -- pass - go vet ./... -- pass - go test -count=1 ./internal/... ./cmd/... -- all pass Doc check before committing: nothing needs updating. The lane's own docs and references (docs/cli.md, docs/ui-ux-review.md, and the orchestration, harnesses and operations skill references) already carry the new features. format.md and the packaged templates already describe the line-oriented result format and the human decision stage from the earlier lane-go merge. The root README already describes the omni search popup. Model overrides, the search command, run events and the artifacts view are runtime/CLI/TUI surfaces that cli.md documents; no template, README or topic-doc edit is required by this merge.
303 lines
20 KiB
Markdown
303 lines
20 KiB
Markdown
# System, storage and run evidence
|
|
|
|
Pipelines turns a reusable process into sequential stages with explicit handoffs,
|
|
outcome routing and retained evidence. A Go CLI/TUI connects to an execution server;
|
|
the server owns execution and SQLite history, OpenCode owns model/tool sessions,
|
|
and optional ntfy delivery supplies notifications. It is technology-neutral, not a
|
|
parallel/distributed task engine or a guarantee that model reports are correct.
|
|
|
|
## Lifecycle and location
|
|
|
|
- Drafts are editable Markdown plus JSON metadata. Publishing validates and creates
|
|
immutable versions; drafts remain retained. Keep context create-only/base-version
|
|
guards. Four optional generic starters are task, review, research and
|
|
three-stage; a fresh database starts empty. Templates can be created directly with
|
|
an editable prefilled name or customized first. Editing a bundled repository
|
|
template changes no installed or published definition; only explicit authoring
|
|
then submission creates a new immutable version.
|
|
- A run pins its exact definition version, initial input and resolved directory.
|
|
The client invocation directory is the default, but files/tools execute on the
|
|
server. Docker needs that directory mounted at the same absolute path. A remote
|
|
client does not upload its repository or make host credentials available remotely.
|
|
- Host and Docker OpenCode credentials are separate. Preserve setup-selected model
|
|
inheritance; do not select a provider just because its ID appears in an example.
|
|
- Runtime caps are 30 stage steps and five visits to a stage. Runs in different
|
|
resolved workspaces execute concurrently; only a second run in a workspace an
|
|
active run is already changing is refused, naming the conflicting run and
|
|
advising you to wait or use a different workspace (for example another
|
|
checkout). Agent-session quiescence is scoped to the run: a run fails only when
|
|
one of its own sessions cannot be confirmed idle, so one run's active agent never
|
|
fails another's concurrent run. An active session left over by a restart with no
|
|
owning active run is an orphan and still refuses new work until interrupted.
|
|
Reusing that active run's request key reattaches idempotently.
|
|
`type=human` decision stages exist and pause the run (see `references/format.md`);
|
|
there are no parallel branches. Optional worktrees, full backup/restore,
|
|
automatic database sync and independent verifier stage types are future ideas,
|
|
not existing capabilities. Do not invent them in definitions.
|
|
- Services are native or Docker-based. With an existing Docker installation do not
|
|
start a competing native server. Server APIs are unauthenticated: trusted networks
|
|
only; never recommend public exposure. Tool scope is not a universal OS sandbox.
|
|
- Docker setup resolves the current supported OpenCode V2 release from the single
|
|
documented complete `GET .../git/matching-refs/tags/v2` response (a bounded
|
|
complete array with no pagination parameters; never the moving `latest` tag or
|
|
the registry's insertion order) and the current Pi release from npm, resolves the
|
|
selected tag's immutable GHCR `sha256` digest, builds `FROM` that digest, and
|
|
verifies the built image reports exactly those executable versions plus a
|
|
Node-compatible Pi engine.
|
|
It fails closed, keeping the last verified Dockerfile fallback pins, when discovery,
|
|
digest resolution, completeness, build or compatibility cannot be established: never
|
|
a hardcoded fallback, never a silent stale-pin substitution. A full install skips a
|
|
rebuild only when the release receipt **and** the actual built image's release and
|
|
source identity labels already match; a dirty source always invokes the Docker
|
|
build and relies on layer caching. A model/login-only reconfiguration compares only
|
|
the verified release image identity, so a changed source alone does not rebuild.
|
|
`./install --opencode|--pi` reconfigures inside the existing container and
|
|
refuses an image that does not match; it does not rebuild or restart services (an
|
|
explicit ordinary `./install` does when the user asks).
|
|
|
|
## Where data lives
|
|
|
|
SQLite retains pipelines/immutable versions, runs, recorded stage visits,
|
|
inputs/outputs, status/errors, audit transcripts, session ownership/references,
|
|
retry links, schedules/state, notification receipts and file changes. Not everything
|
|
is in that database: drafts/settings are client files; complete OpenCode sessions
|
|
and credentials use OpenCode storage; repository artifacts are files; ntfy has its
|
|
own storage. A pipeline ZIP is not a full backup of these components.
|
|
Never merge or sync live SQLite files using generic file synchronization. Do not
|
|
delete databases, sessions, volumes or credentials to repair a workflow.
|
|
|
|
## Inspect results first
|
|
|
|
Use the authoritative `CLI_PREFIX` from the prompt:
|
|
|
|
- `CLI_PREFIX runs --json`: run history (optional `--pipeline NAME` scope).
|
|
- `CLI_PREFIX search QUERY [--type KIND] [--limit N] --json`: one read-only query
|
|
across pipelines, versions, runs, steps, field text and artifacts. QUERY is free
|
|
text, a bare status word (`queued`/`running`/`succeeded`/`failed`/`canceled`/
|
|
`interrupted`) selecting those runs, or a lone date expression (`today`,
|
|
`yesterday`, `this week` — Monday 00:00 local — or `YYYY-MM-DD`) selecting the
|
|
runs created in that window; anything else stays free text. `--type` is the only
|
|
narrowing mechanism and takes one of `pipeline`, `version`, `run`, `step`,
|
|
`field`, `artifact`; an unknown kind is an error. Ranking is exact name (3) >
|
|
name substring (2) > body text (1), equal tiers by recency, and every hit
|
|
reports `score`, `matched_field` and `recency`. The cap defaults to 50 and clamps
|
|
to 200; every kind scans at most 2000 rows and field scanning stops at 2000
|
|
steps / 1 MiB and reports
|
|
`truncated`/`scanned_steps`/`scanned_bytes`/`cap`/`scan_bound_bytes`. Targets are
|
|
stable: `pipeline:<id>`, `pipeline:<id>:v<version>`, `run:<id>`,
|
|
`run:<id>:<sequence>`, `run:<id>:<sequence>:<field>`, or an artifact URL.
|
|
- `CLI_PREFIX runs --json`: run history (optional `--pipeline NAME` scope). Each run
|
|
carries a bounded, single-line `title` saying what it was for, set explicitly with
|
|
`CLI_PREFIX run ... --title "..."` or generated from its prompt by the shared
|
|
summariser on the configured default harness. It is never empty: when generation is
|
|
unavailable or times out the server derives the prompt's first line, and
|
|
`title_source` (`model`, `fallback`, or `explicit`) plus `title_model` record how it
|
|
was produced, so a diagnosis can tell them apart. Human `runs` and `inspect` show the
|
|
title beside the pipeline name and version, so two runs of one pipeline are
|
|
distinguishable; `CLI_PREFIX title RUN_ID "..."` renames a run after the fact
|
|
without changing its input or steps. The stored prompt is unchanged, and
|
|
listing/renaming never calls a model.
|
|
- `CLI_PREFIX inspect RUN_ID --json`: selected run, visits and retained evidence.
|
|
Exact stage outputs are authoritative; a transcript over the 32 KiB retained
|
|
bound is a valid, credential-redacted preview envelope (with preserved stage
|
|
limits, failure/format diagnostics and the retained session pointer), not the
|
|
complete tool history.
|
|
- `CLI_PREFIX decide RUN_ID [STAGE] --option KEY [--feedback TEXT] [--json]`: answer
|
|
a run that is `waiting` on a `type=human` stage. The status and the stage's
|
|
instructions/options are visible in `runs` and `inspect --json` (and the TUI
|
|
picker); the stage is inferred from the run's current stage unless named. The
|
|
choice and feedback are recorded as the step's evidence (its output and derived
|
|
verdict; the feedback is never an error diagnostic) and the run resumes along
|
|
`edges[KEY]`. An unknown option is
|
|
refused with the valid list; a stage that is not waiting is refused; a repeated
|
|
decision for an already-decided visit does not double-apply. The wait is persisted,
|
|
so `decide` still works after a server restart. `cancel` and `retry` keep working
|
|
while waiting and after a decision; a human stage is never killed by a stage
|
|
timeout or `max_visits` (the shared 30-step run budget still applies).
|
|
- `CLI_PREFIX run ... --json --wait --wait-timeout DURATION`: `--wait` polls until
|
|
terminal and `--wait-timeout` bounds only the client wait after admission,
|
|
independent of per-request `--timeout`. A reached wait limit leaves the admitted run
|
|
active; inspect it read-only with `inspect RUN_ID --json`, or keep waiting by
|
|
rerunning `--wait` and reusing the original `--request-key`. Never start a duplicate
|
|
run, and never expect an automatic replay or cancel of admitted work.
|
|
- `CLI_PREFIX show NAME --version N --json`: exact pinned definition, not whichever
|
|
version happens to be latest now.
|
|
- `CLI_PREFIX status --json`: connection details; candidate addresses are not proof
|
|
they are reachable from another device.
|
|
- `CLI_PREFIX ping --json`: `pong.build.version`, `build.commit` (full Git revision)
|
|
and `build.dirty` identify the compiled **server**, not the client or its runtime
|
|
checkout. A dirty build includes `-dirty`; `commit: "unknown"` or `dirty: null`
|
|
means metadata was unavailable. Older servers omit `build`. Never infer a deployed
|
|
fix from source/skill contents. Native Go builds use embedded VCS metadata;
|
|
Docker setup stamps build arguments. For an explicitly requested Docker rebuild,
|
|
`python3 scripts/build_metadata.py` resolves the current releases and stamps
|
|
`docker compose build` from the checkout with the immutable digest; it does not
|
|
restart/deploy. A plain `docker compose build` without that resolved digest cannot
|
|
build the image.
|
|
Plain `ping` also displays server identity; `CLI_PREFIX version --json` (alias
|
|
`--version`) identifies the client offline. Include both in sanitized bug reports.
|
|
Two dirty builds at the same commit can differ. Docker setup verifies its started
|
|
server identity against the build arguments before declaring setup complete.
|
|
|
|
Pipelines is a single-user first-version preview for supervised work, not a
|
|
multi-user service. Recommend a disposable repository for initial trials. Tool
|
|
permissions and same-path home mounts do not isolate valuable files. Never collect
|
|
raw credential/session databases or publish private inspect output as diagnostics.
|
|
In a source checkout, `python3 scripts/smoke-first-install.py` checks a disposable
|
|
source-built test binaries, draft/publication/transfer and restart persistence without
|
|
inference, notifications or deployment. It is not a live-provider or phone test.
|
|
That is a developer test fixture, not the native installer. End-user installation
|
|
runs natively through `./install` by default (`scripts/setup-native.py`, with
|
|
`--native` equivalent); the Docker server + ntfy installation is explicit with
|
|
`./install --docker`.
|
|
|
|
In the TUI, Home advertises `/` omni search and `:` commands. Home `/` opens one
|
|
popup over pipelines, versions, runs, steps/visits, artifact names and stage
|
|
input/output/evidence text; each row shows a short type badge and a text match
|
|
shows a bracketed snippet, and a bare status word or date selects runs. The same
|
|
query is available as `pipelines search QUERY [--type KIND]`, where `--type`
|
|
narrows the kind. Run Tab switches only
|
|
Details ↔ Graph. `n/p` select recorded visits, including repeated stages; `d`
|
|
focuses/toggles file changes, `A` opens the run's grouped artifacts view (name,
|
|
content type, size and URL per entry, `PgUp`/`PgDn` scroll so a trailing visit
|
|
that recorded none stays reachable, `Enter`/`o` to open the selected URL and
|
|
`y` to copy it), `a` selects the active/latest retained
|
|
configured-harness agent session's bounded, redacted tool evidence, `D`/`:debug`
|
|
opens fresh investigation, and `P` opens the pinned pipeline version. `:quit`
|
|
exits; `q` does not. Results are retained reports, not independently confirmed
|
|
achievements. Distinguish partial output from full success.
|
|
|
|
## Retained stage file changes
|
|
|
|
Each newly executed visit compares a bounded, read-only workspace snapshot just
|
|
before execution with one after execution/cleanup. SQLite retains unified patches
|
|
and metadata in `steps[].file_changes`; snapshots themselves are transient memory.
|
|
The stage's File changes section highlights added/removed lines and hunk headers.
|
|
|
|
- The baseline is stage-start content, **not HEAD/index**. Pre-existing edits and
|
|
earlier stages' changes are not attributed to this visit. Added/deleted/modified
|
|
files, modes and binary-change metadata are recorded; renames are delete/add.
|
|
- Git scope is tracked and nonignored untracked files within the run directory;
|
|
outside Git, regular workspace files. Symlinks/special files, Git internals,
|
|
node_modules and common environment/key/credential/database paths are excluded.
|
|
This is path-based protection, **not secret-content scanning**. Patches may still
|
|
contain sensitive text; never publish them without approval.
|
|
- Limits: 10,000 entries, 1 MiB/file, 32 MiB/snapshot, five-second snapshot context,
|
|
5,000 combined text lines, 64 KiB/patch, 128 KiB serialized file evidence/visit.
|
|
Missing/skipped/truncated captures must be described as partial/unavailable.
|
|
- Failed/canceled stages retain changes when execution returns; unconfirmed agent
|
|
cleanup is partial. Crashes may have no after-capture. Historical visits say
|
|
not recorded—**never infer no edits from absent evidence**.
|
|
- These are observed workspace changes, not exclusive authorship. Concurrent edits,
|
|
changes outside scope, ignored new files and changes reverted before capture can
|
|
be absent or mixed in. Capture failure does not change the declared run verdict.
|
|
|
|
Do not add model output fields, new stages or extra model calls merely to obtain
|
|
file diffs: current capture is server-owned. Do not promise full files/rollback.
|
|
|
|
## Retained stage artifacts
|
|
|
|
Every agent visit gets a per-run/step artifacts directory under the data
|
|
directory (`<data>/artifacts/<run>/<step>/`); the root is configurable and
|
|
defaults next to the database. The directory is passed as
|
|
`PIPELINES_ARTIFACTS_DIR` to a locally spawned Pi process and exported to the
|
|
OpenCode CLI where controllable, and it is stated in the agent prompt next to
|
|
`Run directory:` for every harness. OpenCode V2 tools run inside the shared serve
|
|
process, so there the prompt line, not the environment, is the reliable carrier.
|
|
The instruction is general: save images, screenshots, reports and logs there.
|
|
|
|
After the visit the server makes one bounded inventory and records each file's
|
|
name, size, content type (extension, sniff fallback) and a relative url; these
|
|
appear under each step's `artifacts` in `inspect RUN_ID --json` and the TUI's
|
|
`A` artifacts view, which groups them by stage and visit, marks images and shows
|
|
the recorded count, capture-pending/not-recorded wording and partial-capture
|
|
notes.
|
|
`GET /artifacts/<run>/<step>/<name>` serves the bytes (correct Content-Type and
|
|
Content-Length, HEAD, and Range/206/416 via `http.ServeContent`), refusing
|
|
traversal, absolute paths, symlink escapes and unknown runs/steps.
|
|
|
|
- Limits: 200 files, 8 MiB/file, 64 MiB total per visit; over-cap files are
|
|
skipped and counted; symlinks and non-regular files are excluded.
|
|
- Artifacts are read-only evidence, not a gate: absent, unreadable, symlinked or
|
|
oversized directories record nothing and never change the run verdict. The
|
|
database keeps the recorded list, not file copies.
|
|
- Do not invent artifact files or claim a stage produced one without the recorded
|
|
evidence; the server observes the directory after the stage.
|
|
|
|
## Brokered OpenCode chat (unshipped)
|
|
|
|
The protocol declares the brokered `chat_*` surface (`chat_sessions`,
|
|
`chat_create`, `chat_prompt`, `chat_messages`, `chat_interrupt`, `chat_permission`,
|
|
`chat_reply`). The server dispatches `chat_sessions`, `chat_prompt`,
|
|
`chat_messages`, `chat_interrupt` and `chat_reply`, answers the declared
|
|
`chat_create` with an explicit protocol error because it is not yet brokered, and
|
|
pushes `chat_event` frames derived from OpenCode's global `/api/event` firehose,
|
|
with retained per-session/per-message state so a reconnecting client re-syncs
|
|
instead of replaying deltas.
|
|
|
|
`chat_sessions` is brokered but restricted: OpenCode's `GET /api/session` lists
|
|
every session on the machine, so the server filters it to the sessions recorded
|
|
in `run_steps` as owned by this server (deduped by session id), fills any missing
|
|
directory from the owning run's workspace, and fails closed to an empty list when
|
|
no ownership source is configured rather than exposing the operator's whole
|
|
environment. A session another client currently holds (an in-server claim,
|
|
released when the stream ends or the client disconnects) or that this server
|
|
records as an active run is marked `busy` so the phone can show it read-only; a
|
|
second client's `chat_prompt` for a held session is rejected before any OpenCode
|
|
call. `chat_create` remains unbrokered.
|
|
|
|
While a `chat_prompt` subscription is open the server also polls
|
|
`GET /api/permission/request` on a fixed small interval (first poll immediately) and
|
|
pushes one `chat_permission` frame per newly pending approval request, deduped by
|
|
request id so a repeated poll never re-pushes the same request. The app answers with
|
|
`chat_reply`, which the server forwards as
|
|
`POST /api/session/{id}/permission/{requestID}/reply` with
|
|
`{"decision":"once"|"always"|"reject"}`. A missing session/request id or an invalid
|
|
decision is rejected before any OpenCode call, and the poller stops when the stream
|
|
ends or the client disconnects.
|
|
|
|
The permission list and reply are implemented against the V2 HTTP API only: their
|
|
field names match the service's `Permission.Request`/`Permission.Reply` schemas
|
|
(verified against the installed service's `/openapi.json`), and the native v1 serve
|
|
translation rejects them explicitly instead of guessing an endpoint. Because this
|
|
build enables brokered chat only when the OpenCode connection exposes a URL-bearing
|
|
SSE endpoint (the owned v1 serve process), the approval flow is not reachable on the
|
|
backend that currently enables chat; on that backend the permission poller stops
|
|
quietly rather than reporting the documented limitation as a per-prompt error. This is
|
|
an internal, unshipped capability: no client consumes it, and on a V2 CLI
|
|
(`opencode api`, no owned serve) that endpoint is absent, so every dispatched `chat_*`
|
|
request returns a protocol error naming the missing configuration. The live
|
|
end-to-end approval flow has NOT been tested against a real OpenCode service; the wire
|
|
shapes and the exact reply path/body are covered by unit tests over an httptest
|
|
fixture, not against a live pending request. Do not describe brokered chat as a
|
|
finished or user-facing feature.
|
|
|
|
## Debug and retry
|
|
|
|
Debug opens a new conversation with retained failure context, exact definition and
|
|
session evidence; it does not resume implementation or grant permission to edit.
|
|
Inspect first, identify evidence-supported causes, and say what remains unknown.
|
|
Server-side session evidence is the latest bounded, credential-redacted evidence
|
|
for up to the 3 most recent session-bearing steps; OpenCode V2 selects the latest
|
|
assistant message with the same bounded readback as the final result, so a small
|
|
final result is visible behind large prior tool content. Inspect the retained
|
|
session itself when a preview is bounded or omitted; never assume a complete tool
|
|
history from a preview.
|
|
Retry is allowed for a failed run after cleanup stops: a fresh run/session starts
|
|
at entry with the original input/directory/pinned version/edit scope. Previous
|
|
side effects remain and can happen again. Never describe retry or cancellation as
|
|
rollback, or resume/replay admitted tool work just to recover formatting.
|
|
|
|
Retained model-stage failures carry a fixed sanitized category. `FINAL RESULT
|
|
VALIDATION` names the exact duplicate-field/syntax detail and byte offset for a
|
|
strictly invalid final result; `HARNESS TRANSPORT` covers request, active-session
|
|
poll and final-readback failures. A validation failure never re-prompts the
|
|
admitted tool-using stage: completed edits and tool side effects are retained and
|
|
not undone, so a full retry repeats work. An explicit read-only stage is the one
|
|
exception: a malformed result there mutates nothing, so it may be re-emitted up
|
|
to `retries` with its `format_repairs` evidence retained, and a post-admission
|
|
deadline or transport failure may be retried. A stage whose tools include
|
|
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
|
|
bundle), is never replayed. Neither category exposes provider/model bodies or
|
|
credentials.
|