Files
pipelines/docs/operator-reference.md
alan 5090f1cf4e Pipelines skill: outcome-first output, aligned review and loop guidance, operator material moved to docs
Rewrite format.md around turn files, the output.md order (# outcome, edge word, # handoff, detail), checks, configs, human stages and the land stage delivery gate (sh -c in a fresh detached worktree, clean checkouts incl. untracked, 64 KB output cap, opencode only). Align review guidance across practices, verification, orchestration and the feedback-loop example. Fix loop-limit semantics in control.md against runner.go, budget_tokens sizing, read for large inputs, blocked-else-error and one stage naming set. Move dated verification logs to docs/verification-history.md and brokered chat, TUI keys and harness internals to docs/operator-reference.md; label operator-only references in SKILL.md.
2026-11-04 17:27:01 +00:00

292 lines
19 KiB
Markdown

# Operator reference
Operator and developer material moved out of the packaged Pipelines skill
(`internal/author/skill/references/operations.md`) because authoring agents
never need it.
## TUI keys
In the TUI, Home advertises `/` omni search and `:` commands. Home `/` opens one
popup over pipelines, versions, runs, steps/visits, artifact names and stage
input/output/evidence text; each row shows a short type badge and a text match
shows a bracketed snippet, and a bare status word or date selects runs. The same
query is available as `pipelines search QUERY [--type KIND]`, where `--type`
narrows the kind. Run Tab switches only
Details ↔ Graph. `n/p` select recorded visits, including repeated stages; `d`
focuses/toggles file changes, `A` opens the run's grouped artifacts view (name,
content type, size and URL per entry, `PgUp`/`PgDn` scroll so a trailing visit
that recorded none stays reachable, `Enter`/`o` to open the selected URL and
`y` to copy it), `a` selects the active/latest retained
configured-harness agent session's bounded, redacted tool evidence, `D`/`:debug`
opens fresh investigation, and `P` opens the pinned pipeline version. `:quit`
exits; `q` does not. Results are retained reports, not independently confirmed
achievements. Distinguish partial output from full success.
## Brokered OpenCode chat
The protocol declares the brokered `chat_*` surface (`chat_sessions`,
`chat_create`, `chat_directories`, `chat_prompt`, `chat_messages`,
`chat_interrupt`, `chat_permission`, `chat_reply`). The server dispatches all of
them, and pushes `chat_event` frames derived from OpenCode's global SSE firehose
(`/event` on the native serve, `/api/event` on the V2 API), with retained
per-session/per-message state so a reconnecting client re-syncs instead of
replaying deltas.
At startup `cmd/pipelines-server` calls `opencode.OpenChat`, which detects the
installed backend and opens exactly one owned `opencode serve` endpoint for it:
the native wire for v1, the V2 API for the installed V2 CLI. Chat is enabled only
when that connection carries a URL-bearing, SSE-capable event endpoint; otherwise
chat stays disabled and every `chat_*` request is answered with a protocol error
naming the missing configuration rather than hanging. Stage generation still
reaches V2 through the `opencode api` CLI, so the owned serve process is chat's
single transport and starts no process for stages. The adapter skips firehose
events it does not surface before decoding their data, so an unrelated event
whose `delta` has a different shape (`session.instructions.updated` carries an
object) never fails the stream.
The operator can instead broker chat from their own OpenCode service with the
opt-in `-opencode-endpoint` flag or `PIPELINES_OPENCODE_ENDPOINT`, with precedence
flag > environment > the endpoint recorded on a previous run and the spawned
instance as the default. `opencode.OpenEndpoint` validates the reference once
(the established http(s)/host/no-credentials/no-query/no-path idiom), detects the
backend from the service's own `/api/info` (V2) or `/global/health` (native)
contract and requires a streamable `text/event-stream` event path; it returns a
connection with **no owned process** and never calls `startServe`. A malformed
reference, an unreachable host, a reachable non-OpenCode service or a
health-ok-but-no-SSE service each disable chat with an error naming the endpoint
and never fall back to the spawned instance. An authenticating endpoint accepts an
optional password only through the private file
`PIPELINES_OPENCODE_ENDPOINT_PASSWORD_FILE` (regular, 0600, single-owner), and
`PIPELINES_OPENCODE_ENDPOINT_USERNAME` overrides the detected default user. The
validated choice is recorded in the additive `server_settings` table so a restart
with no flag/environment reuses it; an unreachable recorded endpoint disables chat
with a clear message and no fallback. `pipelines service enable` preserves the
endpoint and the password-file reference in autostart.
`chat_sessions` lists the operator's own chat sessions, not the sessions pipeline
runs create. Operator sessions are recorded in the dedicated `operator_sessions`
table (`session_id` primary key, plus `directory`, `title`, `tag`, `created_at`,
`last_used_at`), separate from `run_steps`/`run_session_owners`, so an operator
chat is a first-class thing and a pipeline stage session — recorded as run
ownership, never in the operator registry — is never listed. The handler reads the
registry once (bounded to 128 rows, most-recently-used first), calls OpenCode's
session list once, and keeps only registry rows the endpoint still reports, so a
session with no recorded tag is never listed and a deleted upstream session does
not linger. It fills each session's `directory`, `tag` and `created_at`/
`updated_at` from the registry and marks `busy` when another client currently
holds the session (an in-server claim, released when the stream ends or the client
disconnects); a second client's `chat_prompt` for a held session is rejected
before any OpenCode call. It is fail-closed: with no registry configured it
returns an empty list, never the endpoint's whole session list. The OpenCode
session list is read unscoped by directory (the V2 `GET /api/session` otherwise
defaults to the caller's directory), because an operator session may live in any
directory.
`chat_create` creates one operator session and records it in the same registry
with the stored tag `operator` (the display label `Operator` is derived from it,
so there is no second tag vocabulary). It takes a `title` (default "New chat") and
a `directory` (default `~`): an omitted or empty value, `~`, and any `~/...` value
all resolve to `os.UserHomeDir`, never the server's working directory; a relative
path or a missing directory is rejected before any OpenCode call, and a session
created but not recordable is reported as an error naming its id. Create posts the
verified V2 shape `POST /api/session` `{"title":...,"location":{"directory":...}}`.
`chat_directories` returns the operator's home first, then the directories of
previous operator sessions most-recently-used first, deduplicated and bounded (8);
it is a dedicated request so the create dialog can load the choices without first
listing sessions and `chat_sessions` stays a pure, fail-closed session list. A
successful `chat_prompt` touches the session's `last_used_at` in one UPDATE, so it
moves to the front of the list; a non-operator session id touches no row.
`chat_adopt` records a session that already exists on the configured endpoint in
the same operator registry, using the endpoint's own directory and title
verbatim (never a guess, the server's working directory or the home directory).
It reads the endpoint once through the shared `Chat.Session` (which reuses
`Sessions`/`decodeChatSessions`), records through the existing
`RecordOperatorSession` with the `operator` tag, and echoes the adopted session
plus its registry row like `chat_create`, so `chat_sessions` lists it and
`chat_prompt` continues it. A nil payload, a blank or foreign-shaped id, a nil
broker or a nil registry is refused before any endpoint call; an id the endpoint
does not report is refused naming the id and records nothing; an endpoint record
with an empty directory is refused rather than substituting a directory. The CLI
equivalent is `pipelines chat-adopt <session-id>` (`--json` prints the full
response, otherwise a short `Adopted <id> · <directory>` line).
While a `chat_prompt` subscription is open the server also polls
`GET /api/permission/request` on a fixed small interval (first poll immediately) and
pushes one `chat_permission` frame per newly pending approval request, deduped by
request id so a repeated poll never re-pushes the same request. The app answers with
`chat_reply`, which the server forwards as
`POST /api/session/{id}/permission/{requestID}/reply` with
`{"decision":"once"|"always"|"reject"}`. A missing session/request id or an invalid
decision is rejected before any OpenCode call, and the poller stops when the stream
ends or the client disconnects.
The permission list and reply are implemented against the V2 HTTP API only: their
field names match the service's `Permission.Request`/`Permission.Reply` schemas
(verified against the installed service's `/openapi.json`). The native v1 serve
translation rejects them explicitly instead of guessing an endpoint, so on v1 the
permission poller stops quietly rather than reporting the documented limitation as
a per-prompt error.
### Verification evidence (2026-10-17)
Verified against the installed OpenCode 2.0.20 on 2026-10-17:
- `opencode --version` reports `2.0.20`; `opencode --help` advertises `api` and
`serve`. `GET /openapi.json` on an owned `opencode serve` lists `GET /api/event`
(SSE) and the session/prompt/interrupt/permission endpoints.
- A live turn streamed token-level `session.text.delta` and
`session.reasoning.delta` events (each with `assistantMessageID` and `ordinal`),
plus `session.step.*` and `session.usage.updated`; the V2 API server
authenticates the fixed `opencode` basic-auth user.
- The opt-in live end-to-end test (`PIPELINES_TEST_OPENCODE_CHAT_LIVE=1 go test
./internal/server -run TestChatLiveEndToEnd -v`) opened an owned V2 endpoint,
created an operator session through `chat_create` in a chosen directory, listed
exactly that session through `chat_sessions` (with its directory and tag
`operator`), recorded a genuine pipeline stage session in
`run_steps`/`run_session_owners` and showed it never appeared in the list, and
streamed a completed turn as `[step:started,text,step:streamed,step:ended]`
(first delta `"pong"`). The operator-session registry, the filter, directory
defaulting and the recents bound are also covered by the `pipeline`, `opencode`
and `server` unit tests, which run without the live gate.
- Not verified live: a genuine pending-approval round trip and any backend other
than the installed 2.0.20. The permission wire shapes are unit-tested over an
httptest fixture, not against a live pending request.
- Operator-supplied endpoint selection and `chat_adopt` (added 2026-10-17):
`opencode.OpenEndpoint` and the flag/env/recorded precedence, the URL-shape
validation failures (scheme, empty host, userinfo, query, fragment, non-empty
path), the unreachable/non-OpenCode/no-SSE/auth-mismatch failures, the
no-owned-process property (`conn.process == nil`) and the password-file rule
are unit-tested over httptest fixtures in `internal/opencode/endpoint_test.go`.
`chat_adopt` (endpoint directory/title recorded with the `operator` tag, echo,
missing id refused naming the id and recording nothing, nil payload/blank id/nil
broker/nil registry refused before any endpoint call, empty endpoint directory
refused) is unit-tested over the server fake broker in
`internal/server/chat_test.go`; the recorded-endpoint round-trip across a store
Close/Open is in `internal/pipeline/settings_test.go`; the protocol round-trip
and the CLI dispatch (`pipelines chat-adopt`) are covered in
`internal/protocol/chat_test.go` and `internal/cli/chat_adopt_test.go`. The
live scratch-server check against the operator's real endpoint (adopt, list,
prompt) was **not** run in this change; only the unit fixtures above were
executed.
Deployment state: not deployed. This is reviewed and committed locally only;
deploying it (and restarting the server) is the operator's step. Do not describe
brokered chat as already deployed.
Decide a waiting human stage in the TUI: open the run, press `c`, pick an option,
optionally type feedback, Enter.
## Docker release resolution
Docker setup resolves the current supported OpenCode V2 release from the single
documented complete `GET .../git/matching-refs/tags/v2` response (a bounded
complete array with no pagination parameters; never the moving `latest` tag or
the registry's insertion order) and the current Pi release from npm, resolves the
selected tag's immutable GHCR `sha256` digest, builds `FROM` that digest, and
verifies the built image reports exactly those executable versions plus a
Node-compatible Pi engine.
It fails closed, keeping the last verified Dockerfile fallback pins, when discovery,
digest resolution, completeness, build or compatibility cannot be established: never
a hardcoded fallback, never a silent stale-pin substitution. A full install skips a
rebuild only when the release receipt **and** the actual built image's release and
source identity labels already match; a dirty source always invokes the Docker
build and relies on layer caching. A model/login-only reconfiguration compares only
the verified release image identity, so a changed source alone does not rebuild.
`pipelines harness setup opencode|pi` reconfigures inside the existing container and
refuses an image that does not match; it does not rebuild or restart services (an
explicit ordinary `./install` does when the user asks).
## Harness internals
CLI `new/edit/debug` and TUI author/debug build one shared, Docker-aware CLI
prefix. A Docker-linked profile embeds the in-container `/usr/local/bin/pipelines`
(never the host binary: Linux can see but cannot execute a mounted macOS host
executable). Loopback is recognized only via `net.ParseIP(...).IsLoopback()` plus
an exact case-insensitive `localhost`, so a remote DNS name such as
`127.remote.example` stays remote and runs no probe. For a loopback logical
`--url`, a single bounded `docker compose port` probe parses each returned line
as a complete `host:port` binding (IPv4/IPv6/wildcard) and the prefix maps only
when a binding covers the requested loopback address and port; malformed or
non-matching bindings fail closed. A loopback `wss` endpoint is rejected as
having no supported local TLS mapping rather than downgraded to plaintext. On a
match it embeds scoped `PIPELINES_TRANSPORT_SOURCE`/`PIPELINES_TRANSPORT_TARGET`
aliases in the prefix string (not `cmd.Env`, so the pre-existing harness
server's tools inherit them). The client rewrites an endpoint only on an exact
source match; remote endpoints stay remote. Draft metadata (server, base
version, create-only guard) is preserved unchanged.
`provider-check` with a Docker-linked profile runs inside that profile's existing
Compose service with `-T`, the original cwd/model/timeout and exact stdout/exit
propagation; container profiles have no `DockerDirectory` and cannot recurse, and
explicit native profiles keep the host path. The picker uses a true `x/term` TTY
check, so `/dev/null` and pipes fail fast instead of hanging.
On an existing Docker setup, `pipelines harness setup opencode|pi` configures inside its container and link the host
launcher without restarting/deploying services. They resolve and verify the current
OpenCode V2 release from the single documented complete
`GET .../git/matching-refs/tags/v2` response (a bounded complete array with no
pagination parameters; never the moving `latest` tag or the registry's insertion
order) and the current Pi release from npm, resolve the selected tag's immutable
GHCR `sha256` digest, and require a rebuild only when the verified release image
identity no longer matches. A full install always invokes the Docker build for a
dirty source (relying on layer caching); a model/login-only reconfiguration does
not rebuild. Setup writes its generated release receipt to the machine-local root
file `.pipelines-releases.json` (with atomic `.pipelines-releases-*` temporary
files); both are git-ignored, so a fresh clean checkout stays clean and reinstall
still reuses the verified release/image identity. The actual built image's labels
stay authoritative and a dirty source still rebuilds. If discovery, digest
resolution or completeness cannot be established they stop with an error (keeping
the last verified Dockerfile fallback pins); they never guess, fall back to a host
runtime or install a stale pin.
(`./install --opencode`/`--pi` were removed and only point to that command.) The native installation reuses the host OpenCode
configuration, writes **host** (non-Docker) profiles for the harnesses it detects
into XDG config home's `pipelines/harnesses.json`, and clears a stale Docker link
so a Docker-era profile is not left Docker-linked. It never installs, upgrades or
downloads a harness. Container and host credentials and accounts remain independent and
are reused on reinstall/reconfigure. `./install` installs natively by default
(`--native` is equivalent); explicit `./install --docker` selects the Docker
server + ntfy installation.
Fresh setup accepts `./install --docker --harness pi` to skip OpenCode
login. Existing setup can configure either harness independently. Pi setup does
not run a generation check; `provider-check --harness pi --model PROVIDER/MODEL`
is a separate confirmed, potentially chargeable check.
Profiles live in XDG config home's `pipelines/harnesses.json`, store preferences
only, and preserve unrelated configuration. A profile launches a native host
harness unless it records a Docker checkout directory, in which case it launches
through that installation's Compose service. `harness setup <opencode|pi> --host`
configures or converts a host profile and clears any Docker link; without
`--host`, reconfiguring a Docker-linked profile re-verifies that the recorded
directory still has `compose.yaml` and fails naming it rather than saving a link
that cannot launch. Setup's summary and `harness list` report the resulting mode
(and, for Docker, the directory). Harness CLI setup/default commands
always configure the local execution user, not a remote `--url` server.
OpenCode interactive launch uses
`mini --model`: verified V2 full CLI lacks a model flag. Pi uses its CLI with the
packaged skill explicitly loaded. Pi has no standalone `auth login` command
(verified `pi auth --help`: only credential check/print): setup prints a
title-plus-three-lines Pi sign-in handoff and opens a fresh, no-context,
extension-free, no-prompt-templates, no-tools session with Pi's documented
`--tui-mode regular` flag, so the header and handoff stay visible above Pi's
interface at 80x30 instead of the default fullscreen clear. The user types
`/login`, chooses an authentication method ("Sign in with an API key" for
provider API credentials; account and Radius remain the user's choice), finishes
provider sign-in, then exits with `ctrl+c`/`ctrl+d` to return to Pipelines model
configuration. Existing sign-ins are reused by exiting when no new sign-in is
needed; completing a sign-in can update the credentials stored in the Docker
volume. No coding/task prompt or scripted keystrokes are injected.
OpenCode V2 model stages settle long sessions by polling `GET /api/session/active`
about once per second under the stage context, instead of the `.../wait` request
whose CLI-side request timeout failed long stages after roughly six minutes
(HTTP 499, dated 2026-10-10). V1 native settlement is unchanged. Final results
and debug session evidence use the same single bounded latest-assistant readback
(`order=desc&limit=1&type=assistant`), so a small final result is visible behind
large prior tool content; retained transcripts over 32 KiB are stored as valid
bounded preview envelopes that preserve stage limits and diagnostics, and exact
outputs remain authoritative. Pi evidence is JSONL and is not parsed as one JSON
object. Short-lived context-controlled CLI calls (`--help` detection, API calls
and the native export fallback) share one finite inherited-pipe drain grace
(`os/exec.Cmd.WaitDelay` ~250ms) so a canceled context or a descendant that still
holds stdout/stderr cannot block for the child's natural lifetime; the
`context.Canceled`/`DeadlineExceeded` and `exec.ErrWaitDelay` identities are
preserved and partial stdout after the grace is never accepted as success.