Files
pipelines/internal/author/skill/references/operations.md
T
alan 18398dfa5b chat: give the operator their own sessions, separate from run sessions
chat_sessions used to list exactly the OpenCode sessions the pipelines own,
filtered through pipeline.SessionOwner (run_steps/run_session_owners), so an
operator-created chat was an accident of the run-ownership table and the
operator's working sessions were mixed with the internals of running pipelines.

Add a first-class operator session:

- internal/pipeline/operator_sessions.go: a dedicated operator_sessions table
  (session_id PK, directory, title, tag, created_at, last_used_at) with
  RecordOperatorSession (upsert preserving created_at), a one-statement
  TouchOperatorSession (no-op for a non-operator id), OperatorSessions bounded to
  128 rows most-recently-used first, and OperatorDirectories (bounded MRU,
  deduped). run_steps/run_session_owners and SessionOwner/OwnedSessions are
  untouched.
- internal/server/chat.go: chat_sessions returns operator sessions only - it
  reads the registry once, intersects it with one broker session-list call, fills
  directory/tag/created/updated from the registry, sets busy for a session held
  by another connection, keeps MRU order, and stays fail-closed (empty list, never
  the endpoint's whole list). A pipeline stage session and an untagged row can
  never be listed. chat_create resolves the directory (empty/'~'/'~/...' ->
  operator home; relative or missing rejected before any OpenCode call), calls
  broker.Create(title, resolved) and records the row with the operator tag.
  chat_directories returns home first then bounded (8) deduped recents. A
  successful prompt touches the operator session without affecting pipeline
  prompts.
- internal/opencode, internal/protocol, internal/server/websocket.go and
  message.go: unscoped session list, Create posts {"title":...,
  "location":{"directory":...}} to POST /api/session, additive protocol fields
  (omitempty) and the ChatDirectories request/response, plus the OperatorRegistry
  option defaulting to the first store. One OpenCode transport per server is
  unchanged.

Docs and references: docs/cli.md gains a "Chat sessions" section documenting the
operator-only fail-closed list, directory defaulting/rejection, chat_directories
(home first, deduped, bounded 8) and why a dedicated request was chosen.
internal/author/skill/references/operations.md's stale "Brokered OpenCode chat"
text (chat_create unbrokered, run_steps filtering, "listed the owned scratch
session" evidence) is rewritten to the operator-session registry, the fail-closed
filter, directory defaulting and the corrected live-test wording. No new
reference file, so internal/author/install.go's inventory is unchanged.

Checked for staleness elsewhere: README.md, the docs/ topic pages and the
packaged server/TUI templates carry no chat-session ownership claims, so none
needed updating.

Verified: gofmt -l clean; go vet and go build ./... clean; go test -count=1
./internal/pipeline/... ./internal/protocol/... ./internal/opencode/...
./internal/server/... and the full go test -count=1 ./... pass.
2026-10-17 11:14:25 +01:00

357 lines
24 KiB
Markdown

# System, storage and run evidence
Pipelines turns a reusable process into sequential stages with explicit handoffs,
outcome routing and retained evidence. A Go CLI/TUI connects to an execution server;
the server owns execution and SQLite history, OpenCode owns model/tool sessions,
and optional ntfy delivery supplies notifications. It is technology-neutral, not a
parallel/distributed task engine or a guarantee that model reports are correct.
## Lifecycle and location
- Drafts are editable Markdown plus JSON metadata. Publishing validates and creates
immutable versions; drafts remain retained. Keep context create-only/base-version
guards. Four optional generic starters are task, review, research and
three-stage; a fresh database starts empty. Templates can be created directly with
an editable prefilled name or customized first. Editing a bundled repository
template changes no installed or published definition; only explicit authoring
then submission creates a new immutable version.
- A run pins its exact definition version, initial input and resolved directory.
The client invocation directory is the default, but files/tools execute on the
server. Docker needs that directory mounted at the same absolute path. A remote
client does not upload its repository or make host credentials available remotely.
- Host and Docker OpenCode credentials are separate. Preserve setup-selected model
inheritance; do not select a provider just because its ID appears in an example.
- Runtime caps are 30 stage steps and five visits to a stage. Runs in different
resolved workspaces execute concurrently; only a second run in a workspace an
active run is already changing is refused, naming the conflicting run and
advising you to wait or use a different workspace (for example another
checkout). Agent-session quiescence is scoped to the run: a run fails only when
one of its own sessions cannot be confirmed idle, so one run's active agent never
fails another's concurrent run. An active session left over by a restart with no
owning active run is an orphan and still refuses new work until interrupted.
Reusing that active run's request key reattaches idempotently.
`type=human` decision stages exist and pause the run (see `references/format.md`);
there are no parallel branches. Optional worktrees, full backup/restore,
automatic database sync and independent verifier stage types are future ideas,
not existing capabilities. Do not invent them in definitions.
- Services are native or Docker-based. With an existing Docker installation do not
start a competing native server. Server APIs are unauthenticated: trusted networks
only; never recommend public exposure. Tool scope is not a universal OS sandbox.
- Docker setup resolves the current supported OpenCode V2 release from the single
documented complete `GET .../git/matching-refs/tags/v2` response (a bounded
complete array with no pagination parameters; never the moving `latest` tag or
the registry's insertion order) and the current Pi release from npm, resolves the
selected tag's immutable GHCR `sha256` digest, builds `FROM` that digest, and
verifies the built image reports exactly those executable versions plus a
Node-compatible Pi engine.
It fails closed, keeping the last verified Dockerfile fallback pins, when discovery,
digest resolution, completeness, build or compatibility cannot be established: never
a hardcoded fallback, never a silent stale-pin substitution. A full install skips a
rebuild only when the release receipt **and** the actual built image's release and
source identity labels already match; a dirty source always invokes the Docker
build and relies on layer caching. A model/login-only reconfiguration compares only
the verified release image identity, so a changed source alone does not rebuild.
`./install --opencode|--pi` reconfigures inside the existing container and
refuses an image that does not match; it does not rebuild or restart services (an
explicit ordinary `./install` does when the user asks).
## Where data lives
SQLite retains pipelines/immutable versions, runs, recorded stage visits,
inputs/outputs, status/errors, audit transcripts, session ownership/references,
retry links, schedules/state, notification receipts and file changes. Not everything
is in that database: drafts/settings are client files; complete OpenCode sessions
and credentials use OpenCode storage; repository artifacts are files; ntfy has its
own storage. A pipeline ZIP is not a full backup of these components.
Never merge or sync live SQLite files using generic file synchronization. Do not
delete databases, sessions, volumes or credentials to repair a workflow.
## Inspect results first
Use the authoritative `CLI_PREFIX` from the prompt:
- `CLI_PREFIX runs --json`: run history (optional `--pipeline NAME` scope).
- `CLI_PREFIX search QUERY [--type KIND] [--limit N] --json`: one read-only query
across pipelines, versions, runs, steps, field text and artifacts. QUERY is free
text, a bare status word (`queued`/`running`/`succeeded`/`failed`/`canceled`/
`interrupted`) selecting those runs, or a lone date expression (`today`,
`yesterday`, `this week` — Monday 00:00 local — or `YYYY-MM-DD`) selecting the
runs created in that window; anything else stays free text. `--type` is the only
narrowing mechanism and takes one of `pipeline`, `version`, `run`, `step`,
`field`, `artifact`; an unknown kind is an error. Ranking is exact name (3) >
name substring (2) > body text (1), equal tiers by recency, and every hit
reports `score`, `matched_field` and `recency`. The cap defaults to 50 and clamps
to 200; every kind scans at most 2000 rows and field scanning stops at 2000
steps / 1 MiB and reports
`truncated`/`scanned_steps`/`scanned_bytes`/`cap`/`scan_bound_bytes`. Targets are
stable: `pipeline:<id>`, `pipeline:<id>:v<version>`, `run:<id>`,
`run:<id>:<sequence>`, `run:<id>:<sequence>:<field>`, or an artifact URL.
- `CLI_PREFIX runs --json`: run history (optional `--pipeline NAME` scope). Each run
carries a bounded, single-line `title` saying what it was for, set explicitly with
`CLI_PREFIX run ... --title "..."` or generated from its prompt by the shared
summariser on the configured default harness. It is never empty: when generation is
unavailable or times out the server derives the prompt's first line, and
`title_source` (`model`, `fallback`, or `explicit`) plus `title_model` record how it
was produced, so a diagnosis can tell them apart. Human `runs` and `inspect` show the
title beside the pipeline name and version, so two runs of one pipeline are
distinguishable; `CLI_PREFIX title RUN_ID "..."` renames a run after the fact
without changing its input or steps. The stored prompt is unchanged, and
listing/renaming never calls a model.
- `CLI_PREFIX inspect RUN_ID --json`: selected run, visits and retained evidence.
Exact stage outputs are authoritative; a transcript over the 32 KiB retained
bound is a valid, credential-redacted preview envelope (with preserved stage
limits, failure/format diagnostics and the retained session pointer), not the
complete tool history.
- `CLI_PREFIX decide RUN_ID [STAGE] --option KEY [--feedback TEXT] [--json]`: answer
a run that is `waiting` on a `type=human` stage. The status and the stage's
instructions/options are visible in `runs` and `inspect --json` (and the TUI
picker); the stage is inferred from the run's current stage unless named. The
choice and feedback are recorded as the step's evidence (its output and derived
verdict; the feedback is never an error diagnostic) and the run resumes along
`edges[KEY]`. An unknown option is
refused with the valid list; a stage that is not waiting is refused; a repeated
decision for an already-decided visit does not double-apply. The wait is persisted,
so `decide` still works after a server restart. `cancel` and `retry` keep working
while waiting and after a decision; a human stage is never killed by a stage
timeout or `max_visits` (the shared 30-step run budget still applies).
- `CLI_PREFIX run ... --json --wait --wait-timeout DURATION`: `--wait` polls until
terminal and `--wait-timeout` bounds only the client wait after admission,
independent of per-request `--timeout`. A reached wait limit leaves the admitted run
active; inspect it read-only with `inspect RUN_ID --json`, or keep waiting by
rerunning `--wait` and reusing the original `--request-key`. Never start a duplicate
run, and never expect an automatic replay or cancel of admitted work.
- `CLI_PREFIX show NAME --version N --json`: exact pinned definition, not whichever
version happens to be latest now.
- `CLI_PREFIX status --json`: connection details; candidate addresses are not proof
they are reachable from another device.
- `CLI_PREFIX ping --json`: `pong.build.version`, `build.commit` (full Git revision)
and `build.dirty` identify the compiled **server**, not the client or its runtime
checkout. A dirty build includes `-dirty`; `commit: "unknown"` or `dirty: null`
means metadata was unavailable. Older servers omit `build`. Never infer a deployed
fix from source/skill contents. Native Go builds use embedded VCS metadata;
Docker setup stamps build arguments. For an explicitly requested Docker rebuild,
`python3 scripts/build_metadata.py` resolves the current releases and stamps
`docker compose build` from the checkout with the immutable digest; it does not
restart/deploy. A plain `docker compose build` without that resolved digest cannot
build the image.
Plain `ping` also displays server identity; `CLI_PREFIX version --json` (alias
`--version`) identifies the client offline. Include both in sanitized bug reports.
Two dirty builds at the same commit can differ. Docker setup verifies its started
server identity against the build arguments before declaring setup complete.
Pipelines is a single-user first-version preview for supervised work, not a
multi-user service. Recommend a disposable repository for initial trials. Tool
permissions and same-path home mounts do not isolate valuable files. Never collect
raw credential/session databases or publish private inspect output as diagnostics.
In a source checkout, `python3 scripts/smoke-first-install.py` checks a disposable
source-built test binaries, draft/publication/transfer and restart persistence without
inference, notifications or deployment. It is not a live-provider or phone test.
That is a developer test fixture, not the native installer. End-user installation
runs natively through `./install` by default (`scripts/setup-native.py`, with
`--native` equivalent); the Docker server + ntfy installation is explicit with
`./install --docker`.
In the TUI, Home advertises `/` omni search and `:` commands. Home `/` opens one
popup over pipelines, versions, runs, steps/visits, artifact names and stage
input/output/evidence text; each row shows a short type badge and a text match
shows a bracketed snippet, and a bare status word or date selects runs. The same
query is available as `pipelines search QUERY [--type KIND]`, where `--type`
narrows the kind. Run Tab switches only
Details ↔ Graph. `n/p` select recorded visits, including repeated stages; `d`
focuses/toggles file changes, `A` opens the run's grouped artifacts view (name,
content type, size and URL per entry, `PgUp`/`PgDn` scroll so a trailing visit
that recorded none stays reachable, `Enter`/`o` to open the selected URL and
`y` to copy it), `a` selects the active/latest retained
configured-harness agent session's bounded, redacted tool evidence, `D`/`:debug`
opens fresh investigation, and `P` opens the pinned pipeline version. `:quit`
exits; `q` does not. Results are retained reports, not independently confirmed
achievements. Distinguish partial output from full success.
## Retained stage file changes
Each newly executed visit compares a bounded, read-only workspace snapshot just
before execution with one after execution/cleanup. SQLite retains unified patches
and metadata in `steps[].file_changes`; snapshots themselves are transient memory.
The stage's File changes section highlights added/removed lines and hunk headers.
- The baseline is stage-start content, **not HEAD/index**. Pre-existing edits and
earlier stages' changes are not attributed to this visit. Added/deleted/modified
files, modes and binary-change metadata are recorded; renames are delete/add.
- Git scope is tracked and nonignored untracked files within the run directory;
outside Git, regular workspace files. Symlinks/special files, Git internals,
node_modules and common environment/key/credential/database paths are excluded.
This is path-based protection, **not secret-content scanning**. Patches may still
contain sensitive text; never publish them without approval.
- Limits: 10,000 entries, 1 MiB/file, 32 MiB/snapshot, five-second snapshot context,
5,000 combined text lines, 64 KiB/patch, 128 KiB serialized file evidence/visit.
Missing/skipped/truncated captures must be described as partial/unavailable.
- Failed/canceled stages retain changes when execution returns; unconfirmed agent
cleanup is partial. Crashes may have no after-capture. Historical visits say
not recorded—**never infer no edits from absent evidence**.
- These are observed workspace changes, not exclusive authorship. Concurrent edits,
changes outside scope, ignored new files and changes reverted before capture can
be absent or mixed in. Capture failure does not change the declared run verdict.
Do not add model output fields, new stages or extra model calls merely to obtain
file diffs: current capture is server-owned. Do not promise full files/rollback.
## Retained stage artifacts
Every agent visit gets a per-run/step artifacts directory under the data
directory (`<data>/artifacts/<run>/<step>/`); the root is configurable and
defaults next to the database. The directory is passed as
`PIPELINES_ARTIFACTS_DIR` to a locally spawned Pi process and exported to the
OpenCode CLI where controllable, and it is stated in the agent prompt next to
`Run directory:` for every harness. OpenCode V2 tools run inside the shared serve
process, so there the prompt line, not the environment, is the reliable carrier.
The instruction is general: save images, screenshots, reports and logs there.
After the visit the server makes one bounded inventory and records each file's
name, size, content type (extension, sniff fallback) and a relative url; these
appear under each step's `artifacts` in `inspect RUN_ID --json` and the TUI's
`A` artifacts view, which groups them by stage and visit, marks images and shows
the recorded count, capture-pending/not-recorded wording and partial-capture
notes.
`GET /artifacts/<run>/<step>/<name>` serves the bytes (correct Content-Type and
Content-Length, HEAD, and Range/206/416 via `http.ServeContent`), refusing
traversal, absolute paths, symlink escapes and unknown runs/steps.
- Limits: 200 files, 8 MiB/file, 64 MiB total per visit; over-cap files are
skipped and counted; symlinks and non-regular files are excluded.
- Artifacts are read-only evidence, not a gate: absent, unreadable, symlinked or
oversized directories record nothing and never change the run verdict. The
database keeps the recorded list, not file copies.
- Do not invent artifact files or claim a stage produced one without the recorded
evidence; the server observes the directory after the stage.
## Brokered OpenCode chat
The protocol declares the brokered `chat_*` surface (`chat_sessions`,
`chat_create`, `chat_directories`, `chat_prompt`, `chat_messages`,
`chat_interrupt`, `chat_permission`, `chat_reply`). The server dispatches all of
them, and pushes `chat_event` frames derived from OpenCode's global SSE firehose
(`/event` on the native serve, `/api/event` on the V2 API), with retained
per-session/per-message state so a reconnecting client re-syncs instead of
replaying deltas.
At startup `cmd/pipelines-server` calls `opencode.OpenChat`, which detects the
installed backend and opens exactly one owned `opencode serve` endpoint for it:
the native wire for v1, the V2 API for the installed V2 CLI. Chat is enabled only
when that connection carries a URL-bearing, SSE-capable event endpoint; otherwise
chat stays disabled and every `chat_*` request is answered with a protocol error
naming the missing configuration rather than hanging. Stage generation still
reaches V2 through the `opencode api` CLI, so the owned serve process is chat's
single transport and starts no process for stages. The adapter skips firehose
events it does not surface before decoding their data, so an unrelated event
whose `delta` has a different shape (`session.instructions.updated` carries an
object) never fails the stream.
`chat_sessions` lists the operator's own chat sessions, not the sessions pipeline
runs create. Operator sessions are recorded in the dedicated `operator_sessions`
table (`session_id` primary key, plus `directory`, `title`, `tag`, `created_at`,
`last_used_at`), separate from `run_steps`/`run_session_owners`, so an operator
chat is a first-class thing and a pipeline stage session — recorded as run
ownership, never in the operator registry — is never listed. The handler reads the
registry once (bounded to 128 rows, most-recently-used first), calls OpenCode's
session list once, and keeps only registry rows the endpoint still reports, so a
session with no recorded tag is never listed and a deleted upstream session does
not linger. It fills each session's `directory`, `tag` and `created_at`/
`updated_at` from the registry and marks `busy` when another client currently
holds the session (an in-server claim, released when the stream ends or the client
disconnects); a second client's `chat_prompt` for a held session is rejected
before any OpenCode call. It is fail-closed: with no registry configured it
returns an empty list, never the endpoint's whole session list. The OpenCode
session list is read unscoped by directory (the V2 `GET /api/session` otherwise
defaults to the caller's directory), because an operator session may live in any
directory.
`chat_create` creates one operator session and records it in the same registry
with the stored tag `operator` (the display label `Operator` is derived from it,
so there is no second tag vocabulary). It takes a `title` (default "New chat") and
a `directory` (default `~`): an omitted or empty value, `~`, and any `~/...` value
all resolve to `os.UserHomeDir`, never the server's working directory; a relative
path or a missing directory is rejected before any OpenCode call, and a session
created but not recordable is reported as an error naming its id. Create posts the
verified V2 shape `POST /api/session` `{"title":...,"location":{"directory":...}}`.
`chat_directories` returns the operator's home first, then the directories of
previous operator sessions most-recently-used first, deduplicated and bounded (8);
it is a dedicated request so the create dialog can load the choices without first
listing sessions and `chat_sessions` stays a pure, fail-closed session list. A
successful `chat_prompt` touches the session's `last_used_at` in one UPDATE, so it
moves to the front of the list; a non-operator session id touches no row.
While a `chat_prompt` subscription is open the server also polls
`GET /api/permission/request` on a fixed small interval (first poll immediately) and
pushes one `chat_permission` frame per newly pending approval request, deduped by
request id so a repeated poll never re-pushes the same request. The app answers with
`chat_reply`, which the server forwards as
`POST /api/session/{id}/permission/{requestID}/reply` with
`{"decision":"once"|"always"|"reject"}`. A missing session/request id or an invalid
decision is rejected before any OpenCode call, and the poller stops when the stream
ends or the client disconnects.
The permission list and reply are implemented against the V2 HTTP API only: their
field names match the service's `Permission.Request`/`Permission.Reply` schemas
(verified against the installed service's `/openapi.json`). The native v1 serve
translation rejects them explicitly instead of guessing an endpoint, so on v1 the
permission poller stops quietly rather than reporting the documented limitation as
a per-prompt error.
### Verification evidence (2026-10-17)
Verified against the installed OpenCode 2.0.20 on 2026-10-17:
- `opencode --version` reports `2.0.20`; `opencode --help` advertises `api` and
`serve`. `GET /openapi.json` on an owned `opencode serve` lists `GET /api/event`
(SSE) and the session/prompt/interrupt/permission endpoints.
- A live turn streamed token-level `session.text.delta` and
`session.reasoning.delta` events (each with `assistantMessageID` and `ordinal`),
plus `session.step.*` and `session.usage.updated`; the V2 API server
authenticates the fixed `opencode` basic-auth user.
- The opt-in live end-to-end test (`PIPELINES_TEST_OPENCODE_CHAT_LIVE=1 go test
./internal/server -run TestChatLiveEndToEnd -v`) opened an owned V2 endpoint,
created an operator session through `chat_create` in a chosen directory, listed
exactly that session through `chat_sessions` (with its directory and tag
`operator`), recorded a genuine pipeline stage session in
`run_steps`/`run_session_owners` and showed it never appeared in the list, and
streamed a completed turn as `[step:started,text,step:streamed,step:ended]`
(first delta `"pong"`). The operator-session registry, the filter, directory
defaulting and the recents bound are also covered by the `pipeline`, `opencode`
and `server` unit tests, which run without the live gate.
- Not verified live: a genuine pending-approval round trip and any backend other
than the installed 2.0.20. The permission wire shapes are unit-tested over an
httptest fixture, not against a live pending request.
Deployment state: not deployed. This is reviewed and committed locally only;
deploying it (and restarting the server) is the operator's step. Do not describe
brokered chat as already deployed.
## Debug and retry
Debug opens a new conversation with retained failure context, exact definition and
session evidence; it does not resume implementation or grant permission to edit.
Inspect first, identify evidence-supported causes, and say what remains unknown.
Server-side session evidence is the latest bounded, credential-redacted evidence
for up to the 3 most recent session-bearing steps; OpenCode V2 selects the latest
assistant message with the same bounded readback as the final result, so a small
final result is visible behind large prior tool content. Inspect the retained
session itself when a preview is bounded or omitted; never assume a complete tool
history from a preview.
Retry is allowed for a failed run after cleanup stops: a fresh run/session starts
at entry with the original input/directory/pinned version/edit scope. Previous
side effects remain and can happen again. Never describe retry or cancellation as
rollback, or resume/replay admitted tool work just to recover formatting.
Retained model-stage failures carry a fixed sanitized category. `FINAL RESULT
VALIDATION` names the exact duplicate-field/syntax detail and byte offset for a
strictly invalid final result; `HARNESS TRANSPORT` covers request, active-session
poll and final-readback failures. A validation failure never re-prompts the
admitted tool-using stage: completed edits and tool side effects are retained and
not undone, so a full retry repeats work. An explicit read-only stage is the one
exception: a malformed result there mutates nothing, so it may be re-emitted up
to `retries` with its `format_repairs` evidence retained, and a post-admission
deadline or transport failure may be retried. A stage whose tools include
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
bundle), is never replayed. Neither category exposes provider/model bodies or
credentials.