Commit Graph

239 Commits

Author SHA1 Message Date
alan 648c9c33d8 Merge main into chat-persist: combine OpenCode pairing chat persistence with main's bottom selector strip
Brings main (78725e9, "bottom-strip": the Android detail and visit selectors
moved to a bottom strip, with its docs/notes/bottom-selector-strip.md) into this
branch on top of the chat-persistence commit. The two sides touch disjoint files
- main changes android/ and its own note, this branch changes
cmd/pipelines-server, internal/opencode, internal/protocol, internal/server and
docs/notes/chat-persistence.md - so the merge applied with no conflict and no
resolution was needed. Repository checks after the merge: go build ./...,
go vet ./... and gofmt -l . clean; go test ./internal/... ./cmd/... green.
2026-10-18 00:55:48 +01:00
alan 0ba1204895 chat: attach to the operator's own OpenCode by pairing, fall back honestly
Brokered chat accepted an operator-supplied endpoint (-opencode-endpoint /
PIPELINES_OPENCODE_ENDPOINT) but failed closed against the operator's running
V2 service: that API authenticates every request with HTTP Basic auth (fixed
user "opencode" plus a password), the broker had no credential, so the health
probe answered 401 and startup logged "OpenCode endpoint ... is unreachable or
is not an OpenCode service", disabling chat entirely.

Established from the installed binary (opencode v2.0.20), not from docs:
`opencode --help` advertises `api`, `pair`, `serve`, `service` and `auth`;
`GET /api/info` is unauthenticated-401 with www-authenticate Basic realm
"Secure Area"; Basic auth user `opencode` with the password from
~/.local/state/opencode/service.json answers 200; `POST /api/pair` returns a
code and `GET /auth/connect/{code}` returns a token that is then accepted as
the Basic password.

- internal/opencode/pair.go (new): PairEndpoint does the pair exchange over the
  shared nativeRequest transport (same auth, bounds and JSON checks);
  ReadServiceCredential reads the operator's local service password under the
  private-file rule; OpenEndpointAttached resolves the credential in precedence
  order - explicit password file, a paired token stored by an earlier run, then
  a fresh pairing bootstrapped by the local service credential - stores the
  token owner-only and re-pairs once when a stored token is refused (401).
- internal/opencode/endpoint.go: OpenEndpoint wraps probeEndpoint, which returns
  the connection even on failure; Connection.unauthorized (set on HTTP 401 by
  the request paths, cleared per probe) distinguishes an authentication refusal
  from any other refusal.
- cmd/pipelines-server/main.go: the endpoint attach uses OpenEndpointAttached
  with the paired-credential file beside the pipeline database. An endpoint that
  cannot be attached logs a line naming it and falls back to the server's own
  OpenCode instance instead of leaving chat refused.
- internal/protocol/status.go, internal/server/websocket.go: get_status gains an
  opencode object (health = endpoint | fallback | spawned | unavailable, plus
  the endpoint url), the honest status evidence for the fallback. No credential
  is ever placed there.
- Tests: internal/opencode/pair_test.go, internal/protocol/status_test.go and a
  TestStatusReportsOpenCodeAttachment case in internal/server/status_test.go.

Repository checks: go build ./..., go vet ./... and gofmt -l . clean;
go test ./internal/... ./cmd/... green.

Packaged templates and skill references: the repository has no template tree
that mentions the OpenCode endpoint, and no packaged reference file is added, so
internal/author/install.go's managed inventory is unchanged. The packaged
authoring reference internal/author/skill/references/operations.md (the endpoint
paragraph around line 289, the "clear refusal and no fallback" wording and the
password-file/username-only credential story, and the missing opencode status
field) is now stale; per the maintenance convention the item does not edit the
shared document and instead records the stale points, with the exact wording
they should take, in docs/notes/chat-persistence.md for the docs-fold writer.
2026-10-18 00:55:26 +01:00
alan 49f40bfe45 android: move the detail and visit selectors to a bottom strip on narrow windows
The two selectors that were a horizontally scrolling Material 3 tab row at
the top of a surface become one deliberate treatment. On a narrow window the
shared detail selector (`DetailTabs`) and the visit selector are now the shared
bottom strip (`ui/components/SelectorStrip.kt`), pinned in the thumb zone above
the system gesture inset with the content filling the area above; a wide window
keeps the left rail for the detail selector and the pinned top row for visits.

- SelectorStrip: the one bottom strip plus its pure model (`SelectorEntry`,
  `SelectorPlacement`, `detailSelectorPlacement`, `visitSelectorPlacement`,
  `selectorStripShown`, `detailSelectorEntries`, `visitSelectorKey/Label`,
  `visitMarker`, `visitSelectorEntries`). Each entry keeps at least the platform
  minimum touch target, shows its whole label, and a visit shows its recorded
  status/outcome as a small `RunStatusIcon` marker rather than an invented one.
  Selection is carried by tab semantics and container colour, not colour alone.
  A single entry (or none) draws no strip.
- `DetailTabs` narrow branch replaced by `SelectorStrip` after the panel in a
  Column; the wide rail is unchanged.
- `StageDetailPane`: `StageVisitStrip` is the narrow visit selector; the bodies
  put a `weight(1f)` scroll above it and keep the top row only on a wide window.
  The stage definition section still shows no strip.
- `SessionExport`: `SessionExportSection` gains `visitTabs` so the narrow pane
  does not render the visit selector twice.
- `PipelinesApp`: the strip publishes its measured height through the new
  `LocalBottomStripInset`/`BottomStripInset`, and the floating pill's overlay
  lifts by `pillBottomOffset(stripHeight) = SPACING_L + stripHeight` (pure, zero
  when no strip is up), so the pill never overlaps the strip.
- Tests: new `SelectorStripTest` (placements, single-entry no strip, detail
  entries keep tab keys/titles/icons, visit labels and markers, stable visit
  keys) and a new `PipelinesNavTest` case pinning the pill lift.

Pill/strip clearance is the brief's "pill moves clear of the strip" option,
implemented once in `pillBottomOffset`. All geometry and motion come from the
existing named tokens and the layout; no new dependency.

Documentation checked: no packaged OpenCode skill reference (`internal/author/
skill/references/`) or packaged template (`examples/*.md`) documents the app's
detail/visit selector presentation, so neither is stale and `internal/author/
install.go`'s managed inventory needs no change. The shared `android/README.md`
is made stale by this change (the narrow tab-row wording at lines ~182-184,
~233-235 and ~487-488) and is recorded for the single `docs-fold` writer in the
new `docs/notes/bottom-selector-strip.md` rather than edited here.
2026-10-18 00:50:15 +01:00
alan d6cb1931d0 Merge main into lane-pill: combine the pill overlay with main's chat-close, omni-search and status work
Brings the branch up to date with main's tip so it can be merged back without
conflict. Merged cleanly, with no conflicted path: git auto-merged
ui/PipelinesApp.kt, ui/chat/ChatListPane.kt and ui/screens/RunHistoryScreen.kt,
keeping main's changes (the omni-search panel, the one status colour, the chat
close/tool-work surfaces) alongside this item's pill overlay, its
LocalPillBottomInset clearance and the SettingsDestination templates stack.

android/README.md is main's version, not this item's: this item records its
documentation change in docs/notes/pill-overlay-and-position.md for the single
docs-fold writer instead of editing that shared document.

Checks re-run on the merged tree (macOS, JAVA_HOME=Android Studio jbr,
ANDROID_HOME=~/Library/Android/sdk, Gradle 8.11.1, --offline --no-daemon):
`gradle :app:assembleDebug :app:testDebugUnitTest` -> BUILD SUCCESSFUL, 67 result
XMLs aggregating to 627 tests, 0 failures, 0 errors, 0 skipped.
2026-10-18 00:21:20 +01:00
alan 5a44ff1568 android: make the destinations pill a true overlay, centred narrow and right when wide
The floating destination pill no longer takes layout space: it is drawn in the
bottom overlay layer above the content, horizontally centred on a narrow window
and anchored to the bottom-right corner on a wide one. Templates leaves the pill
for a Templates row on the Settings page.

- PipelinesApp: the content Column's pill reservation is gone; contentPillPadding
  (pillVisible) is the named zero and a unit test pins both states. The pill and
  the attention chip stay in their own bottom-anchored overlay Column, a sibling
  of the content in the Box, so nothing positions the content. The one placement
  rule is destinationPillAlignment(wide) (CenterHorizontally narrow, End wide);
  the corner margin is the named SPACING_L, so no new number. The attention chip
  keeps its own bottom-right place at every width via Modifier.align(End).
- Trailing clearance moves into each scrollable's own content: pillScrollInset
  (DESTINATION_PILL_RESERVE + ATTENTION_CHIP_RESERVE while drawn, zero while
  hidden) is published through the new LocalPillBottomInset
  (components/PillInset.kt) and applied with pillContentPadding /
  Modifier.pillBottomPadding in every list, grid and scrolling detail, stage and
  settings body. The container keeps its full height, a hidden pill leaves no
  empty band, and the last row still scrolls clear of the pill.
- Templates is no longer a pill destination: TemplatesDestination.kt is deleted
  and destinations is pipelines/runs/chat/settings. The new SettingsDestination
  sliding stack (pane 0 SettingsScreen, 1 TemplateGalleryScreen, 2
  TemplateDetailScreen) with the pure, Bundle-able SettingsSelection model gives
  the gallery the shared covered-pane strip and one-level Back; SettingsScreen
  gains the Templates row that opens it. This re-lands the intent of a2043f9,
  which never reached main.
- Dimens: the two reserve tokens' docs now describe the scroll-content clearance
  and the chip's share of it; no new token and no hardcoded geometry.

Checked where the change belongs: no packaged template (examples/*.md) and no
packaged authoring skill reference (internal/author/skill/references/*) describes
the Android pill, its destinations or the Templates gallery, so none needs
updating and internal/author/install.go's inventory is unchanged. android/README.md
does document the surfaces this item changed, so it is now stale: this item does
not edit that shared document - docs/notes/pill-overlay-and-position.md records
the two places that need it (the Templates bullet and the dated section to append)
for the single docs-fold writer.

Tests: at a297fb6 with these working changes the build and unit suite were green
(593 tests, 0 failures); no device command was run and the geometry stays
token-derived. Visual rendering of the centred/right pill needs the device and
was not verified here.
2026-10-18 00:20:37 +01:00
alan f85c172dc2 Merge main into chat-close: combine the shared chat room with the server-side close
The item adds chat_close/chat_reopen to the brokered chat protocol and the
operator-session registry's closed state. main replaced the per-session claim
model with shared chat rooms (chat_subscribe, a prompt queue, one broker stream
fanned out to every watcher). This merges the two.

Resolved conflicts:
- internal/protocol/chat.go, internal/protocol/message.go: kept both additions -
  ChatClose/ChatReopen and main's ChatSubscribe/ChatQueued.
- internal/server/chat_sessions.go: kept the closed fields on a listed session;
  dropped the `Busy` assignment, because main no longer marks a session Busy (any
  number of clients may watch and write one).
- internal/server/chat_claims.go: main deleted the claim model, so the file goes;
  close's "is a turn running" check is now `chatHub.turnInFlight` in
  internal/server/chat_room.go, reading the room's `running` flag. That is the
  same question the old claim answered, against main's model.
- internal/server/chat_close.go: calls turnInFlight instead of the removed
  `active`.
- android AppViewModel.kt, ChatFormat.kt, ChatListPane.kt, Message.kt: kept both
  sides - main's search state, queued-prompt handling, `Busy`-free status label
  and row/menu shape, plus the item's close/reopen actions, closed chip and
  section, close confirmation and "Hide on this device" rename.

Also in this merge: main's new convention (AGENTS.md) that an item does not edit
a shared document; the item's stale-document record for this change lives in
docs/notes/chat-close.md, and docs/cli.md, android/README.md and
internal/author/skill/references/operations.md are left to the docs-fold writer.

Checks after resolving: gofmt clean, go build ./... and go vet ./... clean, the
Go module passes, and the app's :app:assembleDebug and :app:testDebugUnitTest
pass (--offline --no-daemon).
2026-10-18 00:19:23 +01:00
alan 971e555ceb chat: close and reopen an operator session on the server, keeping its history
Add a server-side close for operator chats, so finishing with a chat takes it out
of the active list for every client without deleting anything. The brokered
protocol gains chat_close and chat_reopen; chat_close stops a running turn by
reusing the existing broker interrupt (its result carries an interrupted flag, so
a stream is never left running into nothing) and then sets closed_at on the
session's operator_sessions row. The session, its transcript and its history
survive, and because the closed state lives in the registry chat_sessions reports
it to every client, not just the one that asked. An id that is not a recorded
operator session, including a pipeline-owned run session, is refused before
anything is written. chat_reopen clears the state again, and simply prompting a
closed session also clears it (the store clears closed_at on touch).

The registry gains closed_at for a fresh database and an idempotent migration for
one created before it existed. The app renders closed sessions under a labelled
Closed section with a Closed chip, offers a Close action with a confirmation that
says it stops a running turn, marks the chat closed on every device and deletes
nothing, and a Reopen action on each closed row. The device-local hide is renamed
Hide on this device, with a snackbar saying the chat stays on the server, so it is
never confused with the real close. No new dependency, the installer and Docker
files are untouched, and no run is started from the app.

Checked before committing: the packaged templates carry no chat surface and need
no update, and internal/author/install.go's managed inventory needs none either
(this adds no reference file). The packaged authoring skill reference
internal/author/skill/references/operations.md and the shared docs/cli.md and
android/README.md are stale because of this change, but an item no longer edits a
shared document: what is stale and what should change there is recorded in
docs/notes/chat-close.md for the docs-fold writer, which is committed here with
the change.

Checks: gofmt clean, go build ./... and go vet ./... clean, and
go test ./internal/protocol/... ./internal/pipeline/... ./internal/server/...
./internal/author/... pass. The app side is verified with :app:assembleDebug and
:app:testDebugUnitTest (--offline --no-daemon, BUILD SUCCESSFUL); the on-device
close, second-client and reopen check is deferred to visual review.

Not deployed; deploying is the operator's step.
2026-10-18 00:16:24 +01:00
alan 672d260b33 Merge main into multiclient: combine the multi-client chat fan-out with main's chat tool-work, notifications, status palette and docs-fold changes
No conflicts: main's chat-surface work (the in-flight tool work in ChatFormat,
ChatStore and ChatTranscript, run notifications, the status palette) sits in
different hunks from this branch's fan-out edits, and main did not touch
docs/cli.md. Checks re-run on the merged tree: gofmt -l . (clean), go build ./...,
go vet ./..., go test -count=1 ./... (all packages pass; the focused multi-client
tests fan-out, per-session serialisation, first-answer-wins, pending replay and
resync pass), and the Android unit tests
(gradle --offline --no-daemon :app:testDebugUnitTest, BUILD SUCCESSFUL, 598 tests,
0 failures).
2026-10-18 00:10:15 +01:00
alan 6cb61578de Fan a chat session out to every client and share approvals exactly-once
A session was owned by one connection: the claim map in chat_claims.go let the
first client hold it, every other client saw the session Busy and read-only, and
a second client's prompt was rejected. Replace the claim with a per-session
"room" so the operator's phone, tablet and terminal all watch and write the same
conversation, and a pending approval is a shared, exactly-once object.

Server (internal/server, internal/protocol):
- New chat_room.go: one room per session holding the session's single broker
  stream, its one bounded prompt queue (8) and the fan-out to every attached
  client. A chat_subscribe frame attaches a connection to a session's shared
  stream and replays the still-unresolved approval requests the room holds.
- chat_stream.go: handleChatPrompt accepts a prompt from any client. It attaches
  the sender, either starts a turn or queues the prompt behind the in-flight turn
  and acknowledges it with chat_queued:true ("queued" is the chosen behaviour,
  stated on the wire and in docs/cli.md), and echoes the user turn to every other
  client attributed to the sender's label. The prompt handler no longer owns the
  stream; one turn runner per room drains the queue, so two turns never
  interleave. Per-connection write serialisation is unchanged (each chatSub has
  its own writer goroutine and bounded buffer).
- chat_permissions.go / chat_commands.go: an approval is reserved by the first
  answer and confirmed once delivered; the first answer wins, is recorded with
  the answering client's label, and the resolution is pushed to every watching
  client as chat_permission with resolved/decision/resolved_by so their cards
  clear. A later answer is refused with an error naming the client that answered,
  and never reaches the agent twice; a reservation whose reply failed is
  released.
- chat_hub.go / chat_sessions.go / chat.go: the hub now owns rooms and the
  user-turn id sequence; the resulting user turns are retained (role and client
  carried) so a reconnect re-syncs who said what. chat_sessions no longer marks a
  session Busy and its clientID parameter is gone; the protocol keeps the Busy
  field for wire compatibility only. websocket.go detaches a connection's chat
  subscriptions when the read loop ends.
- chat_claims.go is removed, not left vestigial: the only exclusive thing left is
  a turn, and the room's queue is that.

App (android):
- PipelinesRepository carries a stable per-device client label (Build.MODEL, with
  a fallback) on prompts and replies, and adds chatSubscribe; AppViewModel
  subscribes on opening a chat (best-effort, an older server's error frame does
  not stop history loading) and records the queued ack.
- ChatStore folds a user frame from another client into the transcript once,
  attributed by the sender's label, and never receives its own prompt back, so
  the sender's own message is never duplicated. ChatThread.readOnly is gone:
  canSend no longer depends on another client, the composer says "Queued - it
  will send when the current turn finishes." instead of disabling input, and the
  status label is Running/Idle only. Scroll/follow behaviour is untouched.
- Tests: ChatProtocolTest round-trips the new wire fields, ChatFormatTest asserts
  the stale Busy flag cannot produce a read-only label, MultiChatStateTest asserts
  a session another client watches stays writable.

Reflected where it belongs: no packaged template changes (no bundled starter
embeds the chat protocol). Checked the packaged authoring skill references: none
of them describes the chat wire beyond operations.md, and
internal/author/skill/references/operations.md is now stale - its "Brokered
OpenCode chat" section still says chat_sessions marks "busy" when another client
holds the session and that a second client's chat_prompt is rejected, and it does
not mention chat_subscribe, the per-session prompt queue/chat_queued, the
per-client label or the shared exactly-once approval (it also still describes the
per-subscription permission poller, which the room's single poller replaced). The
supported semantics are the ones in docs/cli.md, updated in this commit; the
operations.md section needs a rewrite to match. No README or other shared
document was edited here.

Checks run on the committed tree: gofmt -l internal cmd examples (nothing), go
build ./..., go vet ./internal/... ./cmd/..., go test ./internal/server
./internal/protocol ./internal/stage ./internal/pipeline.
2026-10-18 00:09:14 +01:00
alan f339503f68 Merge branch 'main' into session-activity 2026-10-18 00:05:10 +01:00
alan 597c7b5e76 android: show a chat's in-flight tool work instead of a bare Running
A session said "Running" with an empty tool chip while it worked, so a silent
wait was indistinguishable from a stuck one. Derive the in-flight work from the
stream the app already folds and show it on both surfaces.

- ChatFormat: runningToolCalls() reads the tool parts still running, in
  transcript order, so the conversation and the session row share one set;
  toolDetail() takes the command/description the part actually carries (the
  first of description/command/filePath/file_path/path/pattern/query/url),
  collapsed to one line and elided at 72 chars, and returns null rather than a
  fabricated summary; runningToolElapsedMillis() uses the protocol's own
  time.ran/created when present and the local per-part stamp otherwise;
  runningToolStamps() keeps a stable start per part across frames;
  TOOL_LONG_RUNNING_MILLIS (60s) and toolLongRunning() mark an unusually long
  call; sessionWorkLabel() names what a working session is doing while a
  text-streaming turn stays plain "Running" and a busy session never reads
  "Idle".
- ChatStore: ChatThread gains toolStartedAt, stamped when a part turns running
  and dropped when it settles, plus runningTools/toolElapsedMillis; history()
  and the frame fold maintain it; setRunning/turn end clear it.
- ChatTranscript: RunningIndicator lists one line per running call (name,
  detail where present, live elapsed) keyed by part id so concurrent calls
  render together without flicker; no percentage, ETA or progress bar.
- ChatListPane: the session row renders sessionWorkLabel(busy, thread).
- ChatRunningWorkTest covers name/detail/elapsed, clear on completion, error
  state, concurrent calls and the no-detail stream.

Only fields the harness reports are used: tool name, state.status and input;
the live brokered frame carries no timing envelope, so elapsed falls back to the
local stamp. Existing chips, INPUT/OUTPUT, reasoning blocks, stop control,
activity marks and scroll/follow behaviour are untouched.

Checked and unchanged: no packaged template or packaged authoring-skill
reference describes this app surface (the chat references in
internal/author/skill/references/ document the server's brokered chat_* protocol,
which this change does not touch), so none is stale. android/AGENTS.md already
names ui/chat/ and its files. The shared documents this change does make stale
are recorded, not edited, in docs/notes/chat-in-flight-tool-work.md: the
Running-indicator and session-status statements in android/README.md.
2026-10-18 00:05:07 +01:00
alan a3f571d672 docs: make docs-fold the single writer of shared documentation
Every item wrote the shared, append-only docs (android/README.md, docs/cli.md and
the packaged skill references), so every merge conflicted on prose. Keep the duty
to notice stale docs; move the writing to one deliberate place.

- Per-item note, not a shared edit. An item that makes a shared document stale
  writes docs/notes/<slug>.md - unique per item, so two items never touch the same
  file - and commits that note with its change. It never edits a README, topic doc
  or packaged skill reference itself.
- One writer. The manual-only docs-fold pipeline, bundled as the fifth starter
  examples/docs-fold.md, folds every note under docs/notes/ into the shared
  document it names and deletes the consumed notes in its own commit; an empty
  notes directory is a success no-op. Operator trigger:
  pipelines new "Docs fold" --template docs-fold --no-open; pipelines submit
  <draft.md>; pipelines run "Docs fold".
- Implemented where the convention lives: orchestration.md (brief item 5, the
  hard-constraint bullet, a "Single documentation writer" section and the
  "Pipeline choice" committer bullet), the canonical committer checklist in
  format.md (new step 6), the committer prompts in examples/task.md and
  examples/three-stage.md, the new examples/docs-fold.md with its catalog and
  README entries, and the repository guideline AGENTS.md. Test-expectation counts
  (examples/catalog_test.go, internal/server/templates_test.go,
  scripts/smoke-first-install.py) follow the fifth starter.

Reflected where it belongs: the packaged templates (examples/task.md,
examples/three-stage.md, the new examples/docs-fold.md), the packaged skill
references (orchestration.md, format.md) and the repo guideline AGENTS.md all carry
the new convention. No new reference file was added, so install.go's managed
inventory is unchanged. The shared descriptive docs were deliberately not
rewritten: this change makes README.md:191, docs/cli.md:170 and
internal/author/skill/references/operations.md:13 name four starters where there are
now five. Those stay as they are - correcting them now is exactly the per-item
shared edit this change removes - and the stale-doc note
docs/notes/docs-fold-single-writer.md records each one for the docs-fold writer to
fold. Nothing else needed updating. No new dependency; no Android, Gradle, server,
installer, Docker or published-definition change; no definition published over the
running pipelines.

Evidence: go test -count=1 ./... passed; go build ./... and go vet on the touched
packages clean; ./internal/author/... covers embedding, reference links, the managed
inventory and install/upgrade/protection; ./examples/... validates the fifth starter
and its manual-only, committer-terminated graph; ./internal/server/... covers the
served catalog. Not run: the docs-fold writer against the shared docs (nothing to
fold yet); no install or deploy.

(cherry picked from commit f29656cf084d7eee2e033f2a37a9d231d7acaaf0)
2026-10-17 23:59:59 +01:00
alan 4426f8cacd docs: re-verify the chat record at main's build (ec27a2d)
Update docs/chat-verification.md from the earlier `3ffc126` record to the
current revision `ec27a2d`, which is both this lane's HEAD and main:

- revision, APK name/size/sha256 and LAN URL refreshed to the rebuilt
  artifact `pipelines-debug-wip-ec27a2d.apk` (18,585,084 bytes,
  sha256 5ee4b38c…da84);
- unit-test totals refreshed to 61 suites / 558 tests / 0 failures, and the
  server-side `go build ./...` plus the chat/session/adopt Go tests recorded;
- a "Changes since the previous record" section describing the chat-relevant
  delta `3ffc126..ec27a2d` (the `chat_adopt` request, the lazy
  `MarkdownBlocks` renderer, the run-watch scope change) and correcting the
  chat protocol file path to `data/protocol/Chat.kt`.

The change is documentation only: no app or server behaviour changes.
Checked where the change belongs: packaged templates (no template encodes
this record) and the packaged authoring skill references
(internal/author/skill/references/) — none is made stale by a
verification-record update, so none is touched. `go build ./...` passes.
The debug APK lives in the run's artifacts directory outside the repo and is
therefore not committed.

Refs: brief "Verify the chat end to end, on the phone, against the real server".
2026-10-17 20:55:19 +01:00
alan 3f5c11b7bf Record and expose per-run model usage (tokens, split, cost)
Carry the usage a model stage already measures for its budget
(stage.Budget.Snapshot()) onto the step as typed evidence and total it in
one pass, rather than a second harness read, a transcript parse or an
output-length estimate. A shared budget across stage revisits records only
the visit's increment, and a revisit that observes no new report records no
usage instead of a fabricated known zero.

- protocol: StageUsage/RunUsage/TotalUsage and UsageTotals.
- pipeline: step_usage table + migration, COALESCEd into the one step
  projection, FinishStep persistence, one json_extract history aggregate,
  retention removes the rows with their run.
- server: stageUsage maps the budget snapshot, stamping per-stage model and
  usage for every model stage (including failures).
- cli: inspect/stats human and JSON output.
- budget/pi: retain the reported in/out split, mark a total-only (Pi) report
  split-unknown, and keep an unreported cost unknown (never zero).

Evidence and docs updated in the packaged skill references and docs/cli.md;
the live database has no step_usage rows yet (all history predates this), so
per-run figures are reported as unavailable, not zero.
2026-10-17 20:40:56 +01:00
alan 1f75e4797e Broker chat from an operator's own OpenCode endpoint and adopt existing sessions
Add an opt-in way to broker chat from an operator-supplied OpenCode service
instead of the instance the server spawns for itself, and a way to adopt a
session that already exists on that endpoint so it appears in chat_sessions and
can be continued from the app.

- Endpoint selection: the `-opencode-endpoint` server flag (defaulting to
  `PIPELINES_OPENCODE_ENDPOINT`) opts in; precedence is explicit flag >
  explicit environment > the endpoint recorded on a previous run, with the
  spawned instance as the default. opencode.OpenEndpoint validates the
  reference once (http(s), host, no userinfo/query/fragment/path), detects the
  backend from its own /api/info (v2) or /global/health (v1) contract, and
  requires a streamable text/event-stream event path. It owns no process
  (Close is a no-op) and never calls startServe; every failure names the
  endpoint and disables chat rather than silently falling back. An
  authenticated endpoint takes an optional password only through the private
  file PIPELINES_OPENCODE_ENDPOINT_PASSWORD_FILE, with
  PIPELINES_OPENCODE_ENDPOINT_USERNAME overriding the detected default.
- Persistence: the validated choice is recorded in an additive single-row
  server_settings table so a restart with no flag/environment reuses it; an
  unreachable recorded endpoint disables chat with a clear message and no
  fallback.
- chat_adopt: a new protocol request (ChatAdoptRequest) with handleChatAdopt in
  internal/server/chat.go. It refuses a nil broker/registry/payload and a
  blank or foreign-shaped id before any endpoint call, reads the endpoint via
  the shared Chat.Session (which reuses Sessions/decodeChatSessions), records
  through the existing RecordOperatorSession with the endpoint's directory and
  title verbatim, and echoes the adopted session plus its registry row like
  chat_create. A missing id is refused naming the id and records nothing; an
  endpoint record with an empty directory is refused rather than substituting
  the server's working directory or home.
- CLI: `pipelines chat-adopt <session-id> [--json]` (usage, notes, help
  inventory, dispatch) with docs/cli.md updated under Chat sessions.
- Autostart preserves the endpoint variables (and the password-file reference,
  never an inline secret) via serviceEnvironmentKeys.

Checks before committing:
- Package location: the change is user-facing chat behaviour, so the packaged
  authoring skill reference internal/author/skill/references/operations.md is
  updated in the same change (Brokered OpenCode chat section) with a dated
  verification-evidence entry and the not-deployed state. No new reference file
  is added, so internal/author/install.go's managed inventory is unchanged.
- Packaged templates: this repository packages no template set for this
  behaviour and none references the chat endpoint, so no template needs
  updating.
- Verification: `go build ./...`, `go vet` on the touched packages, `gofmt -l`
  clean and `go test ./...` all pass. The unit tests cover the endpoint
  validation/precedence, the adopt happy path and refusals, the recorded
  endpoint round-trip, the protocol round-trip and the CLI dispatch. The live
  scratch-server check against the operator's real endpoint was not run in this
  change.

Nothing unrelated is staged; docs left stale (README and other shared documents)
are not edited here and are for a single folding writer.
2026-10-17 20:14:11 +01:00
alan 58c6b9b7c6 server: export a finished stage's real session as session.md and session.html
Every OpenCode-harness model stage now writes its real session into its own
step artifacts directory as session.md and session.html, so the existing
bounded artifact listing, the /artifacts/<run>/<step>/<name> serving path and
the artifact scan find and serve them with no special casing, while the stage
keeps its bounded preview in the run record unchanged.

- internal/opencode/export.go adds the one shared native export invocation and
  the exact session-ID/messages validation used by both Connection.Messages and
  the new ExportSession, so the readback path does not diverge. The backend's
  published subcommand is used (v1 `export <id>`, v2 `session export <id>`),
  the source read is capped at 8 MiB, and a larger session is read as a prefix
  and reported truncated instead of failing. boundedOutput.ReadFrom keeps the
  cap effective so overflow and the truncation path are real.
- internal/stage/session_export.go renders markdown and html with a metadata
  header, every role and every part (unknown part types as raw JSON, every
  session text escaped in html), atomic temp+rename writes that never overwrite
  an existing file, a visible truncation banner at both ends when the source or
  the 2 MiB rendered cap cuts it, and an honest 0-message header for an empty
  session rather than fabricated text.
- protocol.SessionExport and RunStep.SessionExport are stored in a new
  step_session_exports table (CREATE TABLE IF NOT EXISTS, so an old database
  gains an empty table with no migration) loaded through the existing main
  SELECT alongside the other per-step evidence and deleted by run retention.
- internal/server/runner.go exports after the post-stage filediff capture and
  before scanArtifacts, with context.WithTimeout(context.WithoutCancel(ctx),
  10s); it never waits for session idleness and a failure or a Pi stage (no
  native export path) records a sanitized diagnostic and continues, never
  changing the stage status, outcome, target, timing or result validation.

Reflected where it belongs: the packaged skill references
(internal/author/skill/references/operations.md and delivery.md) carry the new
Session export rules and the run-artifacts bullet, and docs/stage-artifacts.md
documents the names, the <run>/<step> paths, the served URL, the caps, the
truncation labels, failure isolation and the Pi limitation. The packaged
templates (examples/*.md) only tell a stage to save its own artifacts and say
nothing about the server-written session export, so none needed updating; no
new reference file was added, so install.go's managed inventory is unchanged.

Shared documentation: README.md's per-step artifacts sentence is now stale - it
should name the exported session alongside images, screenshots, reports and
logs. It is deliberately not committed here; the shared documents are folded by
the single documentation writer.

Checks: gofmt -l on the touched packages produced no output; go vet ./...
passed; go test -race -count=1 ./... passed.
2026-10-17 19:40:43 +01:00
alan 976169e413 Record the chat end-to-end verification record
Add docs/chat-verification.md, the required record for the chat verification brief: per-flow table (flow, result, evidence path, notes), the known tree-level gaps, and the section on what could not be verified and why. Verification only; no app behaviour changed.

Checked: packaged templates (internal/author/skill/references/) do not mention this record - no template or skill reference needed updating. Shared README and topic docs are intentionally not touched (single-writer fold).
2026-10-17 18:02:00 +01:00
alan 476ecc781c Merge main into lane-srv3: combine the review-loop control change with main's notification and chat work
Keeps this branch's advisory review edge, bounded blocked_retryable implementer
edge and their tests alongside main's run-attention/ntfy notification work, the
split of the Android chat conversation screen, and the retention/history changes.
No conflicts: the two sides touch disjoint files.
2026-10-17 17:07:17 +01:00
alan e70c94e8dd control: make advisory review findings non-blocking and bound retryable blocks
Cut the review loop's repeat cost and make a blocked stage actionable, at the
definition level only, with the runner's existing visit counting and caps.

The published maintenance, quick and focused pipelines gain a reviewer
`advisory -> next stage` edge for non-blocking notes, so an advisory-only review
continues the run instead of buying another implementer visit; `rejected ->
implementer` and its unchanged {"max_visits":3,"on":["rejected"]} limit keep a
must-fix finding looping and still terminate. The implementer gains
blocked_retryable -> implementer with {"max_visits":5,"on":["blocked_retryable"]}
for a transient environment/tooling/device problem a fresh attempt could
survive, while blocked -> #failure stays terminal. The cap is at least the
reviewer's rejection budget so the implementer's own pre-execution cap cannot
preempt the reviewer's rejected control.

No production Go code changed. New tests in internal/server/review_contract_test.go
cover advisory-only never revisiting the implementer, must-fix still looping to
the reviewer limit, the implementer cap not preempting the reviewer, retryable
blocks bounded then failing closed, a retry that recovers, and a terminal block
failing immediately; concurrency/control/human tests share the new submitRunner
and startRun seams.

Documentation checked: internal/author/skill/references/control.md and
docs/harnesses-and-control.md carry the new edges and the visit-cap rule;
no packaged template defines these pipelines (they are published immutable
versions), and docs/model-results.md needs no change because the {outcome,output}
result contract is unchanged. README and other shared docs deliberately untouched.

Verified: go build ./... and go test -count=1 ./... pass.

Residual limitation: the must-fix-vs-advisory split is carried by the prompt
contract, not a machine-readable marker, because the recorded findings have none;
a model that still returns rejected for advisory-only notes would still buy a
visit. All 18 recorded implementer blocked runs classify terminal, so the
blocked_retryable edge recovers 0 of them.
2026-10-17 17:06:51 +01:00
alan 4908452a74 retention: report removed runs and follow a symlinked artifacts root
Fix two honesty gaps in run-history retention that the implementer stage
left uncommitted on top of 14175eb:

- Populate RemovedRuns, RemovedRunCount, RemovedArtifactFiles/Bytes and
  Applied right after the row transaction commits, before staging cleanup.
  A cleanup failure must not report a successful row/artifact removal as
  zero removed, and staging leftovers are excluded from SpaceReclaimedBytes.
  A failed VACUUM after removal now reports the artifact bytes actually
  removed while claiming no database pages.
- Stage artifacts through os.Stat instead of os.Lstat so a symlinked
  artifacts root is followed: the report sized that directory, so apply
  removes through it too instead of counting bytes it never deleted.

Tests cover post-commit staging-cleanup failure reporting, staging through a
symlinked artifacts root (including no staging directory left in the link
target), and an honest reclaim failure after removal.

Checked documentation surfaces:
- docs/cli.md: measured example refreshed with the current before/apply/after
  output and the symlink/staging/VACUUM reporting described accurately.
- README.md: retention prose (explicit, never automatic, protected runs,
  artifacts go with the run) is unchanged and still correct.
- internal/author/skill/references/operations.md: operator-only retention
  summary describes the same behaviour; no user-facing surface changed, so
  no reference update was needed.
- Packaged templates: retention is a CLI/store capability with no template
  change, so nothing to update. internal/author/install.go inventory
  unchanged (no reference added or renamed).

Verification: gofmt clean on the touched Go files, git diff --check clean,
go vet ./internal/pipeline clean, and go test ./internal/pipeline -run
'TestRetention' -count=1 passing.
2026-10-17 16:19:39 +01:00
alan 890d267c71 run attention: publish to ntfy and surface what needs the operator
Nothing told the operator that a run was waiting for a person or had failed; they
found out by going looking. A run that needs attention now reaches the ntfy they
already run, and the app answers "what needs me now" in one live, ordered surface.

Server (smallest change, reusing the existing publisher):
- internal/server/notify.go adds notifyRun, called from finishRunState (failed)
  and when a run parks on a human (runner.go runLoop). It publishes to the same
  already-configured `system` destination through the unchanged stage.PublishNtfy,
  so there is no second publisher, topic or service and the token handling is the
  stage's own. runAttentionEvent maps only waiting and failed; queued, running,
  succeeded, canceled and interrupted publish nothing.
- No storms: runner.attentionSent records sequence:event per run before publishing,
  so a run that stays waiting is never republished on a re-request or restart, and
  a stage notify policy that already covers the same event marks it first so the
  run-level default is suppressed. The set is dropped when the run is terminal. A
  delivery failure is logged, never fails the run, and is not retried.
- The payload names the run and pipeline and carries a click action
  pipelines://run/<id> (validated: absolute URL, no userinfo, no fragment, no
  credentials). PublishNtfy gained the optional click parameter; existing callers
  pass "".

App:
- AndroidManifest/MainActivity declare the pipelines://run/<id> deep link
  (singleTop); runIdFromDeepLink parses it and PipelinesApp opens that run once,
  clearing the link. It opens a run; it starts nothing.
- ui/Attention.kt is the pure model (status -> Decision/Failed/Finished or
  nothing, ordered by kind then newest-first) and ui/components/AttentionSurface.kt
  is the floating indicator plus bottom sheet. It is derived from the same run
  events the server already pushes, so an entry appears with no refresh and clears
  when resolved; a finished run clears once looked at, and any entry can be
  dismissed, persisted in SettingsRepository. The existing destinations, pill and
  bottom strip are untouched and no competing navigation surface is added.

Checked before committing:
- Packaged authoring skill references: internal/author/skill/references/
  notifications.md documents the automatic run-attention notifications and the
  app surface. The only packaged "template" carrying notify is that reference; no
  pipeline definition template exists to update and install.go's managed file list
  is unchanged (notifications.md already existed).
- Repository guidelines: docs/ntfy.md gains an "Automatic run-attention
  notifications" section and no longer claims "No block means no notifications";
  the top-level README phone-notifications note now says a waiting/failed run
  notifies automatically; android/README.md gains the dated feature section and
  supersedes the previous section's note that the attention surface was absent.
- docker.md, install docs and the CLI/TUI docs describe mechanisms this change
  does not alter; left as they are.

Tests: go test ./internal/server/... ./internal/stage/... ./internal/author/...
all ok; notify_test.go covers the event mapping and the real HTTP delivery
(destination, topic, click, no duplicate). Android :app:testDebugUnitTest ->
BUILD SUCCESSFUL, 58 suites / 531 tests / 0 failures (new AttentionTest, 13 cases,
covering ordering, appearing from a push, clearing on resolution, dismissal, the
empty case, persistence and the deep link). Device screenshots and the phone's own
notification are visual review's per the brief's device policy.

Scope: android/ plus the smallest server change; no new dependency, no run started
from the app, no installer or Docker change.
2026-10-17 16:11:57 +01:00
alan 14175eba70 retention: explicit run-history and artifact retention
Add `pipelines retention report` (strictly read-only) and
`pipelines retention apply` (explicit, approval-gated removal) with an age
and/or count policy, protected-run rules, atomic single-transaction removal
and honest database space reclaim. Nothing prunes automatically.
2026-10-17 15:45:46 +01:00
alan 7b72b8f20f stage: retry pre-admission harness reach failures and widen repair deadline
Treat a failure to reach the harness before any prompt or tool call is
admitted (connecting or creating the session) as a distinct retryable
reach failure, so a transport blip at the first stage no longer ends a
mutating run; keep post-admission readback/poll/agent failures retried
only on an explicit read-only stage and never replay admitted tool work.
Preserve the true cause when the context is already done (join ctx.Err
with the typed transport error) and report a sanitized unknown provider
error name/status instead of a blanket summary.

Make the isolated format-repair deadline proportional to the stage
timeout (floored, capped and clamped to remaining minus a margin) so a
slow-but-repairable result is recovered without ever extending the stage
deadline; the bounded ladder, value guard, deny-all formatter session,
cumulative usage accounting and no-tool-replay rule are unchanged.

Update docs/model-results.md, docs/pipeline-performance-review.md and the
packaged skill references; add stage, opencode and server fixtures.
2026-10-17 14:51:09 +01:00
alan 5fafe75267 Merge finish-z02: keep the lane's reviewed work 2026-10-17 11:15:52 +01:00
alan 6a6c1d65fa android: drive the runs list and run detail from server pushes
Replace the Android app's one-second run poll with the server's
workspace-scoped run_event pushes. The runs list, each pipeline row's
last-run line, the run header and the run detail (status, timeline,
stages, artifacts) now advance from the socket instead of polling or a
manual refresh.

- Wire: Message.kt mirrors internal/protocol/runevents.go (subscribe,
  unsubscribe, run_event; RunEventFrame/RunStepDelta). The frames are
  unsolicited and already arrive on the client's events stream;
  PipelinesRepository.subscribeRuns/unsubscribeRuns add the typed
  command, sent via a new PipelinesClient.sendOnly that does not hold the
  send lock waiting for the running server's unreliable subscribe ack.
- One pure fold: ui/RunEvents.kt folds a frame into the list (in place,
  order preserved) and the open detail with one shared status ladder and
  one shared run-field merge. It is monotonic (a stale or duplicate
  delta is dropped whole), reports needsResync for a run the list does
  not hold instead of inventing a row, and only requests a full get_run
  for a settled step or terminal run.
- Wiring: AppViewModel subscribes once per (re)connect and workspace;
  every (re)subscribe re-reads the list and open detail once, because the
  deployed server's subscribe snapshot is partial, then folds the pushes
  on top. Re-reads are push-driven, coalesced and keyed so no push is
  dropped and no timer remains. LiveBadge (RunStream.kt RunStreamState)
  shows honestly live/reconnecting/not live, confirmed by the ack, a
  push or a successful read and dropped when a read fails.
- ui/RunEventsTest.kt pins the delta sequence as pure cases: created run
  not in the list, step started, step finished, run finished, unknown
  run, duplicate/out-of-order deltas, and a reconnect resync.
- android/README.md records the change, the wire mapping and the
  observed limitation of the running server (partial snapshot, no ack
  within 30s), which the client tolerates without inventing a stream.

Checked where the change belongs: android/README.md gained this entry;
docs/cli.md described the Android app as still polling, which this makes
stale, so that sentence now says the app consumes the pushes while the
TUI still polls. The packaged TUI pipeline templates (internal/tui) do
not touch run-state refresh and need no change, and the packaged
authoring skill references (internal/author/skill/references) document
the authoring workflow, not the Android client's run updates, so none of
them needs updating.

Reviewed and approved by the run's review stage (build, unit tests,
Kotlin compile and lint in android/README.md); no merge/rebase/cherry-pick
was in progress and no conflict was present.
2026-10-17 11:15:21 +01:00
alan 18398dfa5b chat: give the operator their own sessions, separate from run sessions
chat_sessions used to list exactly the OpenCode sessions the pipelines own,
filtered through pipeline.SessionOwner (run_steps/run_session_owners), so an
operator-created chat was an accident of the run-ownership table and the
operator's working sessions were mixed with the internals of running pipelines.

Add a first-class operator session:

- internal/pipeline/operator_sessions.go: a dedicated operator_sessions table
  (session_id PK, directory, title, tag, created_at, last_used_at) with
  RecordOperatorSession (upsert preserving created_at), a one-statement
  TouchOperatorSession (no-op for a non-operator id), OperatorSessions bounded to
  128 rows most-recently-used first, and OperatorDirectories (bounded MRU,
  deduped). run_steps/run_session_owners and SessionOwner/OwnedSessions are
  untouched.
- internal/server/chat.go: chat_sessions returns operator sessions only - it
  reads the registry once, intersects it with one broker session-list call, fills
  directory/tag/created/updated from the registry, sets busy for a session held
  by another connection, keeps MRU order, and stays fail-closed (empty list, never
  the endpoint's whole list). A pipeline stage session and an untagged row can
  never be listed. chat_create resolves the directory (empty/'~'/'~/...' ->
  operator home; relative or missing rejected before any OpenCode call), calls
  broker.Create(title, resolved) and records the row with the operator tag.
  chat_directories returns home first then bounded (8) deduped recents. A
  successful prompt touches the operator session without affecting pipeline
  prompts.
- internal/opencode, internal/protocol, internal/server/websocket.go and
  message.go: unscoped session list, Create posts {"title":...,
  "location":{"directory":...}} to POST /api/session, additive protocol fields
  (omitempty) and the ChatDirectories request/response, plus the OperatorRegistry
  option defaulting to the first store. One OpenCode transport per server is
  unchanged.

Docs and references: docs/cli.md gains a "Chat sessions" section documenting the
operator-only fail-closed list, directory defaulting/rejection, chat_directories
(home first, deduped, bounded 8) and why a dedicated request was chosen.
internal/author/skill/references/operations.md's stale "Brokered OpenCode chat"
text (chat_create unbrokered, run_steps filtering, "listed the owned scratch
session" evidence) is rewritten to the operator-session registry, the fail-closed
filter, directory defaulting and the corrected live-test wording. No new
reference file, so internal/author/install.go's inventory is unchanged.

Checked for staleness elsewhere: README.md, the docs/ topic pages and the
packaged server/TUI templates carry no chat-session ownership claims, so none
needed updating.

Verified: gofmt -l clean; go vet and go build ./... clean; go test -count=1
./internal/pipeline/... ./internal/protocol/... ./internal/opencode/...
./internal/server/... and the full go test -count=1 ./... pass.
2026-10-17 11:14:25 +01:00
alan f31f150e52 Merge lane-t2: keep the lane's reviewed work
Combine the two reviewed versions of docs/pipeline-performance-review.md.

- Local (main) review: the exact 201939-byte runs.json capture, the three
  final_result_validation sub-shapes, and the 5-to-4 unresolved busy-wait
  correction.
- Incoming (lane-t2) review: both captures (artifacts/222/2 and
  artifacts/222/4), both stat snapshots (134 succeeded / 3 active and
  135 succeeded / 2 active), plus the server-side rejection, deploy-wait,
  device-lock and refined improvement-ranking analysis.

Checked: no conflict markers remain anywhere in the tree and the two sides are
composed without duplicated headings or dead text. This is a documentation-only
change to a dated analysis document; the packaged templates, the README and
topic docs, and the packaged authoring skill references are not made stale by
it, so nothing further needs updating.
2026-10-17 02:14:41 +01:00
alan 36fad7d130 Merge lane-go: keep the lane's reviewed work
Combines lane-go's typed omni search, launch-time model overrides, TUI
artifacts view, run-state websocket push, and the "reflect the change where
it belongs" committer rule with the Templates gallery, chat/omni-search and
other reviewed work already on main. The 12 files both sides touched keep
both intents composed rather than dropping either side: the omni search keeps
the typed search contract (CLI `pipelines search` and the TUI popup with type
badges and marked snippets) alongside the run-list quick filter; cli.md and
ui-ux-review.md document both the search rework and the artifacts view;
orchestration.md, harnesses.md and operations.md keep the model-override and
reflect-where-it-belongs guidance; and the protocol/CLI/server/TUI code keeps
the new search, run-event, override and artifacts paths alongside the existing
run, title and help surfaces.

Checks before committing:
- go build ./...  -- pass
- go vet ./...  -- pass
- go test -count=1 ./internal/... ./cmd/...  -- all pass

Doc check before committing: nothing needs updating. The lane's own docs and
references (docs/cli.md, docs/ui-ux-review.md, and the orchestration,
harnesses and operations skill references) already carry the new features.
format.md and the packaged templates already describe the line-oriented result
format and the human decision stage from the earlier lane-go merge. The root
README already describes the omni search popup. Model overrides, the search
command, run events and the artifacts view are runtime/CLI/TUI surfaces that
cli.md documents; no template, README or topic-doc edit is required by this
merge.
2026-10-17 02:04:58 +01:00
alan f5610eb4e6 docs: pipeline performance and reliability review 2026-10-17 01:43:46 +01:00
alan 3997e1bbc1 docs: pipeline performance and reliability review 2026-10-17 01:26:35 +01:00
alan c9b904186a Update templates and authoring skill to the current pipeline contract
Bring the packaged starters and the pipelines skill up to the guidelines the
project now works to.

Templates:
- task and three-stage now end in a committer stage that stages the union of the
  runner's captured input.server_changed_files and the producer's declared
  changed_files, runs the repository's checks, commits locally and never pushes;
  their producers no longer commit.
- Every model/agent prompt states the line-oriented result contract (outcome on the
  first non-empty line, "# output" JSON map, other "# <heading>" sections kept
  verbatim as evidence) and an "# evidence" section for exact commands and results,
  with proportionate verification and no fabricated checks.
- Stages that can write save their reports, logs and screenshots into the runtime
  artifacts directory.
- Remove the summary starter: a tool-free single-prompt text transform with no
  repository interaction, artifact or commit is a format demonstration, not a
  reusable workflow. No starter invents a pretend approval stage; a decision point
  is a type=human stage.

Skill:
- references/format.md gains a Result format section (line form, legacy JSON
  precedence, result_parse, raw-output.md, artifacts) and a Committer stages
  section with the pre-commit checklist; the role-split and envelope notes mention
  the line form and the committer.
- orchestration.md points at the checklist and records that a genuine gate is a
  human stage; operations.md and the SKILL.md index drop the removed starter and
  name the result format and committer. No new reference, so the managed inventory
  is unchanged.

Docs/tests: examples/README, README, docs/cli, docs/first-version-testing, the CLI
help example and the smoke script track the four starters; catalog, tool-role,
server, tui and pipeline tests are updated, with a new test pinning the committer
ending and server-captured change set.

Commands run: go build ./...; go vet ./...; go test ./...; python3 -m unittest
discover -s scripts -p 'test_*.py'; python3 scripts/smoke-first-install.py. All
passed. Templates inspected through an isolated CLI server (pipelines templates;
pipelines template KEY --source). No server, service or device was restarted, and
nothing was pushed.
2026-10-17 01:09:42 +01:00
alan 8f3036bd3b Push run state to subscribed websocket clients
Add a server-pushed run_event frame so a client no longer has to poll
once a second to notice run changes.

- protocol: RunEventFrame/RunStepDelta compact delta (run id, workspace,
  status, outcome, current stage, completed-step count, timestamps and an
  optional single step delta), plus subscribe/unsubscribe/run_event types
  and the optional RunEvent/RunIDs message fields. All fields optional so
  older peers ignore them.
- server: runEventHub with per-connection subscriptions scoped to a
  workspace (and optional explicit run ids), one bounded writer goroutine
  per subscription, slow-subscriber drop, per-client subscription cap and
  a ~220ms per-run coalescer that flushes pending deltas before a terminal
  frame.
- runner: emit deltas on created/queued, queued->running and step start,
  human waiting, step finish (outcome, target, diagnostic, artifacts and
  completed count), terminal finish, cancel of a waiting run and
  decide/resume; emitRun is a no-op without a hub.
- websocket: subscribe seeds current state before the ack, unsubscribe and
  connection teardown drop subscriptions, and pushed frames reuse the
  connection write closure.
- tests: protocol golden-JSON wire shape plus websocket coverage for the
  started-run frame sequence, subscribe current-state snapshot, silent
  unsubscribed/never-subscribed clients, workspace isolation and a
  bounded coalescing frame count.
- docs: note subscribe/unsubscribe/run_event in docs/cli.md; TUI and
  Android still poll in this release.
2026-10-17 00:43:05 +01:00
alan 865993b200 Rework omni search onto a typed search contract and add a TUI artifacts view
Protocol: replace the loose Query/Kind/Matches/Limit search fields with a Search request object and a SearchResults response, and have the server answer search from Store.Search over the shared run projections (runFrom/runStamps).

CLI: add the read-only 'pipelines search QUERY [--type KIND] [--limit N] [--json]' command with usage/help, and document the typed targets, ranking and bounds in docs/cli.md.

TUI: the '/' popup keeps type-to-search, up/down, Enter and Esc, renders one ranked list with kind badges and a bracketed (non-colour) match snippet, and Enter opens the matched pipeline, run, step, artifact or field; footer text updated to match.

TUI: add a dedicated run artifacts view ('A') with open/copy of recorded files, and move artifacts out of the run-detail File changes section (artifactCaptureNotes).

Tests: add Store.OpenReadOnly for read-only inspection of a live database plus pipeline/server/TUI search tests and artifacts tests, including env-gated live checks.
2026-10-16 23:48:38 +01:00
alan fae1005962 Allow launching runs with whole-run and per-stage model overrides
Add `pipelines run --model PROVIDER/MODEL` and repeatable
`--stage-model STAGE=PROVIDER/MODEL`, resolved with precedence
per-stage > whole-run > the model pinned in the pipeline version.

- A shared pure resolver (`pipeline.ParseModelOverride` /
  `pipeline.EffectiveModels`) owns the precedence and validation: every value
  must be a `provider/model` pair (`stage.ValidModel`), every stage key must
  exist in the pinned graph, an unknown key fails naming the valid keys, and a
  per-stage key that names a non-model stage is refused.
- The server validates against the pinned version before admission (no run row
  is ever inserted for an invalid request), persists `model`/`stage_models` on
  the run and `provider`/`model` on each step, applies the effective pair to a
  clone of the stage definition so the stage and its repair/formatter path share
  it, overlays the returned run graph with the effective pair (presentation
  only), and reuses the recorded overrides on retry.
- The run record echoes the overrides it was launched with and each step records
  the provider/model it actually ran with; the TUI Evidence pane renders that
  effective pair. The pinned pipeline version and `show --version N` are never
  modified.
- CLI help, `docs/cli.md` and the packaged authoring skill references document
  the flags and the precedence. A TUI override picker remains a documented
  follow-up; the CLI is the deliverable.

Tests: pure precedence/validation table tests, store round-trip and
request-key-mismatch plus migration, server admission/application/overlay/retry
with a controlled executor, CLI flag/help/malformed-value tests, and a TUI
evidence test.
2026-10-16 23:33:53 +01:00
alan bd09c55c47 Merge lane-go: titles, verdicts, human stage, line result format, stale-client guard
Integrates nine commits of reviewed work from the lane-go branch after the
automated resolver failed on this merge five times (each attempt dying on the
malformed-result defect whose fix is contained in this same merge).

Integration decisions:
- Search: kept main's shipped design (Store.Search + SearchResponse.Results) and
  dropped lane-go's parallel, older attempt (an unused Message.Matches field, an
  absent runs.search, and two partial TUI plumbing blocks). lane-go itself did not
  compile: protocol.SearchMatch was undefined and model.go referenced searchNav,
  requestSearch and applyRunNav, which exist nowhere. Those partial TUI blocks are
  removed; the queued TUI omni-search item will implement the surface properly on
  main's design.
- Both sides' additions are kept where they are independent: run titles and
  provenance, the human decision stage and its CLI/TUI surface, the derived stage
  verdict, the stale-client guard, the line-oriented result format, and main's
  search command.
- The search key's help label is now "/ omni search"; the two tests asserting the
  old label are updated rather than the label reverted.
- docs/cli.md and the packaged operations reference carry both sides' text.

Verified: go build ./... , go vet ./... and the full go test ./... suite pass.
2026-10-16 22:21:43 +01:00
alan aac1be18cc docs: document line-oriented model/agent result format 2026-10-16 21:49:14 +01:00
alan 8c1eea53a9 Report a stale client and refuse an unreadable newer server
Exchange a build/protocol identity so an older CLI can no longer talk to a
newer native server silently:

- protocol carries ProtocolVersion and every server frame (including the
  unsolicited welcome) is stamped with the server build and protocol, so a
  client that never pings still learns the server identity.
- buildinfo.Compare classifies the relationship as InSync/Warn/Refuse and
  names both builds; the client Watch reports one warning per connection and
  returns a hard error when the server speaks a newer, uninterpretable
  protocol so the caller never acts on the frame.
- client.ReceiveNotice decodes the raw frame so an unmodeled top-level field
  produces a one-time "unrecognized field(s)" notice instead of being dropped
  silently (the case that hid the artifacts field).
- CLI routes warnings to stderr, keeps machine JSON stdout clean and fails
  non-zero on a refused mismatch; ping/connection print the client/server
  relationship. The TUI surfaces the same notices through the shared log.
- scripts/setup-native.py now warns when an older `pipelines` earlier on PATH
  shadows prefix/bin/pipelines, naming both paths and versions, without
  installing or modifying anything else.
- Docs, the packaged debugging skill reference and README no longer imply the
  installed CLI always matches the server.

Tests cover the comparison table, unknown-field detection and caching, the
server identity stamping on pong and welcome, the CLI/TUI reporting paths and
the installer PATH shadowing warning.
2026-10-16 18:37:01 +01:00
alan a6aff47117 Add a human decision stage that pauses a run for an operator's choice
A new registered stage type `human` (internal/stage/human.go) has no model,
agent, tools or configs: its required `prompt` block carries the operator
instructions and its non-error edges are the selectable options. It parses,
validates, renders, exports and imports like every other type (no parser
special-case, no config keys).

At runtime the runner parks the run instead of executing: the step and the run
persist `waiting` with self-contained {prompt, options} evidence, and the run
returns without starting later stages, capturing file diffs, scanning artifacts,
holding a goroutine/timer or applying a stage timeout. The wait is durable: a
restart leaves the waiting rows untouched (RecoverRuns excludes them) and a
rebuilt runtime over the same database still accepts the decision.

`pipelines decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]` and
the TUI `c` picker resolve the decision from the pinned graph in one store
transaction, record the choice and feedback as the step's evidence and resume
along edges[KEY]. An unknown option is refused with the valid list, a stage that
is not waiting is refused, and a repeated decision for an already-decided visit
does not double-apply. Cancel and retry keep working while waiting and after a
decision; a human stage is never killed by a stage timeout or max_visits, and a
`notify` policy can fire an `on=waiting` event.

Docs: the authoring skill references (format.md, operations.md), docs/cli.md and
README describe the stage, the command and the TUI picker. Tests cover stage
validation, the parsing round-trip, the store transitions, a websocket wait/decide
integration, cancel/retry, waiting notifications and a persistence-layer rebuild
(second Open + newRunner over one database, no process restart).
2026-10-16 16:47:28 +01:00
alan c700b28e6a Add cross-system read-only search
One query now finds anything in the system over the wire and from the CLI.

- protocol: add Search/SearchResults message types and a SearchRequest
  (query, optional kind filter, optional limit) / SearchResponse (typed,
  ranked matches plus truncation and scan-bound counters) pair, with typed
  payloads for pipeline, version, run, step/visit, field text and artifact.
- store: add Store.Search, a read-only query over pipelines,
  pipeline_versions, pipeline_runs, run_steps (+step_control) and
  step_artifacts. Ranks exact name (3) > name substring (2) > body text (1),
  breaks ties by recency then a stable kind/target order, and reports the
  matched field. Bounds work with a 50-default/200-max result cap, a 2000-row
  per-kind scan limit and a 2000-step / 1 MiB field-text scan budget.
- terms: a bare status word or lone date expression (today, yesterday, this
  week, YYYY-MM-DD) selects runs; anything else stays free text.
- server: dispatch protocol.Search to stores[0].Search; read-only and
  unconfirmed.
- cli: add `pipelines search QUERY [--type KIND] [--limit N] [--json]`, help
  text and a BROWSE line.
- store: add OpenReadOnly so the live-database check never runs DDL.
- docs: document the grammar, kinds, ranking and bounds in docs/cli.md,
  scope the TUI omnisearch to pipelines in docs/ui-ux-review.md, and add a
  search bullet to the packaged operations reference.
- tests: protocol round trip, store/ranking/term-parsing/limit/scan-bound
  unit tests, a server end-to-end test, and an env-guarded live check.

No new dependencies; go.mod/go.sum unchanged.
2026-10-16 15:43:25 +01:00
alan f3e0593f4d Let runs carry an operator title shown in the CLI and TUI
Add a bounded, never-empty title to every run. `pipelines run --title "..."`
sets it explicitly and skips model generation; otherwise the server generates
it from the prompt through the shared summariser and falls back to the prompt's
first line, recording title_source (model/fallback/explicit) and title_model.

- protocol/store: RunSummary.Title plus Label(), a set_run_title message, title
  columns with an idempotent migration, CreateRunTitled, and SetRunTitle renames.
- summarise: Title/GenerateTitle/DeriveTitle/ExplicitTitle bounded to TitleCap.
- runner: generate the title outside the admission lock and only for a request
  that starts a run; a replay reuses the stored title and an explicit title
  skips the model entirely.
- CLI: `--title` on run and a `title <run-id> "<title>"` rename command; human
  `runs` and `inspect` show the title beside the pipeline name and version.
- TUI: the run list row, run detail header, Home run cards and last-run line,
  and the pipeline Results line show the title via Label().

Tests cover storage/migration/fallback, explicit-title and rename, the
end-to-end CLI flow, and title rendering; CLI and operations docs updated.
2026-10-16 13:36:48 +01:00
alan 9f88e4ff71 Recover malformed model-stage results with a bounded, work-preserving repair
A model stage whose final result was malformed failed the whole run even when
the work and evidence had survived, because the only recovery was an isolated
formatter call on a fixed 30-second deadline that could time out on its own
(run 55, ui_reviewer of pipeline 9 v6).

Add an ordered, deterministic local repair pass (internal/stage/result_repair.go)
that runs before any model call: (a) append missing closing delimiters via the
existing closeJSONContainers; (b) remove one unambiguous trailing comma before a
closing } or ]; (c) insert exactly the single comma/colon the JSON parser reported
at its byte offset. Every candidate must pass losslessSyntaxRepair(original,
candidate) and decodeModelResult before it is accepted, so no field value is
invented, removed, reordered or rewritten. Ambiguous payloads (an unescaped quote
or colon inside a string value, or a stray delimiter with no single completion)
return not-ok and fall through to the bounded model re-emit rather than guessing.
stageResultText still strips surrounding prose and a single code fence first;
model_prompt.go now also strips prose around a single balanced-but-malformed
object span so the deterministic pass sees just the object.

Replace the fixed 30-second formatter deadline with repairDeadline(payloadBytes,
remaining), which grows monotonically with payload size, has a floor, and is
clamped to the attempt's remaining budget minus a safety margin, so the stage
context remains the outer bound and a large result is not guaranteed to time out.
The one-request-per-attempt, 32 KiB bound and the deny-all, retained, new-session
property of RepairFormat are unchanged for OpenCodeClient and PiClient.

Record an explicit recovery status on every terminal path: repaired, partial or
unrecoverable, plus a byte offset. A candidate that is valid JSON but still fails
envelope validation is marked partial and never returned as a completed result;
the unrecoverable path retains the bounded original payload, the precise parse
error (including its byte offset) and surfaces the status in the FINAL RESULT
VALIDATION diagnostic.

safeToReplay is intentionally unchanged. The deterministic local repair and the
fresh deny-all, no-tools, retained formatter session are independent of the
stage's tool set and never re-prompt or replay the admitted agent session, so a
shell-enabled review stage gains malformed-result recovery without any replay of
admitted work. Risk: no change to mutation safety; the only added cost is one
bounded formatter call, already budgeted. Replayability is not extended for shell
stages.

Docs and packaged skill references (docs/model-results.md,
internal/author/skill/references/{format,debugging,verification}.md) describe the
three-tier recovery, the status words and the derived deadline; the safeToReplay
decision and its risk are recorded there alongside the commit.

Tests add mocked, offline coverage for the deterministic and partial paths, a
repairDeadline monotonicity/floor/clamp table, and a shell-stage regression
proving a recoverable malformed result does not fail the run or replay tool work.
2026-10-16 13:02:53 +01:00
alan 914345cb31 Move harness profiles between host and Docker and stop silent draft retargeting
Native installs left Docker-era harness profiles in harnesses.json, so every
authoring launch failed trying to reach a Docker service, and a retained draft
bound to the old Docker server was silently projected onto the configured one.

- harness: add Profile.Mode and VerifyDockerLink as the single mode/verification
  helpers. `harness setup --host` configures or converts a host profile and
  clears docker_directory; reconfiguring a Docker-linked profile re-verifies the
  recorded directory still owns compose.yaml and fails naming it instead of
  saving a link that cannot launch. Setup's summary and `harness list` (via the
  new List/ListView shape) report the resulting mode and, for Docker, the
  directory, without changing the stored Config schema.
- setup-native.py: after build/install-skill it writes host (no docker_directory)
  profiles for the detected harnesses, defaulting to the detected OpenCode profile
  rather than a stale pi default, taking models from --model or the existing valid
  profile and never inventing one; it clears a stale Docker link and preserves
  unrelated keys/profiles. Detection-only: no harness install, upgrade or download.
- draft: add Draft.Resolve, the shared mismatch helper, which projects a draft
  onto the configured server and returns a dual-URL notice. CLI validate/submit
  and the TUI draft browser use it so a foreign draft is never silently
  retargeted; the TUI pre-wraps the notice to the panel width.
- Tests cover the Docker->host and host->host transitions, the missing-directory
  failure, installer write/migration (Python), and the CLI/TUI notice, using the
  real config file path in a temp dir.
- Update help text, README, docs and the packaged authoring skill reference to
  describe host as the default and the Docker-linked case.
2026-10-15 19:53:10 +01:00
alan f57457aec9 Make agent-stage retries safe and retain timeout failure evidence
Fix four stage failure-handling and observability defects (A2, A3, A6, A9).

- A2: add safeToReplay(def) as the single definition of retryable post-admission
  work - only an explicit tools block that selects neither edit nor shell. The
  agent/pi WorkStarted guards now break only when the stage is not safe to
  replay, so read-only stages genuinely retry a post-admission timeout while
  mutating and legacy-bundle stages still never replay admitted work.
- A6: re-emit a malformed final result on an explicit read-only stage up to the
  retry budget, preserving format_repairs evidence; mutating stages keep the
  immediate return and their session.
- A3: timed-out stages retain usable evidence. waitForSessionIdle records one
  bounded sanitized OperationEvidence on deadline; the agent deferred cleanup
  does a bounded best-effort latest-assistant readback and settles knowable V2
  usage; RetainSession adds a retained_session pointer to failed step
  transcripts so the session is visible beneath the BoundTranscript threshold.
- A9: a stage outcome routed to #failure now yields a non-empty run diagnostic
  ("stage <name> returned outcome <outcome>") while preferring an existing step
  diagnostic.

Adds focused stage and server tests for retry safety, malformed-result
re-emission, timeout evidence retention and run-level diagnostics; updates the
packaged skill references and docs/cli.md, docs/model-results.md.
2026-10-15 18:27:22 +01:00
alan 0f5a188c6a Record and serve per-stage artifacts
Every agent stage now writes into a per-run/step artifacts directory under
the data dir (<data>/artifacts/<run>/<step>), passed to Pi as
PIPELINES_ARTIFACTS_DIR, exposed on the OpenCode client env seam, and stated
in the agent prompt boilerplate so agents save images, screenshots, reports
and logs there.

After each step the server scans that directory with a bounded, symlink- and
traversal-safe inventory (os.OpenRoot + Lstat + O_NOFOLLOW) recording name,
size_bytes, content type (extension then 512-byte sniff fallback) and URL,
enforcing max files, per-file bytes and total bytes while counting skipped
and truncated entries. A missing, unreadable or oversized directory yields
only a best-effort diagnostic and never changes the step outcome.

The list is carried on protocol.RunStep as Artifacts []StageArtifact plus
skipped/truncated counters (omitempty, backwards compatible), persisted in a
new step_artifacts table keyed (run_id, sequence) alongside the
notifications/file-changes tables and loaded by GetRun.

GET /artifacts/{run}/{step}/{name} on the existing handler serves the bytes
with http.ServeContent, so Content-Type, Content-Length, HEAD and Range
(206/416) come from the stdlib; unknown runs/steps, absolute paths,
traversal, symlink escapes and non-regular files are refused.

The TUI lists each selected step's artifacts (name + human size) in the
retained-evidence section and adds an artifacts summary in the style of the
file-changes summary. Adds docs/stage-artifacts.md linked from README and an
artifacts subsection to the packaged operations skill reference.

Tests cover the scan records and caps, missing/oversized directories,
HTTP HEAD/Range/traversal/symlink/unknown refusals and protocol round-trips.
2026-10-15 17:51:18 +01:00
alan 2618f0d244 Allow concurrent runs in different workspaces
Track each active run's resolved workspace alongside its cancel function
in a single runner.active struct map. A new request key is now refused
only when an already-active run is changing the same resolved workspace;
the refusal names the conflicting run id and advises waiting or using a
different workspace. Workspace resolution is unchanged (default cwd,
absolute request workspace, EvalSymlinks), so two spellings of one
directory still compare equal. The conflict check and registration stay
inside the existing r.mu critical section, so concurrent starts for one
workspace cannot both pass. Idempotent repeats of an active request key
and the restart-recovery guard are unchanged.

Add concurrency_test.go covering concurrent different-workspace runs,
same-workspace refusal, symlink/default-cwd equivalence, idempotent
repeats, restart recovery and retry workspace scoping; strengthen the
runner_test.go refusal assertion. Update docs/cli.md,
docs/first-version-testing.md and the packaged operations skill reference
to describe per-workspace concurrency.

Assisted-by: DeepSeek V4.1 Flash
2026-10-15 15:08:44 +01:00
alan 8adc6b8080 Bind the native server to the LAN so the phone client can reach it
The native install hardcoded -addr 127.0.0.1, so the server was unreachable from
the phone: the emulator worked because it proxies to the host loopback, but a
real device could not connect even though ntfy (bound to 0.0.0.0:8090) worked.

The bind address is now a manifest setting (server_address() in manage.py, with
older manifests still defaulting to loopback) and setup-native.py records it,
defaulting to 0.0.0.0 like the Docker install did and offering
--bind-address 127.0.0.1 for a loopback-only server. An invalid address is
rejected, and setup prints the phone client URL.

Verified: scripts test suite 116 tests OK, and after reinstalling the server
listens on *:8080 and answers {"healthy":true} on 127.0.0.1, 10.0.0.75,
10.0.0.249 and the tailnet address.
2026-10-15 01:57:27 +01:00
alan a70e3aef4b Install Pipelines natively by default on macOS and Linux
Add scripts/setup-native.py and make ./install (and --native) build the server, the CLI
and ntfy from source into ~/.local/share/pipelines, detect an existing host OpenCode
and Pi instead of installing them, install the packaged authoring skill, migrate an
existing Docker database and ntfy auth store, save the host client default and start
both services. ./install --docker keeps the previous Docker server + ntfy
installation unchanged.

ntfy is compiled from its pinned Go module on every host because upstream publishes
no macOS server binary (its darwin asset is built with -tags noserver); its cgo
SQLite driver is why a C compiler is now required natively. The ntfy credential
bootstrap and the client default writer are shared by both installers,
installation.Load resolves symlinked prefixes, the Docker migration copies the
SQLite write-ahead log so the newest definitions and runs are not lost, and the
account subcommands read the generated configuration through NTFY_CONFIG_FILE.

Verified on macOS: 115 Python tests, go build/vet/test, and scratch installs that
migrate 9 pipelines and 31 runs, build ntfy v2.30.0 and start both services healthy
on spare ports before stopping them.
2026-10-13 17:20:14 +01:00
alan 9b305df725 Record final live template and catalog deployment verification 2026-10-12 23:17:29 +01:00
alan 143f237268 Strengthen template planning, reuse and independent review 2026-10-12 23:10:37 +01:00
alan 03ed4e02e4 Add real workflow and execution screenshots to README 2026-10-12 22:25:10 +01:00