archived_at on pipelines (migrated in place). archive_pipeline and
unarchive_pipeline by id or name with an expected-version guard;
list_pipelines hides archived unless include_archived. Archived
pipelines refuse runs, retries, new versions and name reuse, and
cannot be archived while an enabled schedule points at them.
CLI archive/unarchive and list --archived/--all, docs and skill note.
Each recorded artifact now carries a server group (deliverable or debug).
public/ files are deliverables; input/memory/history/output.md, session
exports and private/ scratch are debugging files. The app's Artifacts tab
lists deliverables and the Debug toggle reveals a separate Debugging files
group (older servers fall back to the filename rule); the TUI lists
deliverables first and marks debug rows; artifact search excludes debug
files unless include_debug / --debug-files is set.
Every stage used to be told to read memory.md, the full output of every
earlier turn (19 KB average, 92 KB max), and paid for it on each model call.
memory.md is now essential memory, hard-capped at 4 KB and inlined in the
prompt: the task, the previous stage's # handoff section (or its truncated
output head), a one-line-per-stage outcome ledger and the server-captured
changed files. history.md keeps the old full memory on disk; read-capable
stages are pointed at it and input.md for on-demand detail, and it is never
inlined. The prompt asks each stage for a short # handoff section.
The server skip fallback for stages with a skipped edge now applies only
when every earlier visit recorded a complete file-change capture with no
changed files and no image artifact. Any change (no repo-specific UI path
or extension list), any partial/unavailable/missing capture, a human
decision or a checkpoint-retry prefix runs the model. auto_skip=false opts
a stage out. The fallback verdict names the server fallback.
Capture compares over-budget text by hash and, in Git, lists paths it
could not compare when their git status changes, marked no_patch, so a
large workspace no longer records files: [] for a real edit. Run detail
gains a warnings list for partial/unavailable captures, also printed by
inspect on stderr.
chat_sessions rows carry activity (turn_running, background_tasks, seq) from
the hub's own state, and chat_activity is pushed to every connection when a
session's turn starts or ends or its running background-task count changes.
A room whose last viewer left stays observed while background tasks run,
bounded by the unattended limits; a Claude Code process exit ends its tasks.
The template catalog is now Example 1-4: one stage, two stages (handoff),
feedback loop (runner-owned limits, retries, timeout) and human decision
(type=human approve/revise/abort with notify hooks). All read-only or leave
changes uncommitted; none commits, pushes or deploys.
A brand-new database (schema created by this Open) gets the examples once at
server start, bound to the configured default model and harness. A
server_settings marker records the decision, so removed examples never come
back and existing installs are marked without any pipeline being added or
rewritten.
Chat hits sort by the chat's last use whatever field matched, and run,
step, field, artifact and pipeline hits sort by their own date, before
the result cap applies so the cap drops the oldest. Each kind's row scan
also walks newest first. Score still reports the match tier. The app
sorts both groups newest first as well.
A search with kind chat matches the operator-session registry's titles and
directories and the newest messages of recent transcripts through the chat
broker (bounded count, concurrency and one-second budget), answered off the
connection's read loop. The all-kind search is unchanged for older clients.
The app sends the chat search after its run search and shows a Chats group
with title, harness, directory and snippet; a chat hit opens that chat in the
Chat tab. The probe fixture answers chat searches over ?chats=N.
Usage frames carry optional usage.account {harness, source, plan, as_of,
windows[{id,label,used_percent,resets_at}], unsupported}. Claude Code quota
comes from its rate_limit_event stream lines and the initialize account
(plan), shared across chats on the same account. Codex reads
account/rateLimits/read once per connection (re-read when a chat opens after
5 min) and merges account/rateLimits/updated. OpenCode exposes no quota API
and says so. The app's chat info sheet shows an Account usage section with a
progress bar and reset countdown per window, Material and Cupertino.
Server: list_directory returns a directory_listing (path, home, parent,
entries with is_dir/is_git_repo/hidden, truncated) confined to the server
user's home subtree after cleaning and resolving symlinks, refusing pseudo
filesystems and capping entries at 2000. Documented in docs/cli.md.
App: "Choose folder…" in the create dialog opens a picker (full-screen page
on phones, large dialog on wide windows, Material and Cupertino) with
breadcrumbs, filter, Show hidden, git badges, dimmed files, recent folders,
inline errors and "Start chat here". The probe app gains a folder fixture
for captures.
Claude Code (stream-json stdin) and Codex (turn/steer pinned to the active
turn) receive a prompt sent while a turn runs immediately. Other harnesses
(OpenCode) get a durable chat_prompt_queue row owned by the session: it
survives the last client detaching and a server restart, is delivered in
order when the turn ends with or without a client, and is deduplicated by
operation id. chat_subscribe carries the queue, chat_queue frames report
delivered/removed/failed changes, and chat_unqueue removes a waiting prompt.
An exhausted export recovery job left its run parked forever in the
export wait, reserving the workspace and refusing later scheduled runs
(runs 221/226: OpenCode sessions above the 8 MiB export bound are
always truncated, so every retry stays partial).
A gate is now releasable when its job is exhausted: the visit keeps its
partial/unavailable evidence and diagnostic (still repairable), the run
continues to its next stage or terminal, and the workspace is freed.
Gates older than 15 minutes with a pending, lease-expired or missing job
are exhausted by a bounded wait. The worker reconciles on its first
tick after startup and every 30 seconds, so existing wedged runs recover
after a restart. Truncation diagnostics now name the size bound.
Record every notification published to ntfy (run outbox, stage notify
hooks, chat question/approval/reply) in a persisted app_notifications
outbox and stream it to subscribe_notifications clients with an after
cursor; clear_notifications and ack_attention clear across devices, as do
runs leaving waiting, deleted runs and answered chat questions. Add the
notify.ntfy_enabled setting (default on) to stop only the ntfy publish.
Every run summary carries size_bytes (artifact tree, server-created Pi
session directories and an estimate of the run's database rows), cached in
pipeline_runs.size_bytes, invalidated by triggers, measured when a run
finishes and backfilled in the background. delete_runs removes terminal
runs with every scoped row and server-owned file, refuses active ones with
a reason, and pushes runs_removed to subscribers. pipelines runs rm is the
CLI form; retention apply now shares the complete row deletion.
Auto runs no longer alert on failure or success; only notify_mode on does,
unless the run_finished setting (now 'Notify when any run ends', default off)
opts into every run end. Waiting still notifies auto and on runs. The setting
is stored under a new key so a value saved under the old meaning is ignored.
Later visits' input.md held only the previous stage's output, so the
operator's task reached only the first stage; when a Prepare stage
summarised (paths, HEAD), Code never saw the task or its acceptance
checklist and either blocked (run #202) or did part of it (#195, #197,
#198). Later inputs now carry the run prompt under '# Run task' (headings
demoted) before '# Previous turn output'.
Each model visit now gets <run>/stages/<seq>-<stage>/ with input.md,
memory.md, output.md, private/ and a shared run-level public/. The model
reads the two input files, follows the stage's instructions verbatim and
returns complete markdown; only the # outcome section is parsed.
- add internal/turn layout, render and output parsing
- write input.md/memory.md before a visit and output.md from the final
message; inventory all turn files as stage artifacts
- accumulate completed turns and system-written public file additions in
memory, including across retries and resume
- remove the JSON envelope, strict field validation, line format and the
repair ladder for model stages
- republish pipeline authoring prompts and docs in the new style
Checks: go build ./..., go vet ./..., go test -count=1 ./..., gofmt -l .
Drop the resource allocation feature end to end: store tables and migration, runner scheduling and cleanup, protocol types and commands, CLI commands, TUI screens, tests and docs. Existing databases open cleanly and definitions carrying a retired resources block still validate.
The operator's chat list showed ~99 stray "New chat" entries. The listing
was already fail-closed (registry-only); the fault was what got recorded: an
automated UI check drives the real page, so its throwaway chat was recorded as
an operator chat exactly like a manually created one and stayed in the list for
ever (each run added one).
Server: chat_create takes an optional ephemeral flag. The session is created,
bootstrapped, usable, closable and reopenable exactly like any other, but its
registry row is hidden, so chat_sessions never lists it. operator_sessions gains
an ephemeral column (idempotent migration, existing rows default to listed) and
the list query filters it; the single-row lookup does not, so close/reopen keep
working. RecordVerificationSession is the hidden recorder.
Web: the page marks a create ephemeral only when automation asks for it
(__pipelinesEphemeralChat, set by the UI smoke for its throwaway chat), so the
smoke's own runs stop polluting while a manual create is unchanged.
Tests: a hidden row is never listed yet is still looked up and closable; a
normal re-record of the same id lists it again; a pre-ephemeral database
migrates with every existing chat still listed. Live data pruned separately
(107 -> 8 registry rows, backed up).
A session was owned by one connection: the claim map in chat_claims.go let the
first client hold it, every other client saw the session Busy and read-only, and
a second client's prompt was rejected. Replace the claim with a per-session
"room" so the operator's phone, tablet and terminal all watch and write the same
conversation, and a pending approval is a shared, exactly-once object.
Server (internal/server, internal/protocol):
- New chat_room.go: one room per session holding the session's single broker
stream, its one bounded prompt queue (8) and the fan-out to every attached
client. A chat_subscribe frame attaches a connection to a session's shared
stream and replays the still-unresolved approval requests the room holds.
- chat_stream.go: handleChatPrompt accepts a prompt from any client. It attaches
the sender, either starts a turn or queues the prompt behind the in-flight turn
and acknowledges it with chat_queued:true ("queued" is the chosen behaviour,
stated on the wire and in docs/cli.md), and echoes the user turn to every other
client attributed to the sender's label. The prompt handler no longer owns the
stream; one turn runner per room drains the queue, so two turns never
interleave. Per-connection write serialisation is unchanged (each chatSub has
its own writer goroutine and bounded buffer).
- chat_permissions.go / chat_commands.go: an approval is reserved by the first
answer and confirmed once delivered; the first answer wins, is recorded with
the answering client's label, and the resolution is pushed to every watching
client as chat_permission with resolved/decision/resolved_by so their cards
clear. A later answer is refused with an error naming the client that answered,
and never reaches the agent twice; a reservation whose reply failed is
released.
- chat_hub.go / chat_sessions.go / chat.go: the hub now owns rooms and the
user-turn id sequence; the resulting user turns are retained (role and client
carried) so a reconnect re-syncs who said what. chat_sessions no longer marks a
session Busy and its clientID parameter is gone; the protocol keeps the Busy
field for wire compatibility only. websocket.go detaches a connection's chat
subscriptions when the read loop ends.
- chat_claims.go is removed, not left vestigial: the only exclusive thing left is
a turn, and the room's queue is that.
App (android):
- PipelinesRepository carries a stable per-device client label (Build.MODEL, with
a fallback) on prompts and replies, and adds chatSubscribe; AppViewModel
subscribes on opening a chat (best-effort, an older server's error frame does
not stop history loading) and records the queued ack.
- ChatStore folds a user frame from another client into the transcript once,
attributed by the sender's label, and never receives its own prompt back, so
the sender's own message is never duplicated. ChatThread.readOnly is gone:
canSend no longer depends on another client, the composer says "Queued - it
will send when the current turn finishes." instead of disabling input, and the
status label is Running/Idle only. Scroll/follow behaviour is untouched.
- Tests: ChatProtocolTest round-trips the new wire fields, ChatFormatTest asserts
the stale Busy flag cannot produce a read-only label, MultiChatStateTest asserts
a session another client watches stays writable.
Reflected where it belongs: no packaged template changes (no bundled starter
embeds the chat protocol). Checked the packaged authoring skill references: none
of them describes the chat wire beyond operations.md, and
internal/author/skill/references/operations.md is now stale - its "Brokered
OpenCode chat" section still says chat_sessions marks "busy" when another client
holds the session and that a second client's chat_prompt is rejected, and it does
not mention chat_subscribe, the per-session prompt queue/chat_queued, the
per-client label or the shared exactly-once approval (it also still describes the
per-subscription permission poller, which the room's single poller replaced). The
supported semantics are the ones in docs/cli.md, updated in this commit; the
operations.md section needs a rewrite to match. No README or other shared
document was edited here.
Checks run on the committed tree: gofmt -l internal cmd examples (nothing), go
build ./..., go vet ./internal/... ./cmd/..., go test ./internal/server
./internal/protocol ./internal/stage ./internal/pipeline.
Carry the usage a model stage already measures for its budget
(stage.Budget.Snapshot()) onto the step as typed evidence and total it in
one pass, rather than a second harness read, a transcript parse or an
output-length estimate. A shared budget across stage revisits records only
the visit's increment, and a revisit that observes no new report records no
usage instead of a fabricated known zero.
- protocol: StageUsage/RunUsage/TotalUsage and UsageTotals.
- pipeline: step_usage table + migration, COALESCEd into the one step
projection, FinishStep persistence, one json_extract history aggregate,
retention removes the rows with their run.
- server: stageUsage maps the budget snapshot, stamping per-stage model and
usage for every model stage (including failures).
- cli: inspect/stats human and JSON output.
- budget/pi: retain the reported in/out split, mark a total-only (Pi) report
split-unknown, and keep an unreported cost unknown (never zero).
Evidence and docs updated in the packaged skill references and docs/cli.md;
the live database has no step_usage rows yet (all history predates this), so
per-run figures are reported as unavailable, not zero.
Add an opt-in way to broker chat from an operator-supplied OpenCode service
instead of the instance the server spawns for itself, and a way to adopt a
session that already exists on that endpoint so it appears in chat_sessions and
can be continued from the app.
- Endpoint selection: the `-opencode-endpoint` server flag (defaulting to
`PIPELINES_OPENCODE_ENDPOINT`) opts in; precedence is explicit flag >
explicit environment > the endpoint recorded on a previous run, with the
spawned instance as the default. opencode.OpenEndpoint validates the
reference once (http(s), host, no userinfo/query/fragment/path), detects the
backend from its own /api/info (v2) or /global/health (v1) contract, and
requires a streamable text/event-stream event path. It owns no process
(Close is a no-op) and never calls startServe; every failure names the
endpoint and disables chat rather than silently falling back. An
authenticated endpoint takes an optional password only through the private
file PIPELINES_OPENCODE_ENDPOINT_PASSWORD_FILE, with
PIPELINES_OPENCODE_ENDPOINT_USERNAME overriding the detected default.
- Persistence: the validated choice is recorded in an additive single-row
server_settings table so a restart with no flag/environment reuses it; an
unreachable recorded endpoint disables chat with a clear message and no
fallback.
- chat_adopt: a new protocol request (ChatAdoptRequest) with handleChatAdopt in
internal/server/chat.go. It refuses a nil broker/registry/payload and a
blank or foreign-shaped id before any endpoint call, reads the endpoint via
the shared Chat.Session (which reuses Sessions/decodeChatSessions), records
through the existing RecordOperatorSession with the endpoint's directory and
title verbatim, and echoes the adopted session plus its registry row like
chat_create. A missing id is refused naming the id and records nothing; an
endpoint record with an empty directory is refused rather than substituting
the server's working directory or home.
- CLI: `pipelines chat-adopt <session-id> [--json]` (usage, notes, help
inventory, dispatch) with docs/cli.md updated under Chat sessions.
- Autostart preserves the endpoint variables (and the password-file reference,
never an inline secret) via serviceEnvironmentKeys.
Checks before committing:
- Package location: the change is user-facing chat behaviour, so the packaged
authoring skill reference internal/author/skill/references/operations.md is
updated in the same change (Brokered OpenCode chat section) with a dated
verification-evidence entry and the not-deployed state. No new reference file
is added, so internal/author/install.go's managed inventory is unchanged.
- Packaged templates: this repository packages no template set for this
behaviour and none references the chat endpoint, so no template needs
updating.
- Verification: `go build ./...`, `go vet` on the touched packages, `gofmt -l`
clean and `go test ./...` all pass. The unit tests cover the endpoint
validation/precedence, the adopt happy path and refusals, the recorded
endpoint round-trip, the protocol round-trip and the CLI dispatch. The live
scratch-server check against the operator's real endpoint was not run in this
change.
Nothing unrelated is staged; docs left stale (README and other shared documents)
are not edited here and are for a single folding writer.
Fix two honesty gaps in run-history retention that the implementer stage
left uncommitted on top of 14175eb:
- Populate RemovedRuns, RemovedRunCount, RemovedArtifactFiles/Bytes and
Applied right after the row transaction commits, before staging cleanup.
A cleanup failure must not report a successful row/artifact removal as
zero removed, and staging leftovers are excluded from SpaceReclaimedBytes.
A failed VACUUM after removal now reports the artifact bytes actually
removed while claiming no database pages.
- Stage artifacts through os.Stat instead of os.Lstat so a symlinked
artifacts root is followed: the report sized that directory, so apply
removes through it too instead of counting bytes it never deleted.
Tests cover post-commit staging-cleanup failure reporting, staging through a
symlinked artifacts root (including no staging directory left in the link
target), and an honest reclaim failure after removal.
Checked documentation surfaces:
- docs/cli.md: measured example refreshed with the current before/apply/after
output and the symlink/staging/VACUUM reporting described accurately.
- README.md: retention prose (explicit, never automatic, protected runs,
artifacts go with the run) is unchanged and still correct.
- internal/author/skill/references/operations.md: operator-only retention
summary describes the same behaviour; no user-facing surface changed, so
no reference update was needed.
- Packaged templates: retention is a CLI/store capability with no template
change, so nothing to update. internal/author/install.go inventory
unchanged (no reference added or renamed).
Verification: gofmt clean on the touched Go files, git diff --check clean,
go vet ./internal/pipeline clean, and go test ./internal/pipeline -run
'TestRetention' -count=1 passing.
Replace the Android app's one-second run poll with the server's
workspace-scoped run_event pushes. The runs list, each pipeline row's
last-run line, the run header and the run detail (status, timeline,
stages, artifacts) now advance from the socket instead of polling or a
manual refresh.
- Wire: Message.kt mirrors internal/protocol/runevents.go (subscribe,
unsubscribe, run_event; RunEventFrame/RunStepDelta). The frames are
unsolicited and already arrive on the client's events stream;
PipelinesRepository.subscribeRuns/unsubscribeRuns add the typed
command, sent via a new PipelinesClient.sendOnly that does not hold the
send lock waiting for the running server's unreliable subscribe ack.
- One pure fold: ui/RunEvents.kt folds a frame into the list (in place,
order preserved) and the open detail with one shared status ladder and
one shared run-field merge. It is monotonic (a stale or duplicate
delta is dropped whole), reports needsResync for a run the list does
not hold instead of inventing a row, and only requests a full get_run
for a settled step or terminal run.
- Wiring: AppViewModel subscribes once per (re)connect and workspace;
every (re)subscribe re-reads the list and open detail once, because the
deployed server's subscribe snapshot is partial, then folds the pushes
on top. Re-reads are push-driven, coalesced and keyed so no push is
dropped and no timer remains. LiveBadge (RunStream.kt RunStreamState)
shows honestly live/reconnecting/not live, confirmed by the ack, a
push or a successful read and dropped when a read fails.
- ui/RunEventsTest.kt pins the delta sequence as pure cases: created run
not in the list, step started, step finished, run finished, unknown
run, duplicate/out-of-order deltas, and a reconnect resync.
- android/README.md records the change, the wire mapping and the
observed limitation of the running server (partial snapshot, no ack
within 30s), which the client tolerates without inventing a stream.
Checked where the change belongs: android/README.md gained this entry;
docs/cli.md described the Android app as still polling, which this makes
stale, so that sentence now says the app consumes the pushes while the
TUI still polls. The packaged TUI pipeline templates (internal/tui) do
not touch run-state refresh and need no change, and the packaged
authoring skill references (internal/author/skill/references) document
the authoring workflow, not the Android client's run updates, so none of
them needs updating.
Reviewed and approved by the run's review stage (build, unit tests,
Kotlin compile and lint in android/README.md); no merge/rebase/cherry-pick
was in progress and no conflict was present.
chat_sessions used to list exactly the OpenCode sessions the pipelines own,
filtered through pipeline.SessionOwner (run_steps/run_session_owners), so an
operator-created chat was an accident of the run-ownership table and the
operator's working sessions were mixed with the internals of running pipelines.
Add a first-class operator session:
- internal/pipeline/operator_sessions.go: a dedicated operator_sessions table
(session_id PK, directory, title, tag, created_at, last_used_at) with
RecordOperatorSession (upsert preserving created_at), a one-statement
TouchOperatorSession (no-op for a non-operator id), OperatorSessions bounded to
128 rows most-recently-used first, and OperatorDirectories (bounded MRU,
deduped). run_steps/run_session_owners and SessionOwner/OwnedSessions are
untouched.
- internal/server/chat.go: chat_sessions returns operator sessions only - it
reads the registry once, intersects it with one broker session-list call, fills
directory/tag/created/updated from the registry, sets busy for a session held
by another connection, keeps MRU order, and stays fail-closed (empty list, never
the endpoint's whole list). A pipeline stage session and an untagged row can
never be listed. chat_create resolves the directory (empty/'~'/'~/...' ->
operator home; relative or missing rejected before any OpenCode call), calls
broker.Create(title, resolved) and records the row with the operator tag.
chat_directories returns home first then bounded (8) deduped recents. A
successful prompt touches the operator session without affecting pipeline
prompts.
- internal/opencode, internal/protocol, internal/server/websocket.go and
message.go: unscoped session list, Create posts {"title":...,
"location":{"directory":...}} to POST /api/session, additive protocol fields
(omitempty) and the ChatDirectories request/response, plus the OperatorRegistry
option defaulting to the first store. One OpenCode transport per server is
unchanged.
Docs and references: docs/cli.md gains a "Chat sessions" section documenting the
operator-only fail-closed list, directory defaulting/rejection, chat_directories
(home first, deduped, bounded 8) and why a dedicated request was chosen.
internal/author/skill/references/operations.md's stale "Brokered OpenCode chat"
text (chat_create unbrokered, run_steps filtering, "listed the owned scratch
session" evidence) is rewritten to the operator-session registry, the fail-closed
filter, directory defaulting and the corrected live-test wording. No new
reference file, so internal/author/install.go's inventory is unchanged.
Checked for staleness elsewhere: README.md, the docs/ topic pages and the
packaged server/TUI templates carry no chat-session ownership claims, so none
needed updating.
Verified: gofmt -l clean; go vet and go build ./... clean; go test -count=1
./internal/pipeline/... ./internal/protocol/... ./internal/opencode/...
./internal/server/... and the full go test -count=1 ./... pass.
Combines lane-go's typed omni search, launch-time model overrides, TUI
artifacts view, run-state websocket push, and the "reflect the change where
it belongs" committer rule with the Templates gallery, chat/omni-search and
other reviewed work already on main. The 12 files both sides touched keep
both intents composed rather than dropping either side: the omni search keeps
the typed search contract (CLI `pipelines search` and the TUI popup with type
badges and marked snippets) alongside the run-list quick filter; cli.md and
ui-ux-review.md document both the search rework and the artifacts view;
orchestration.md, harnesses.md and operations.md keep the model-override and
reflect-where-it-belongs guidance; and the protocol/CLI/server/TUI code keeps
the new search, run-event, override and artifacts paths alongside the existing
run, title and help surfaces.
Checks before committing:
- go build ./... -- pass
- go vet ./... -- pass
- go test -count=1 ./internal/... ./cmd/... -- all pass
Doc check before committing: nothing needs updating. The lane's own docs and
references (docs/cli.md, docs/ui-ux-review.md, and the orchestration,
harnesses and operations skill references) already carry the new features.
format.md and the packaged templates already describe the line-oriented result
format and the human decision stage from the earlier lane-go merge. The root
README already describes the omni search popup. Model overrides, the search
command, run events and the artifacts view are runtime/CLI/TUI surfaces that
cli.md documents; no template, README or topic-doc edit is required by this
merge.