Commit Graph

284 Commits

Author SHA1 Message Date
alan 88dfc051e5 Model stages: outcome-first output shape and one in-session outcome repair
The prompt now says output.md must start with # outcome and the outcome
word alone on the next line, then # handoff, then detail. Runs 241/243
followed the old "start with # handoff" wording and dropped the outcome.

When a completed turn lacks a valid outcome, the OpenCode/agent and Pi
clients send one short follow-up in the same session asking only for the
outcome line. An outcome-only reply goes at the top of the original report;
a still-invalid reply fails with the same validation error as before. The
follow-up's tokens accrue to the session usage. No outcome is inferred.

Missing tools/evidence now name blocked when declared, else error. The
parser tolerates a trailing period on the outcome word.
2026-11-04 17:20:58 +00:00
alan 70391b17c2 Stage diffs: clean Git files take their baseline from the index blob
On a large Git workspace the 32 MiB retained-text budget was spent
alphabetically on files that never change, so small late edits were
recorded with no_patch. The stage-start snapshot now keeps text only for
files dirty or untracked at start; a clean tracked file keeps its hash and
stage-0 index blob id. When it changes or is deleted, the after capture
reads that blob in one `git cat-file --batch` and uses it only if its
SHA-256 matches the start read, so stage commits/resets do not move the
baseline and filters or assume-unchanged never yield a wrong patch. The
after capture shares unchanged files' representation instead of spending
budget on them again.

FileChange gains no_patch_reason (binary, file_size, budget, lines,
evidence_size, git_status, baseline). Non-Git workspaces keep their
behaviour.
2026-11-04 17:08:01 +00:00
alan 6870c81960 Skill install: also refresh Codex's copy under $CODEX_HOME/skills 2026-11-04 16:43:55 +00:00
alan a627b38ef3 Merge main; use the shared chatShutdownDetacher for OpenCode and Codex
OpenCode and Codex chats implement Detach (called by chatManager.Close)
instead of a parallel DetachSessions interface. Docs follow the Claude Code
host and the systemd KillMode=process unit. The opt-in live restart check
copies only the stage provider's login row into its scratch OpenCode
database, builds the CLI for skill install and removes its scratch tree.
2026-11-04 16:24:17 +00:00
alan aa8eddd686 Keep OpenCode/Codex chat and stage sessions running across server restarts
A stopping server no longer interrupts harness sessions. Runs are canceled
with a shutdown cause, so stage clients leave admitted sessions running and
owned; start-up recovery re-adopts live OpenCode V2 stage sessions (wait,
read back the result, continue the graph), restarts unadmitted visits,
continues runs between stages, and falls back to interrupted with a reason
when the harness is gone. Chats with live work are recorded on shutdown and
reattached on start; turns that ended during the downtime are folded in
from history. The server-spawned V2 chat serve is now a detached owned
service. PIPELINES_SHUTDOWN_SESSIONS=stop restores the old behaviour.
2026-11-04 16:17:08 +00:00
alan 6b22389cae Merge: Claude Code chats survive server restarts via a detachable host 2026-11-04 16:15:06 +00:00
alan 34c2ccec25 Install and document pipelines-claude-host alongside the server 2026-11-04 16:14:35 +00:00
alan c810437aa5 Claude Code chats survive server restarts in a detachable host
A pipelines-claude-host process owns each claude -p CLI and its pipes,
spools its stream-json output with offsets and serves a private Unix
socket. The server detaches on shutdown and reattaches on start,
replaying the spool to rebuild turns, approvals, questions, background
tasks and the live model. Explicit close, idle release and CLI exit stop
the host; a crashed host falls back to --resume with a notice.
2026-11-04 16:09:56 +00:00
alan 9351cd7d74 Merge: pipeline stages run on their own OpenCode service with stage-only limits 2026-11-04 14:12:38 +00:00
alan b655278674 Run OpenCode stages on a Pipelines-owned stage service with their own limits
Stages now run on a detached Pipelines-owned opencode serve started with
OPENCODE_CONFIG_CONTENT carrying stage tool_output (16384 bytes / 400 lines)
and compaction.auto defaults, configurable through PIPELINES_OPENCODE_STAGE_*.
Chats keep their existing service and configuration; a server-spawned chat
serve can take PIPELINES_OPENCODE_CHAT_CONFIG. The service shares the user's
OpenCode config, credentials and session database, keeps a stable private
registration so retained references (recovery, interrupt, export) stay valid,
revives on the same port after a crash, replaces itself when idle after a
settings change, and is stopped on shutdown only when idle.
PIPELINES_OPENCODE_STAGE_SERVICE=shared restores the previous behaviour.
2026-11-04 14:04:35 +00:00
alan 3c31bca87f Merge main; align lint and skill practices with essential memory/history and the four graded examples 2026-11-04 13:08:47 +00:00
alan d7e54f4612 Merge: short essential memory per stage; full history on demand 2026-11-04 13:01:16 +00:00
alan 538143554c Split stage memory into capped essential memory.md and full history.md
Every stage used to be told to read memory.md, the full output of every
earlier turn (19 KB average, 92 KB max), and paid for it on each model call.
memory.md is now essential memory, hard-capped at 4 KB and inlined in the
prompt: the task, the previous stage's # handoff section (or its truncated
output head), a one-line-per-stage outcome ledger and the server-captured
changed files. history.md keeps the old full memory on disk; read-capable
stages are pointed at it and input.md for on-demand detail, and it is never
inlined. The prompt asks each stage for a short # handoff section.
2026-11-04 13:00:48 +00:00
alan e5c015f6ed Merge: no UI-review auto-skip without complete change capture; capture lists over-budget changes
# Conflicts:
#	internal/protocol/runs.go
2026-11-04 12:58:45 +00:00
alan f16d61cdfa Skip fallback needs complete, empty evidence; capture lists over-budget changes
The server skip fallback for stages with a skipped edge now applies only
when every earlier visit recorded a complete file-change capture with no
changed files and no image artifact. Any change (no repo-specific UI path
or extension list), any partial/unavailable/missing capture, a human
decision or a checkpoint-retry prefix runs the model. auto_skip=false opts
a stage out. The fallback verdict names the server fallback.

Capture compares over-budget text by hash and, in Git, lists paths it
could not compare when their git status changes, marked no_patch, so a
large workspace no longer records files: [] for a real edit. Run detail
gains a warnings list for partial/unavailable captures, also printed by
inspect on stderr.
2026-11-04 12:56:40 +00:00
alan e727c3fd07 Merge main into run token usage branch 2026-11-03 22:21:38 +00:00
alan 3227d61caf Token usage: keep cache/reasoning split, count only recorded steps, cost only when positive; captures fixture, docs and notes 2026-11-03 22:16:49 +00:00
alan 8c7093d1e6 Merge: chat list shows running turns and background tasks for every chat 2026-11-03 19:54:35 +00:00
alan 1b98877d88 Chat list activity: turn running and background tasks on every row, pushed to every connection
chat_sessions rows carry activity (turn_running, background_tasks, seq) from
the hub's own state, and chat_activity is pushed to every connection when a
session's turn starts or ends or its running background-task count changes.
A room whose last viewer left stays observed while background tasks run,
bounded by the unattended limits; a Claude Code process exit ends its tasks.
2026-11-03 19:45:43 +00:00
alan a48e4a5b6a Replace starters with four graded examples seeded into new databases
The template catalog is now Example 1-4: one stage, two stages (handoff),
feedback loop (runner-owned limits, retries, timeout) and human decision
(type=human approve/revise/abort with notify hooks). All read-only or leave
changes uncommitted; none commits, pushes or deploys.

A brand-new database (schema created by this Open) gets the examples once at
server start, bound to the configured default model and harness. A
server_settings marker records the decision, so removed examples never come
back and existing installs are marked without any pipeline being added or
rewritten.
2026-11-03 19:35:47 +00:00
alan 7ecc12b747 LAN discovery: server advertises _pipelines._tcp over mDNS; app finds servers on the connect screen
Server: internal/discovery publishes a DNS-SD record (hashicorp/mdns) with
the port LAN clients dial (the web proxy via -mdns-port, which the
supervisor passes, or a non-loopback -addr) and TXT v, name, path, tls, auth.
On by default; -mdns=false or PIPELINES_MDNS=off disables it. Failures are
logged and never block startup; the advert is withdrawn on shutdown.

App: the Connect to Pipelines screen gets 'Find servers on this network'
behind a ServerDiscovery interface (Bonsoir: Android NSD with multicast lock,
iOS Bonjour with NSLocalNetworkUsageDescription and NSBonjourServices).
Tapping a server fills the address and connects; manual entry stays. Web has
no scan.
2026-11-03 14:54:54 +00:00
alan ad4233f7b8 Chat: /new continues the same chat on a fresh session
The composer's /new used to create a separate chat. It now sends
chat_new_session. The server starts a fresh native session on the chat's
own harness (OpenCode, Codex or Claude Code) and re-keys the
operator_sessions row in one transaction. The chat keeps its title,
directory, backend, colour, star, model and mode. The old row is kept
unlisted with replaced_by, and its native session stays resumable.

The reply and the chat_session_changed push carry previous_id, so every
client swaps the row in place and an open conversation follows it to the
fresh, empty transcript with a notice.
2026-11-03 14:06:44 +00:00
alan 6ca415c134 Search: order results newest first in every group
Chat hits sort by the chat's last use whatever field matched, and run,
step, field, artifact and pipeline hits sort by their own date, before
the result cap applies so the cap drops the oldest. Each kind's row scan
also walks newest first. Score still reports the match tier. The app
sorts both groups newest first as well.
2026-11-03 13:35:27 +00:00
alan bff7115e4e Search: find operator chats by title, directory and message text
A search with kind chat matches the operator-session registry's titles and
directories and the newest messages of recent transcripts through the chat
broker (bounded count, concurrency and one-second budget), answered off the
connection's read loop. The all-kind search is unchanged for older clients.

The app sends the chat search after its run search and shows a Chats group
with title, harness, directory and snippet; a chat hit opens that chat in the
Chat tab. The probe fixture answers chat searches over ?chats=N.
2026-11-03 13:01:53 +00:00
alan 1de74f76bb Chat info: Session section with native id, resume command and export
The chat information sheet gains a Session section: the harness's native
session id (copy), harness, directory, a verified resume command (copy)
and Export conversation (Markdown), Export JSON and, for Claude Code,
Export native session (the JSONL transcript).

Server: chat_sessions rows carry resume_command and native_export; new
chat_export command builds Markdown/JSON from the same paged history
chat_messages serves (bounded at 8 MiB), refuses non-operator sessions.
App: exports go through the artifact file bridge (web download, share
sheet on phones).
2026-11-03 10:59:41 +00:00
alan 10049b8062 Chat info: account quota per harness (Claude rate_limit_event, Codex account/rateLimits, OpenCode unsupported)
Usage frames carry optional usage.account {harness, source, plan, as_of,
windows[{id,label,used_percent,resets_at}], unsupported}. Claude Code quota
comes from its rate_limit_event stream lines and the initialize account
(plan), shared across chats on the same account. Codex reads
account/rateLimits/read once per connection (re-read when a chat opens after
5 min) and merges account/rateLimits/updated. OpenCode exposes no quota API
and says so. The app's chat info sheet shows an Account usage section with a
progress bar and reset countdown per window, Material and Cupertino.
2026-11-02 18:57:13 +00:00
alan 9393345acb Merge: worktree-agent-adedec543a1830ea7 2026-11-02 16:47:33 +00:00
alan 07d4740d0c Show injected Claude input as agent reports and notices, not operator bubbles
Claude Code records subagent/peer reports, task notifications, reminders and
skill bodies as user-role entries. Classify them server-side (history and live
stream) by origin/isMeta/queued_command mode: reports become kind agent_report,
task notifications kind notice, harness meta is hidden, and the mid-turn
wrapper is unwrapped to the operator's text. The app draws a collapsed Agent
report card and a one-line system notice on the agent's side.
2026-11-02 16:44:07 +00:00
alan b4382b08ad Apply the chat directory's environment and account to Claude Code chats
Claude Code chats now start with the chat directory's own environment:
direnv (allowed .envrc only, via direnv export json) and mise (mise env
--json), cached per directory and invalidated when config, watched files or
allow records change. A blocked .envrc is logged and shown in the chat as a
notice and never applied. When the directory chooses the Claude account
(CLAUDE_CONFIG_DIR, API key or token variables), the server's service token
and login-file fallbacks are skipped and inherited credentials dropped.
Transcripts in a directory's CLAUDE_CONFIG_DIR are found for history,
resume and after a restart. PWD is set to the chat directory.
2026-11-02 16:37:20 +00:00
alan d78cbe3f66 Merge: one way to set up harnesses (pipelines harness setup) 2026-11-02 16:14:50 +00:00
alan 108c29f9eb One harness setup path: pipelines harness setup <claude|codex|opencode|pi>
Checks the CLI on the server's PATH (install command when missing), signs in
with the harness's own login only when needed (--login forces it), keeps the
OpenCode/Pi model pickers, and verifies with the server's check. Works on
native and Docker installs (Docker delegates to the container setup); --host
is a hidden compatibility alias. harness list shows all four harnesses'
installed/signed-in/configured/default status. ./install --opencode|--pi now
point to the command; ./install ends with the next-step hint.
2026-11-02 16:11:04 +00:00
alan 80022ea41a Keep observing chats with an in-flight turn after the last client leaves
Chat attention notifications (questions, approvals, finished replies) come
from a session's event stream, which the hub only read while a room existed;
detach destroyed the room when the last client left, so a turn that finished
or asked with the app closed notified nobody.

A room whose last client leaves now stays observing, unattended, while its
turn is in flight or it holds an unanswered question/approval, and is torn
down when the turn ends (or after 6h for a stuck turn); at most 16 such rooms.
At server start, recently used open sessions whose harness reports a running
turn (OpenCode /api/session/active, Codex thread/turns/list) are observed
again so their reply still notifies. Settings, dedupe and app_notifications
recording are unchanged.
2026-11-02 16:07:42 +00:00
alan 3e100f78f3 Document the current install, harnesses and notifications
Rewrite the README around the server, web/Android app and CLI, and
replace docs/install.md with the current native install (all-or-nothing
Docker web build, flags from setup-native.py, lifecycle commands, data
locations, password, upgrading). Add docs/harnesses.md and
docs/notifications.md; point ntfy.md at the new overview and fix its
topic name.
2026-11-02 15:54:25 +00:00
alan eed3806efb Add a home-restricted folder picker to the new-chat dialog
Server: list_directory returns a directory_listing (path, home, parent,
entries with is_dir/is_git_repo/hidden, truncated) confined to the server
user's home subtree after cleaning and resolving symlinks, refusing pseudo
filesystems and capping entries at 2000. Documented in docs/cli.md.

App: "Choose folder…" in the create dialog opens a picker (full-screen page
on phones, large dialog on wide windows, Material and Cupertino) with
breadcrumbs, filter, Show hidden, git badges, dimmed files, recent folders,
inline errors and "Start chat here". The probe app gains a folder fixture
for captures.
2026-11-01 21:51:48 +00:00
alan e42f9cc031 Send mid-turn chat prompts to the harness and queue the rest durably on the server
Claude Code (stream-json stdin) and Codex (turn/steer pinned to the active
turn) receive a prompt sent while a turn runs immediately. Other harnesses
(OpenCode) get a durable chat_prompt_queue row owned by the session: it
survives the last client detaching and a server restart, is delivered in
order when the turn ends with or without a client, and is deduplicated by
operation id. chat_subscribe carries the queue, chat_queue frames report
delivered/removed/failed changes, and chat_unqueue removes a waiting prompt.
2026-11-01 18:30:20 +00:00
alan d35739cb49 Release session export gates whose recovery is exhausted or overdue
An exhausted export recovery job left its run parked forever in the
export wait, reserving the workspace and refusing later scheduled runs
(runs 221/226: OpenCode sessions above the 8 MiB export bound are
always truncated, so every retry stays partial).

A gate is now releasable when its job is exhausted: the visit keeps its
partial/unavailable evidence and diagnostic (still repairable), the run
continues to its next stage or terminal, and the workspace is freed.
Gates older than 15 minutes with a pending, lease-expired or missing job
are exhausted by a bounded wait. The worker reconciles on its first
tick after startup and every 30 seconds, so existing wedged runs recover
after a restart. Truncation diagnostics now name the size bound.
2026-11-01 17:24:56 +00:00
alan d1846441cc Stream ntfy notifications to app clients over /connect
Record every notification published to ntfy (run outbox, stage notify
hooks, chat question/approval/reply) in a persisted app_notifications
outbox and stream it to subscribe_notifications clients with an after
cursor; clear_notifications and ack_attention clear across devices, as do
runs leaving waiting, deleted runs and answered chat questions. Add the
notify.ntfy_enabled setting (default on) to stop only the ntfy publish.
2026-11-01 02:03:58 +00:00
alan 31e89e872b Add run size_bytes and delete_runs to free disk space
Every run summary carries size_bytes (artifact tree, server-created Pi
session directories and an estimate of the run's database rows), cached in
pipeline_runs.size_bytes, invalidated by triggers, measured when a run
finishes and backfilled in the background. delete_runs removes terminal
runs with every scoped row and server-owned file, refuses active ones with
a reason, and pushes runs_removed to subscribers. pipelines runs rm is the
CLI form; retention apply now shares the complete row deletion.
2026-10-31 22:11:14 +00:00
alan b109ab3e06 Notify on run end only for runs marked Notify me
Auto runs no longer alert on failure or success; only notify_mode on does,
unless the run_finished setting (now 'Notify when any run ends', default off)
opts into every run end. Waiting still notifies auto and on runs. The setting
is stored under a new key so a value saved under the old meaning is ignored.
2026-10-31 16:52:04 +00:00
alan be932c846f Catch up a moved main in the delivery gate before landing
When canonical main is not an ancestor of the item HEAD, the gate now
merges main into the item workspace under the delivery lock, re-runs the
declared checks on the merged commit, and fast-forwards main to it. A
conflicting merge is aborted with the workspace verified unchanged and a
diagnostic naming the files. Successful catch-ups are logged.
2026-10-31 15:24:12 +00:00
alan d1aa36163a Sync Needs-you dismissals and chat read state across devices 2026-10-31 12:47:25 +00:00
alan 13c4035175 fix(turns): keep the run task in every stage's input
Later visits' input.md held only the previous stage's output, so the
operator's task reached only the first stage; when a Prepare stage
summarised (paths, HEAD), Code never saw the task or its acceptance
checklist and either blocked (run #202) or did part of it (#195, #197,
#198). Later inputs now carry the run prompt under '# Run task' (headings
demoted) before '# Previous turn output'.
2026-10-31 02:52:16 +00:00
alan 498c6f17a6 Add server notification settings with app checkboxes
Persist three global ntfy switches in server_settings: run_finished
(default on: auto-mode runs notify on success, read inside the
transition transaction; on/off modes unchanged), chat_needs_you
(default on: gates chat question/approval alerts) and
chat_reply_finished (default off: one alert per finished assistant
turn). Expose notification_settings / set_notification_settings over
the websocket protocol, add a Phone notifications section with
Material and Cupertino checkboxes in Settings, document the settings,
and keep the cli, tui, stage and pipeline test packages from
inheriting a real ntfy configuration.
2026-10-30 21:36:27 +00:00
alan 40ba2ac117 Open ntfy notifications in the app and notify chats that need the operator
Run notifications now always click through to pipelines://run/<id>; a valid
PIPELINES_WEB_PUBLIC_URL adds an "Open in browser" ntfy view action instead
of replacing the click. Queued rows derive links from the run id at delivery,
so legacy rows holding a web URL still open the app.

Unresolved chat questions/forms and permission requests publish one async,
bounded notification to the system destination (pipelines://chat/<id> plus the
browser action), deduplicated per form/request id across reconnects and
replays. stage.PublishNtfy gains optional, strictly validated view actions.
Server tests now clear the operator's real ntfy environment.
2026-10-30 20:40:06 +00:00
alan 41a3be8715 Keep web Material styling and preserve visible installer access URLs 2026-10-26 18:37:13 +00:00
alan 9bad1bd824 feat(install): pipelines update re-runs the recorded install from the source checkout; --fresh skips Docker migration 2026-10-23 23:37:35 +01:00
alan 8527506da1 chore: remove the JavaScript web client and the Kotlin Android app; the Flutter app replaces both 2026-10-23 21:46:58 +01:00
alan 83d013f10e docs: record Claude Code chat backend deployment 2026-10-22 18:23:48 +01:00
alan 0acd7346bb feat(chat): add Claude Code chat backend 2026-10-22 18:20:10 +01:00
alan ceffd5d0d1 fix(stage): retain inline audit payloads once without losing native evidence 2026-10-21 16:59:20 +01:00