Commit Graph

31 Commits

Author SHA1 Message Date
alan 277384c864 Add persistent machine terminals with tmux-backed reattach 2026-11-09 21:30:15 +00:00
alan 96b71d122e Merge branch 'worktree-agent-aae8756320b8309f6'
# Conflicts:
#	app/lib/features/chat/conversation/chat_thread_view.dart
#	app/tool/perf/perf_fixture.dart
#	internal/protocol/message.go
#	internal/server/chat_hub.go
2026-11-09 16:56:55 +00:00
alan 9791f09230 Serve the Pipelines chat tools over a loopback MCP endpoint and register it with Claude Code, Codex and OpenCode chats 2026-11-09 16:34:21 +00:00
alan 7841aa7dea Merge branch 'worktree-agent-a672970b929d7358c'
# Conflicts:
#	internal/pipeline/store.go
#	internal/server/chat_hub.go
2026-11-09 16:21:55 +00:00
alan 0696ab849a Record each chat turn's workspace changes for every harness
Snapshot the chat directory with internal/filediff when a turn starts and
compare it when the turn ends; store changed turns (newest 100 per chat),
push chat_turn_diffs summaries, attach them to chat_messages replies and
serve one turn's patches through chat_turn_diff.
2026-11-09 16:08:35 +00:00
alan e6bc57c901 Run operator shell commands in a chat's directory on the server
chat_shell_run starts $SHELL -lc in the chat session's directory in its own
process group, streams merged output to every watcher of the session, keeps a
1 MiB tail and persists each run; chat_shell_stop interrupts the group and
kills it after a grace period. chat_shell_share queues an ordinary prompt
with the command and a bounded output tail.
2026-11-09 15:55:14 +00:00
alan f5bf8ada8a Merge: archive and unarchive pipelines
# Conflicts:
#	internal/cli/commands.go
#	internal/pipeline/store.go
2026-11-04 23:39:02 +00:00
alan 06bf580154 Archive pipelines: hide unused ones while keeping their history
archived_at on pipelines (migrated in place). archive_pipeline and
unarchive_pipeline by id or name with an expected-version guard;
list_pipelines hides archived unless include_archived. Archived
pipelines refuse runs, retries, new versions and name reuse, and
cannot be archived while an enabled schedule points at them.
CLI archive/unarchive and list --archived/--all, docs and skill note.
2026-11-04 23:25:55 +00:00
alan aa8eddd686 Keep OpenCode/Codex chat and stage sessions running across server restarts
A stopping server no longer interrupts harness sessions. Runs are canceled
with a shutdown cause, so stage clients leave admitted sessions running and
owned; start-up recovery re-adopts live OpenCode V2 stage sessions (wait,
read back the result, continue the graph), restarts unadmitted visits,
continues runs between stages, and falls back to interrupted with a reason
when the harness is gone. Chats with live work are recorded on shutdown and
reattached on start; turns that ended during the downtime are folded in
from history. The server-spawned V2 chat serve is now a detached owned
service. PIPELINES_SHUTDOWN_SESSIONS=stop restores the old behaviour.
2026-11-04 16:17:08 +00:00
alan 3c31bca87f Merge main; align lint and skill practices with essential memory/history and the four graded examples 2026-11-04 13:08:47 +00:00
alan 37e941bbba Authoring lint: required checks, undeclared outcomes, delivery claims, loop limits, silent guards, boilerplate 2026-11-04 13:04:11 +00:00
alan a48e4a5b6a Replace starters with four graded examples seeded into new databases
The template catalog is now Example 1-4: one stage, two stages (handoff),
feedback loop (runner-owned limits, retries, timeout) and human decision
(type=human approve/revise/abort with notify hooks). All read-only or leave
changes uncommitted; none commits, pushes or deploys.

A brand-new database (schema created by this Open) gets the examples once at
server start, bound to the configured default model and harness. A
server_settings marker records the decision, so removed examples never come
back and existing installs are marked without any pipeline being added or
rewritten.
2026-11-03 19:35:47 +00:00
alan e42f9cc031 Send mid-turn chat prompts to the harness and queue the rest durably on the server
Claude Code (stream-json stdin) and Codex (turn/steer pinned to the active
turn) receive a prompt sent while a turn runs immediately. Other harnesses
(OpenCode) get a durable chat_prompt_queue row owned by the session: it
survives the last client detaching and a server restart, is delivered in
order when the turn ends with or without a client, and is deduplicated by
operation id. chat_subscribe carries the queue, chat_queue frames report
delivered/removed/failed changes, and chat_unqueue removes a waiting prompt.
2026-11-01 18:30:20 +00:00
alan d1846441cc Stream ntfy notifications to app clients over /connect
Record every notification published to ntfy (run outbox, stage notify
hooks, chat question/approval/reply) in a persisted app_notifications
outbox and stream it to subscribe_notifications clients with an after
cursor; clear_notifications and ack_attention clear across devices, as do
runs leaving waiting, deleted runs and answered chat questions. Add the
notify.ntfy_enabled setting (default on) to stop only the ntfy publish.
2026-11-01 02:03:58 +00:00
alan 31e89e872b Add run size_bytes and delete_runs to free disk space
Every run summary carries size_bytes (artifact tree, server-created Pi
session directories and an estimate of the run's database rows), cached in
pipeline_runs.size_bytes, invalidated by triggers, measured when a run
finishes and backfilled in the background. delete_runs removes terminal
runs with every scoped row and server-owned file, refuses active ones with
a reason, and pushes runs_removed to subscribers. pipelines runs rm is the
CLI form; retention apply now shares the complete row deletion.
2026-10-31 22:11:14 +00:00
alan f4b3ef17cc Persist and broadcast shared attention acknowledgements 2026-10-31 12:23:51 +00:00
alan 4d8b34939d Separate saved run schedules from pipeline definitions 2026-10-21 00:48:34 +01:00
alan 9cdfd9d9ab search: browse bounded recent items with kind-only queries 2026-10-20 17:54:55 +01:00
alan 4e35da7473 fix: stop store-bound workers and reconcile settled repair fixtures 2026-10-20 16:19:21 +01:00
alan b96b9d94a6 fix: keep cancellation pending until execution cleanup is confirmed 2026-10-20 15:49:08 +01:00
alan 016fcb552a fix: persist and retry run alerts and honor notification modes 2026-10-20 15:44:23 +01:00
alan 526c19fc21 web: shared chat stars, pipeline mini graphs and agent questions
- Persist chat stars in the operator-session registry (chat_star) so every
  browser/device shows the same Starred group; migrate old local stars once.
- Add a compact pipeline graph shape to list responses and draw a lightweight
  list miniature; replace the cryptic id/version line with the stage count.
- Surface OpenCode form questions as a chat card (chat_answer), replay a
  pending form to late subscribers, and notify for background questions.
- Make the mobile dock's Search cell equal to every destination, keep the
  tab row visible with 100svh, and let Source use the full pane height.
- Let a fresh sign-in auto-connect to its own origin; use a Lax session cookie
  with Expires so Remember me survives app/link launches.
- Keep shipped starters and the skill agnostic: no repository paths, project
  areas or personal device commands; add Diagnose/Fix and Research starters.
- Refresh the installed skill and document the new browser conventions.
2026-10-20 03:11:37 +01:00
alan 72aecd8b1f Remove resource pools, leases and wait queues
Drop the resource allocation feature end to end: store tables and migration, runner scheduling and cleanup, protocol types and commands, CLI commands, TUI screens, tests and docs. Existing databases open cleanly and definitions carrying a retired resources block still validate.
2026-10-19 22:27:27 +01:00
alan 9130ae8c7a Add stage-exclusive resource waiting
Add durable FIFO capacity-one resource leases for executable stages, including cleanup-confirmed release, conservative restart quarantine, queue state events, and CLI/TUI/Android waiting presentation. Add scheduler, cleanup, protocol, client, and Android coverage plus a proposed Android v3 resource draft.

Checked packaged templates and authoring skill references: neither needs updating in this item. The per-item docs note records the shared documentation that docs-fold must update; no shared README or packaged reference was edited.
2026-10-18 12:30:30 +01:00
alan 1f75e4797e Broker chat from an operator's own OpenCode endpoint and adopt existing sessions
Add an opt-in way to broker chat from an operator-supplied OpenCode service
instead of the instance the server spawns for itself, and a way to adopt a
session that already exists on that endpoint so it appears in chat_sessions and
can be continued from the app.

- Endpoint selection: the `-opencode-endpoint` server flag (defaulting to
  `PIPELINES_OPENCODE_ENDPOINT`) opts in; precedence is explicit flag >
  explicit environment > the endpoint recorded on a previous run, with the
  spawned instance as the default. opencode.OpenEndpoint validates the
  reference once (http(s), host, no userinfo/query/fragment/path), detects the
  backend from its own /api/info (v2) or /global/health (v1) contract, and
  requires a streamable text/event-stream event path. It owns no process
  (Close is a no-op) and never calls startServe; every failure names the
  endpoint and disables chat rather than silently falling back. An
  authenticated endpoint takes an optional password only through the private
  file PIPELINES_OPENCODE_ENDPOINT_PASSWORD_FILE, with
  PIPELINES_OPENCODE_ENDPOINT_USERNAME overriding the detected default.
- Persistence: the validated choice is recorded in an additive single-row
  server_settings table so a restart with no flag/environment reuses it; an
  unreachable recorded endpoint disables chat with a clear message and no
  fallback.
- chat_adopt: a new protocol request (ChatAdoptRequest) with handleChatAdopt in
  internal/server/chat.go. It refuses a nil broker/registry/payload and a
  blank or foreign-shaped id before any endpoint call, reads the endpoint via
  the shared Chat.Session (which reuses Sessions/decodeChatSessions), records
  through the existing RecordOperatorSession with the endpoint's directory and
  title verbatim, and echoes the adopted session plus its registry row like
  chat_create. A missing id is refused naming the id and records nothing; an
  endpoint record with an empty directory is refused rather than substituting
  the server's working directory or home.
- CLI: `pipelines chat-adopt <session-id> [--json]` (usage, notes, help
  inventory, dispatch) with docs/cli.md updated under Chat sessions.
- Autostart preserves the endpoint variables (and the password-file reference,
  never an inline secret) via serviceEnvironmentKeys.

Checks before committing:
- Package location: the change is user-facing chat behaviour, so the packaged
  authoring skill reference internal/author/skill/references/operations.md is
  updated in the same change (Brokered OpenCode chat section) with a dated
  verification-evidence entry and the not-deployed state. No new reference file
  is added, so internal/author/install.go's managed inventory is unchanged.
- Packaged templates: this repository packages no template set for this
  behaviour and none references the chat endpoint, so no template needs
  updating.
- Verification: `go build ./...`, `go vet` on the touched packages, `gofmt -l`
  clean and `go test ./...` all pass. The unit tests cover the endpoint
  validation/precedence, the adopt happy path and refusals, the recorded
  endpoint round-trip, the protocol round-trip and the CLI dispatch. The live
  scratch-server check against the operator's real endpoint was not run in this
  change.

Nothing unrelated is staged; docs left stale (README and other shared documents)
are not edited here and are for a single folding writer.
2026-10-17 20:14:11 +01:00
alan 14175eba70 retention: explicit run-history and artifact retention
Add `pipelines retention report` (strictly read-only) and
`pipelines retention apply` (explicit, approval-gated removal) with an age
and/or count policy, protected-run rules, atomic single-transaction removal
and honest database space reclaim. Nothing prunes automatically.
2026-10-17 15:45:46 +01:00
alan 18398dfa5b chat: give the operator their own sessions, separate from run sessions
chat_sessions used to list exactly the OpenCode sessions the pipelines own,
filtered through pipeline.SessionOwner (run_steps/run_session_owners), so an
operator-created chat was an accident of the run-ownership table and the
operator's working sessions were mixed with the internals of running pipelines.

Add a first-class operator session:

- internal/pipeline/operator_sessions.go: a dedicated operator_sessions table
  (session_id PK, directory, title, tag, created_at, last_used_at) with
  RecordOperatorSession (upsert preserving created_at), a one-statement
  TouchOperatorSession (no-op for a non-operator id), OperatorSessions bounded to
  128 rows most-recently-used first, and OperatorDirectories (bounded MRU,
  deduped). run_steps/run_session_owners and SessionOwner/OwnedSessions are
  untouched.
- internal/server/chat.go: chat_sessions returns operator sessions only - it
  reads the registry once, intersects it with one broker session-list call, fills
  directory/tag/created/updated from the registry, sets busy for a session held
  by another connection, keeps MRU order, and stays fail-closed (empty list, never
  the endpoint's whole list). A pipeline stage session and an untagged row can
  never be listed. chat_create resolves the directory (empty/'~'/'~/...' ->
  operator home; relative or missing rejected before any OpenCode call), calls
  broker.Create(title, resolved) and records the row with the operator tag.
  chat_directories returns home first then bounded (8) deduped recents. A
  successful prompt touches the operator session without affecting pipeline
  prompts.
- internal/opencode, internal/protocol, internal/server/websocket.go and
  message.go: unscoped session list, Create posts {"title":...,
  "location":{"directory":...}} to POST /api/session, additive protocol fields
  (omitempty) and the ChatDirectories request/response, plus the OperatorRegistry
  option defaulting to the first store. One OpenCode transport per server is
  unchanged.

Docs and references: docs/cli.md gains a "Chat sessions" section documenting the
operator-only fail-closed list, directory defaulting/rejection, chat_directories
(home first, deduped, bounded 8) and why a dedicated request was chosen.
internal/author/skill/references/operations.md's stale "Brokered OpenCode chat"
text (chat_create unbrokered, run_steps filtering, "listed the owned scratch
session" evidence) is rewritten to the operator-session registry, the fail-closed
filter, directory defaulting and the corrected live-test wording. No new
reference file, so internal/author/install.go's inventory is unchanged.

Checked for staleness elsewhere: README.md, the docs/ topic pages and the
packaged server/TUI templates carry no chat-session ownership claims, so none
needed updating.

Verified: gofmt -l clean; go vet and go build ./... clean; go test -count=1
./internal/pipeline/... ./internal/protocol/... ./internal/opencode/...
./internal/server/... and the full go test -count=1 ./... pass.
2026-10-17 11:14:25 +01:00
alan c700b28e6a Add cross-system read-only search
One query now finds anything in the system over the wire and from the CLI.

- protocol: add Search/SearchResults message types and a SearchRequest
  (query, optional kind filter, optional limit) / SearchResponse (typed,
  ranked matches plus truncation and scan-bound counters) pair, with typed
  payloads for pipeline, version, run, step/visit, field text and artifact.
- store: add Store.Search, a read-only query over pipelines,
  pipeline_versions, pipeline_runs, run_steps (+step_control) and
  step_artifacts. Ranks exact name (3) > name substring (2) > body text (1),
  breaks ties by recency then a stable kind/target order, and reports the
  matched field. Bounds work with a 50-default/200-max result cap, a 2000-row
  per-kind scan limit and a 2000-step / 1 MiB field-text scan budget.
- terms: a bare status word or lone date expression (today, yesterday, this
  week, YYYY-MM-DD) selects runs; anything else stays free text.
- server: dispatch protocol.Search to stores[0].Search; read-only and
  unconfirmed.
- cli: add `pipelines search QUERY [--type KIND] [--limit N] [--json]`, help
  text and a BROWSE line.
- store: add OpenReadOnly so the live-database check never runs DDL.
- docs: document the grammar, kinds, ranking and bounds in docs/cli.md,
  scope the TUI omnisearch to pipelines in docs/ui-ux-review.md, and add a
  search bullet to the packaged operations reference.
- tests: protocol round trip, store/ranking/term-parsing/limit/scan-bound
  unit tests, a server end-to-end test, and an env-guarded live check.

No new dependencies; go.mod/go.sum unchanged.
2026-10-16 15:43:25 +01:00
alan 0f5a188c6a Record and serve per-stage artifacts
Every agent stage now writes into a per-run/step artifacts directory under
the data dir (<data>/artifacts/<run>/<step>), passed to Pi as
PIPELINES_ARTIFACTS_DIR, exposed on the OpenCode client env seam, and stated
in the agent prompt boilerplate so agents save images, screenshots, reports
and logs there.

After each step the server scans that directory with a bounded, symlink- and
traversal-safe inventory (os.OpenRoot + Lstat + O_NOFOLLOW) recording name,
size_bytes, content type (extension then 512-byte sniff fallback) and URL,
enforcing max files, per-file bytes and total bytes while counting skipped
and truncated entries. A missing, unreadable or oversized directory yields
only a best-effort diagnostic and never changes the step outcome.

The list is carried on protocol.RunStep as Artifacts []StageArtifact plus
skipped/truncated counters (omitempty, backwards compatible), persisted in a
new step_artifacts table keyed (run_id, sequence) alongside the
notifications/file-changes tables and loaded by GetRun.

GET /artifacts/{run}/{step}/{name} on the existing handler serves the bytes
with http.ServeContent, so Content-Type, Content-Length, HEAD and Range
(206/416) come from the stdlib; unknown runs/steps, absolute paths,
traversal, symlink escapes and non-regular files are refused.

The TUI lists each selected step's artifacts (name + human size) in the
retained-evidence section and adds an artifacts summary in the style of the
file-changes summary. Adds docs/stage-artifacts.md linked from README and an
artifacts subsection to the packaged operations skill reference.

Tests cover the scan records and caps, missing/oversized directories,
HTTP HEAD/Range/traversal/symlink/unknown refusals and protocol round-trips.
2026-10-15 17:51:18 +01:00
alan accbe63e39 Simplify Go installation and support standard OpenCode releases 2026-10-08 23:17:38 +01:00
alan 0f30a001c2 Initial pipelines server, CLI, TUI, and Go installation 2026-10-08 22:22:17 +01:00