30 Commits

Author SHA1 Message Date
alan bd07abcde8 Run pipeline stages on Claude Code (harness=claude)
A model stage with harness=claude, provider=anthropic and a Claude model id
runs one claude -p process per visit with the server user's Claude sign-in.
Only the declared tools exist (--tools) and nothing else is approved
(--permission-mode dontAsk); edits are approved only below the run directory
and the stage's artifacts directory. A permissions block, run write paths,
delivery=true and reasoning_effort none/minimal are refused, not widened.

Usage comes from the CLI's final result and counts toward budgets. Failures
are fixed text; billing, quota and rate-limit refusals map to
ErrProviderQuota. The redacted stream is the session export and evidence.
provider-check --harness claude and model overrides are covered.
2026-11-11 10:30:37 +00:00
alan 532ba48927 Run every pipeline run in its own worktree
A fresh run started from a repository's main checkout gets its own
worktree and branch next to the repository, so runs never share a tree
or see the operator's uncommitted work, and several can run at once.
Retries and runs started in a linked worktree keep their directory. A
clean, landed run tree is removed after success; anything else is kept.

The delivery gate commits what the land stage left uncommitted in a run
worktree, and no longer requires the operator's main checkout to be
clean: Git's fast-forward refuses safely when a local change conflicts.
2026-11-10 16:06:40 +00:00
alan 5090f1cf4e Pipelines skill: outcome-first output, aligned review and loop guidance, operator material moved to docs
Rewrite format.md around turn files, the output.md order (# outcome, edge word, # handoff, detail), checks, configs, human stages and the land stage delivery gate (sh -c in a fresh detached worktree, clean checkouts incl. untracked, 64 KB output cap, opencode only). Align review guidance across practices, verification, orchestration and the feedback-loop example. Fix loop-limit semantics in control.md against runner.go, budget_tokens sizing, read for large inputs, blocked-else-error and one stage naming set. Move dated verification logs to docs/verification-history.md and brokered chat, TUI keys and harness internals to docs/operator-reference.md; label operator-only references in SKILL.md.
2026-11-04 17:27:01 +00:00
alan 3c31bca87f Merge main; align lint and skill practices with essential memory/history and the four graded examples 2026-11-04 13:08:47 +00:00
alan d7e54f4612 Merge: short essential memory per stage; full history on demand 2026-11-04 13:01:16 +00:00
alan 538143554c Split stage memory into capped essential memory.md and full history.md
Every stage used to be told to read memory.md, the full output of every
earlier turn (19 KB average, 92 KB max), and paid for it on each model call.
memory.md is now essential memory, hard-capped at 4 KB and inlined in the
prompt: the task, the previous stage's # handoff section (or its truncated
output head), a one-line-per-stage outcome ledger and the server-captured
changed files. history.md keeps the old full memory on disk; read-capable
stages are pointed at it and input.md for on-demand detail, and it is never
inlined. The prompt asks each stage for a short # handoff section.
2026-11-04 13:00:48 +00:00
alan e5c015f6ed Merge: no UI-review auto-skip without complete change capture; capture lists over-budget changes
# Conflicts:
#	internal/protocol/runs.go
2026-11-04 12:58:45 +00:00
alan f16d61cdfa Skip fallback needs complete, empty evidence; capture lists over-budget changes
The server skip fallback for stages with a skipped edge now applies only
when every earlier visit recorded a complete file-change capture with no
changed files and no image artifact. Any change (no repo-specific UI path
or extension list), any partial/unavailable/missing capture, a human
decision or a checkpoint-retry prefix runs the model. auto_skip=false opts
a stage out. The fallback verdict names the server fallback.

Capture compares over-budget text by hash and, in Git, lists paths it
could not compare when their git status changes, marked no_patch, so a
large workspace no longer records files: [] for a real edit. Run detail
gains a warnings list for partial/unavailable captures, also printed by
inspect on stderr.
2026-11-04 12:56:40 +00:00
alan a48e4a5b6a Replace starters with four graded examples seeded into new databases
The template catalog is now Example 1-4: one stage, two stages (handoff),
feedback loop (runner-owned limits, retries, timeout) and human decision
(type=human approve/revise/abort with notify hooks). All read-only or leave
changes uncommitted; none commits, pushes or deploys.

A brand-new database (schema created by this Open) gets the examples once at
server start, bound to the configured default model and harness. A
server_settings marker records the decision, so removed examples never come
back and existing installs are marked without any pipeline being added or
rewritten.
2026-11-03 19:35:47 +00:00
alan be932c846f Catch up a moved main in the delivery gate before landing
When canonical main is not an ancestor of the item HEAD, the gate now
merges main into the item workspace under the delivery lock, re-runs the
declared checks on the merged commit, and fast-forwards main to it. A
conflicting merge is aborted with the workspace verified unchanged and a
diagnostic naming the files. Successful catch-ups are logged.
2026-10-31 15:24:12 +00:00
alan 1e4654381e feat(web): manage independent schedules and restore recorded stage navigation 2026-10-21 02:06:15 +01:00
alan 724de27561 feat(stage): honor per-stage reasoning effort across native harnesses 2026-10-20 23:51:14 +01:00
alan e37b864c40 stage turns: replace JSON results with file-based markdown turns
Each model visit now gets <run>/stages/<seq>-<stage>/ with input.md,
memory.md, output.md, private/ and a shared run-level public/. The model
reads the two input files, follows the stage's instructions verbatim and
returns complete markdown; only the # outcome section is parsed.

- add internal/turn layout, render and output parsing
- write input.md/memory.md before a visit and output.md from the final
  message; inventory all turn files as stage artifacts
- accumulate completed turns and system-written public file additions in
  memory, including across retries and resume
- remove the JSON envelope, strict field validation, line format and the
  repair ladder for model stages
- republish pipeline authoring prompts and docs in the new style

Checks: go build ./..., go vet ./..., go test -count=1 ./..., gofmt -l .
2026-10-19 22:27:44 +01:00
alan bfcfcde415 Rewrite agent rules for accountable direct recovery and small parallel work 2026-10-18 23:05:40 +01:00
alan 8f3ebf74ad skill: replace the lane model with the work queue
The lane machinery is gone: work is a markdown queue, items run in parallel in
their own worktrees, the committer stage lands its own work, land-check.sh gates
cleanup, branches are never deleted, and pipelines wait (rather than a driver
script) is what wakes a session. references/lanes.md becomes references/queue.md;
the remaining lane references in work/orchestration/debugging/delivery/format and
SKILL.md are rewritten, and the installer now prunes a managed file it no longer
bundles when the copy is unmodified.
2026-10-18 01:12:21 +01:00
alan a3f571d672 docs: make docs-fold the single writer of shared documentation
Every item wrote the shared, append-only docs (android/README.md, docs/cli.md and
the packaged skill references), so every merge conflicted on prose. Keep the duty
to notice stale docs; move the writing to one deliberate place.

- Per-item note, not a shared edit. An item that makes a shared document stale
  writes docs/notes/<slug>.md - unique per item, so two items never touch the same
  file - and commits that note with its change. It never edits a README, topic doc
  or packaged skill reference itself.
- One writer. The manual-only docs-fold pipeline, bundled as the fifth starter
  examples/docs-fold.md, folds every note under docs/notes/ into the shared
  document it names and deletes the consumed notes in its own commit; an empty
  notes directory is a success no-op. Operator trigger:
  pipelines new "Docs fold" --template docs-fold --no-open; pipelines submit
  <draft.md>; pipelines run "Docs fold".
- Implemented where the convention lives: orchestration.md (brief item 5, the
  hard-constraint bullet, a "Single documentation writer" section and the
  "Pipeline choice" committer bullet), the canonical committer checklist in
  format.md (new step 6), the committer prompts in examples/task.md and
  examples/three-stage.md, the new examples/docs-fold.md with its catalog and
  README entries, and the repository guideline AGENTS.md. Test-expectation counts
  (examples/catalog_test.go, internal/server/templates_test.go,
  scripts/smoke-first-install.py) follow the fifth starter.

Reflected where it belongs: the packaged templates (examples/task.md,
examples/three-stage.md, the new examples/docs-fold.md), the packaged skill
references (orchestration.md, format.md) and the repo guideline AGENTS.md all carry
the new convention. No new reference file was added, so install.go's managed
inventory is unchanged. The shared descriptive docs were deliberately not
rewritten: this change makes README.md:191, docs/cli.md:170 and
internal/author/skill/references/operations.md:13 name four starters where there are
now five. Those stay as they are - correcting them now is exactly the per-item
shared edit this change removes - and the stale-doc note
docs/notes/docs-fold-single-writer.md records each one for the docs-fold writer to
fold. Nothing else needed updating. No new dependency; no Android, Gradle, server,
installer, Docker or published-definition change; no definition published over the
running pipelines.

Evidence: go test -count=1 ./... passed; go build ./... and go vet on the touched
packages clean; ./internal/author/... covers embedding, reference links, the managed
inventory and install/upgrade/protection; ./examples/... validates the fifth starter
and its manual-only, committer-terminated graph; ./internal/server/... covers the
served catalog. Not run: the docs-fold writer against the shared docs (nothing to
fold yet); no install or deploy.

(cherry picked from commit f29656cf084d7eee2e033f2a37a9d231d7acaaf0)
2026-10-17 23:59:59 +01:00
alan c9b904186a Update templates and authoring skill to the current pipeline contract
Bring the packaged starters and the pipelines skill up to the guidelines the
project now works to.

Templates:
- task and three-stage now end in a committer stage that stages the union of the
  runner's captured input.server_changed_files and the producer's declared
  changed_files, runs the repository's checks, commits locally and never pushes;
  their producers no longer commit.
- Every model/agent prompt states the line-oriented result contract (outcome on the
  first non-empty line, "# output" JSON map, other "# <heading>" sections kept
  verbatim as evidence) and an "# evidence" section for exact commands and results,
  with proportionate verification and no fabricated checks.
- Stages that can write save their reports, logs and screenshots into the runtime
  artifacts directory.
- Remove the summary starter: a tool-free single-prompt text transform with no
  repository interaction, artifact or commit is a format demonstration, not a
  reusable workflow. No starter invents a pretend approval stage; a decision point
  is a type=human stage.

Skill:
- references/format.md gains a Result format section (line form, legacy JSON
  precedence, result_parse, raw-output.md, artifacts) and a Committer stages
  section with the pre-commit checklist; the role-split and envelope notes mention
  the line form and the committer.
- orchestration.md points at the checklist and records that a genuine gate is a
  human stage; operations.md and the SKILL.md index drop the removed starter and
  name the result format and committer. No new reference, so the managed inventory
  is unchanged.

Docs/tests: examples/README, README, docs/cli, docs/first-version-testing, the CLI
help example and the smoke script track the four starters; catalog, tool-role,
server, tui and pipeline tests are updated, with a new test pinning the committer
ending and server-captured change set.

Commands run: go build ./...; go vet ./...; go test ./...; python3 -m unittest
discover -s scripts -p 'test_*.py'; python3 scripts/smoke-first-install.py. All
passed. Templates inspected through an isolated CLI server (pipelines templates;
pipelines template KEY --source). No server, service or device was restarted, and
nothing was pushed.
2026-10-17 01:09:42 +01:00
alan a6aff47117 Add a human decision stage that pauses a run for an operator's choice
A new registered stage type `human` (internal/stage/human.go) has no model,
agent, tools or configs: its required `prompt` block carries the operator
instructions and its non-error edges are the selectable options. It parses,
validates, renders, exports and imports like every other type (no parser
special-case, no config keys).

At runtime the runner parks the run instead of executing: the step and the run
persist `waiting` with self-contained {prompt, options} evidence, and the run
returns without starting later stages, capturing file diffs, scanning artifacts,
holding a goroutine/timer or applying a stage timeout. The wait is durable: a
restart leaves the waiting rows untouched (RecoverRuns excludes them) and a
rebuilt runtime over the same database still accepts the decision.

`pipelines decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]` and
the TUI `c` picker resolve the decision from the pinned graph in one store
transaction, record the choice and feedback as the step's evidence and resume
along edges[KEY]. An unknown option is refused with the valid list, a stage that
is not waiting is refused, and a repeated decision for an already-decided visit
does not double-apply. Cancel and retry keep working while waiting and after a
decision; a human stage is never killed by a stage timeout or max_visits, and a
`notify` policy can fire an `on=waiting` event.

Docs: the authoring skill references (format.md, operations.md), docs/cli.md and
README describe the stage, the command and the TUI picker. Tests cover stage
validation, the parsing round-trip, the store transitions, a websocket wait/decide
integration, cancel/retry, waiting notifications and a persistence-layer rebuild
(second Open + newRunner over one database, no process restart).
2026-10-16 16:47:28 +01:00
alan 9f88e4ff71 Recover malformed model-stage results with a bounded, work-preserving repair
A model stage whose final result was malformed failed the whole run even when
the work and evidence had survived, because the only recovery was an isolated
formatter call on a fixed 30-second deadline that could time out on its own
(run 55, ui_reviewer of pipeline 9 v6).

Add an ordered, deterministic local repair pass (internal/stage/result_repair.go)
that runs before any model call: (a) append missing closing delimiters via the
existing closeJSONContainers; (b) remove one unambiguous trailing comma before a
closing } or ]; (c) insert exactly the single comma/colon the JSON parser reported
at its byte offset. Every candidate must pass losslessSyntaxRepair(original,
candidate) and decodeModelResult before it is accepted, so no field value is
invented, removed, reordered or rewritten. Ambiguous payloads (an unescaped quote
or colon inside a string value, or a stray delimiter with no single completion)
return not-ok and fall through to the bounded model re-emit rather than guessing.
stageResultText still strips surrounding prose and a single code fence first;
model_prompt.go now also strips prose around a single balanced-but-malformed
object span so the deterministic pass sees just the object.

Replace the fixed 30-second formatter deadline with repairDeadline(payloadBytes,
remaining), which grows monotonically with payload size, has a floor, and is
clamped to the attempt's remaining budget minus a safety margin, so the stage
context remains the outer bound and a large result is not guaranteed to time out.
The one-request-per-attempt, 32 KiB bound and the deny-all, retained, new-session
property of RepairFormat are unchanged for OpenCodeClient and PiClient.

Record an explicit recovery status on every terminal path: repaired, partial or
unrecoverable, plus a byte offset. A candidate that is valid JSON but still fails
envelope validation is marked partial and never returned as a completed result;
the unrecoverable path retains the bounded original payload, the precise parse
error (including its byte offset) and surfaces the status in the FINAL RESULT
VALIDATION diagnostic.

safeToReplay is intentionally unchanged. The deterministic local repair and the
fresh deny-all, no-tools, retained formatter session are independent of the
stage's tool set and never re-prompt or replay the admitted agent session, so a
shell-enabled review stage gains malformed-result recovery without any replay of
admitted work. Risk: no change to mutation safety; the only added cost is one
bounded formatter call, already budgeted. Replayability is not extended for shell
stages.

Docs and packaged skill references (docs/model-results.md,
internal/author/skill/references/{format,debugging,verification}.md) describe the
three-tier recovery, the status words and the derived deadline; the safeToReplay
decision and its risk are recorded there alongside the commit.

Tests add mocked, offline coverage for the deterministic and partial paths, a
repairDeadline monotonicity/floor/clamp table, and a shell-stage regression
proving a recoverable malformed result does not fail the run or replay tool work.
2026-10-16 13:02:53 +01:00
alan f57457aec9 Make agent-stage retries safe and retain timeout failure evidence
Fix four stage failure-handling and observability defects (A2, A3, A6, A9).

- A2: add safeToReplay(def) as the single definition of retryable post-admission
  work - only an explicit tools block that selects neither edit nor shell. The
  agent/pi WorkStarted guards now break only when the stage is not safe to
  replay, so read-only stages genuinely retry a post-admission timeout while
  mutating and legacy-bundle stages still never replay admitted work.
- A6: re-emit a malformed final result on an explicit read-only stage up to the
  retry budget, preserving format_repairs evidence; mutating stages keep the
  immediate return and their session.
- A3: timed-out stages retain usable evidence. waitForSessionIdle records one
  bounded sanitized OperationEvidence on deadline; the agent deferred cleanup
  does a bounded best-effort latest-assistant readback and settles knowable V2
  usage; RetainSession adds a retained_session pointer to failed step
  transcripts so the session is visible beneath the BoundTranscript threshold.
- A9: a stage outcome routed to #failure now yields a non-empty run diagnostic
  ("stage <name> returned outcome <outcome>") while preferring an existing step
  diagnostic.

Adds focused stage and server tests for retry safety, malformed-result
re-emission, timeout evidence retention and run-level diagnostics; updates the
packaged skill references and docs/cli.md, docs/model-results.md.
2026-10-15 18:27:22 +01:00
alan 143f237268 Strengthen template planning, reuse and independent review 2026-10-12 23:10:37 +01:00
alan 05ae530442 Make Pipelines Docker-only with OpenCode and Pi
Install current verified harness releases inside Docker, preserve persistent accounts and settings, and provide separate interactive and pipeline model defaults with terminal onboarding and fuzzy pickers.

Add Docker-aware authoring transport, Pi stage execution, bounded workflow controls, reliable final-result readback and usage settlement, and finite short-lived CLI pipe draining. Keep dated installation evidence and packaged authoring guidance consistent.

Verified with a real Docker installation, both harness stages and interactive authors, the built-in plan/implement/review starter, independent host race/vet checks, installer tests, and packaged skill tests. Generated release receipts stay ignored in clean checkouts.
2026-10-11 23:00:25 +01:00
alan c42a19f7d3 Improve workflow UI, retain stage diffs, and add portable pipeline transfers
Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
2026-10-10 20:55:42 +01:00
alan a66d540125 Add stage execution limits and result-first workflow views 2026-10-10 03:03:46 +01:00
alan 7b10e3889e Replace niche templates with simple starters and direct template submission 2026-10-10 02:37:31 +01:00
alan 08b8647d5b Validate providers and retain complete run debugging context 2026-10-09 23:21:01 +01:00
alan 67ac79765f Harden model results with lossless isolated repair and scoped permissions 2026-10-09 20:02:41 +01:00
alan 6bd0e28ce6 Record run input directory and retry failed runs with original inputs 2026-10-09 14:23:21 +01:00
alan ed93d055bf Run tasks directly with language-independent project tools 2026-10-09 01:48:46 +01:00
alan 0f30a001c2 Initial pipelines server, CLI, TUI, and Go installation 2026-10-08 22:22:17 +01:00