diff --git a/README.md b/README.md index 50fec222..4eff462e 100644 --- a/README.md +++ b/README.md @@ -188,7 +188,7 @@ pipelines is required on your client computer. Describe the stages, their inputs/outputs, and the tools they need. Templates (**b** on Home) are optional; a fresh installation has no published pipelines. - To skip authoring, use **:templates**, choose one of the five generic starters, + To skip authoring, use **:templates**, choose one of the four generic starters, and press **s** or Enter on **Create pipeline**. Review the prefilled name (edit it if needed), then press Enter to create—no separate naming or editor step. **e Customize** edits it first. Templates include a smaller graph preview; @@ -241,9 +241,9 @@ See [Tailscale troubleshooting](docs/ntfy.md#tailscale-address-times-out-while-l pipelines uninstall # remove Go binaries; keep history, drafts, and ntfy data ``` -## A three-stage pipeline +## A four-stage pipeline -**Plan → Implement → Review**. The first stage receives your task prompt. +**Plan → Implement → Review → Commit**. The first stage receives your task prompt. Later stages receive the previous output or fields selected by an explicit `memory` block. One outcome selects the next stage—no parallel branches. Definitions use Markdown `output`, `prompt`, `tools`, @@ -284,15 +284,16 @@ error -> #failure ``` ```` -The [complete three-stage example](examples/three-stage.md) adds the implementer -and reviewer: the planner may search/fetch official sources, the implementer has no -web tools and edits and runs checks from the planner's research, and the reviewer -independently inspects the real changes with its own shell access, receiving the -original task and the planner-owned acceptance checks rather than the -implementer's summary. The reviewer approves only when the applicable acceptance -checks were independently observed passing and no unresolved substantive issue -remains, and otherwise rejects with concrete findings. Review and explicitly -submit it: +The [complete three-stage example](examples/three-stage.md) adds the implementer, +reviewer and committer: the planner may search/fetch official sources, the +implementer has no web tools and edits and runs checks from the planner's research, +the reviewer independently inspects the real changes with its own shell access, +receiving the original task and the planner-owned acceptance checks rather than the +implementer's summary, and the committer stages the runner's captured changed paths +and commits locally without ever pushing. The reviewer approves only when the +applicable acceptance checks were independently observed passing and no unresolved +substantive issue remains, and otherwise rejects with concrete findings. Review and +explicitly submit it: ```sh pipelines validate examples/three-stage.md @@ -301,8 +302,8 @@ pipelines run "three stage" ``` Every submission is validated and creates an immutable version. Examples are -optional; nothing is imported automatically. The three stages give a visible -plan → implement → review path using your own accounts and repository; no +optional; nothing is imported automatically. The four stages give a visible +plan → implement → review → commit path using your own accounts and repository; no maintainer accounts or published definitions are assumed. Run from your repository directory; stage tools are enabled automatically. Edits support any language/file type and new files. No file list or typed RUN confirmation is diff --git a/docs/cli.md b/docs/cli.md index 35d489dd..8dda2363 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -126,7 +126,7 @@ pipelines edit # fuzzy pipeline selection pipelines edit "bug fix" # ask the harness what to change pipelines edit "bug fix" "Add a review step" pipelines new "release notes" "Summarize changes and notify my phone" -pipelines new "Text summary" --template summary +pipelines new "Add a feature" --template task ``` `new` opens the configured harness (Pi or OpenCode; host by default, or @@ -165,14 +165,14 @@ conflicting run and advising you to wait or use a different workspace (for example another checkout of the repository). Reusing the original request key reattaches idempotently. -Use `:templates` for five simple, manual-only starters: task, summary, review, -research, and plan/implement/review. **Submit template** (`s` or Enter) asks for a +Use `:templates` for four simple, manual-only starters: task, review, research, +and plan/implement/review. **Submit template** (`s` or Enter) asks for a name without opening an editor. Enter saves a retained draft, then a separate confirmation publishes it as a new pipeline. Esc before naming creates nothing; Esc at publication keeps the draft. **Customize** (`e`) opens the editor picker instead. Both paths inherit the configured default model and preserve existing pipelines/history. The research starter requires configured web search support. -Starter stages use goal-appropriate tools: the task stage and the three-stage +Starter stages use goal-appropriate tools: the task worker and the three-stage implementer edit and run checks, the task, planner and review stages may search/fetch official sources only when external knowledge is genuinely needed, the three-stage implementer has no web tools and follows the planner's research, @@ -180,8 +180,11 @@ the standalone review stage never edits or runs commands and says so explicitly, and the three-stage reviewer independently inspects the real changes and receives the planner-owned acceptance checks rather than the implementer's summary, approving only when the applicable acceptance checks were independently observed -passing and no unresolved substantive issue remains. Editing a bundled template -source changes no installed or published definition; a starter becomes an +passing and no unresolved substantive issue remains. The task and three-stage +templates end in a committer stage that stages the runner's captured changed paths +and commits locally without ever pushing; a genuine operator gate is a `type=human` +decision stage, never a model stage pretending to approve. Editing a bundled +template source changes no installed or published definition; a starter becomes an immutable version only after explicit authoring and publication. Home's `n` / `:new` and **Customize** (`e`) on a template first ask how to author the new diff --git a/docs/first-version-testing.md b/docs/first-version-testing.md index 90e75537..25cbdfa4 100644 --- a/docs/first-version-testing.md +++ b/docs/first-version-testing.md @@ -76,10 +76,11 @@ override the saved endpoint; check that you are connected to the intended server 1. Create a disposable repository, then launch `pipelines` **from that directory**. Docker must see the same absolute path; repositories on another computer are not uploaded automatically. -2. Use `:templates` and create the **summary** starter. Review its name and publish - explicitly. A fresh database is empty; templates are never auto-published. -3. Open the pipeline and use `:run` with a small task, such as summarizing a short - passage included in the task itself. Review the execution confirmation. +2. Use `:templates` and create the **review** starter (read-only, no repository + change). Review its name and publish explicitly. A fresh database is empty; + templates are never auto-published. +3. Open the pipeline and use `:run` with a small task, such as reviewing one short + file in the disposable repository. Review the execution confirmation. 4. Read the retained report and recorded visits. For an editing workflow, inspect File changes as well as actual file contents and any verification commands. Model-written claims alone are not verification. @@ -91,13 +92,13 @@ override the saved endpoint; check that you are connected to the intended server Phone reception is a separate manual check. Install/subscribe in the ntfy app and confirm receipt before depending on it. The installer confirms publishing, not -delivery to your phone. A summary starter does not send alerts automatically. +delivery to your phone. A review starter does not send alerts automatically. ## Known boundaries -- Sequential stages within a run; no parallel branches or built-in - human-approval stages. Independent runs in different workspaces do execute - concurrently. +- Sequential stages within a run; no parallel branches. `type=human` decision + stages pause a run for an operator's choice. Independent runs in different + workspaces do execute concurrently. - Cancellation/retry is **not rollback**. Retry starts from the entry stage with fresh sessions and the pinned definition; earlier side effects can repeat. - Diffs are bounded stage-start/stage-end observations, not full backups or proof @@ -144,7 +145,7 @@ python3 scripts/smoke-first-install.py The developer-only smoke check builds source binaries and installs the skill into a disposable directory, starts an isolated loopback server/database, checks build identity, empty initial -state, five starters, draft/validate/publish, identical ZIP import and restart +state, four starters, draft/validate/publish, identical ZIP import and restart persistence, then shuts it down and removes only its temporary directory. It does not call models, send alerts, update your installed clients or deploy Docker services. It does not replace real-provider, terminal interaction or fresh Docker-install tests. diff --git a/docs/pipeline-performance-review.md b/docs/pipeline-performance-review.md new file mode 100644 index 00000000..3ee44733 --- /dev/null +++ b/docs/pipeline-performance-review.md @@ -0,0 +1,371 @@ +# Pipeline performance and reliability review + +Read-only analysis of the live Pipelines run history, step records and +orchestration logs, on 2026-10-17. The only change is this document. + +## Sources and method + +All numbers come from commands run in this session against the running server +(`pipelines ping` reports `build.version v0.0.0-20261016212730-910a7aa798af`, +`commit 910a7aa798af3eecff72fcde28b9eafde470286c`, `dirty false`). + +- `pipelines runs --json` was captured once into + `/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json` + (223 entries, 201939 bytes) and reused for the taxonomy and trend. +- `pipelines stats --json` was captured once + (`/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json`): `total + 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3`. + `134+73+11+2+3 = 223`, so the run inventory reconciles with the totals. +- `pipelines inspect --json` was called exactly once per run id (223 calls, + saved under `artifacts/222/2/inspect/`); step intervals were aggregated in one + pass from those files. +- Orchestration logs: `/tmp/lane{A,B,C,D,E,Go,T2}.log`, `/tmp/merger.log`, + `/tmp/deploy-server.log`, and + `.../T/opencode/lane-{hg,hs}.log`, plus + `~/.local/share/pipelines/logs/server.log` (16.2 MB, 143080 lines). + +Quantile convention: **nearest-rank**, i.e. the p-quantile is element +`ceil(p*n)` (1-based) of the sorted list; the median is the p=0.5 element. +Durations are `finished_at - started_at` parsed with RFC3339/RFC3339Nano rules +(`Z` normalised), as in `internal/pipeline/search.go`. + +## 1. Failure taxonomy over the whole history + +86 runs have a terminal non-success status: `failed 73`, `canceled 11`, +`interrupted 2`. Every one is classified below from the run `diagnostic`, or — +where that is empty — from the failing step's outcome in its inspect record. +The class counts sum to 86; there is no unknown bucket. + +| class | count | run ids | +|---|---|---| +| `stage_outcome_blocked` | 23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 | +| `final_result_validation` | 19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 | +| `cancelled_by_operator` | 11 | 28,39,57,63,78,108,113,114,191,211,219 | +| `harness_transport_execution_error` | 10 | 73,74,75,77,80,149,150,152,195,202 | +| `harness_transport_timeout` | 6 | 33,36,38,59,101,138 | +| `review_loop_exhausted` | 5 | 31,70,96,97,127 | +| `stage_outcome_error` | 4 | 30,137,139,190 | +| `interrupted_by_deploy` | 2 | 13,24 | +| `invalid_agent_messages` | 2 | 2,5 | +| `harness_transport_readback` | 1 | 161 | +| `agent_quiescence_unconfirmed` | 1 | 43 | +| `session_transport_command_failed` | 1 | 4 | +| `duplicate_json_field` | 1 | 3 | + +Classifying diagnostics (quoted verbatim from the run records): + +- **stage_outcome_blocked** — a stage returns `blocked`, e.g. run 32/45 + (`implementer` final step `outcome='blocked'`, run `diagnostic` empty) and + run 105: `stage review returned outcome "blocked"`; run 115/141/145/151/154/ + 156/158/159/160/167/173/175/176/177/178/188/189/207/210: + `stage implementer returned outcome "blocked"` or + `stage ui_reviewer returned outcome "blocked"`. The blocked stages are + dominated by `implementer` and `ui_reviewer`, and their `result_preview` + shows the stage stopping deliberately — e.g. run 146: *"No screenshots/frames + exist for this run (artifacts/146/1 contains only the APK; artifacts/146/3 is + empty), and the task's device rules are mutually ex[c]lusive"*. +- **final_result_validation** — malformed/schema-rejected model result. Example, + run 8: `model stage implementer: FINAL RESULT VALIDATION: invalid model stage + result: invalid character 'I' looking for beginning of value at byte 1; unsafe + to repair incomplete values or a non-object response; completed edits and tool + side effects are retained and not undone; a full retry repeats work`. Two + sub-shapes: `unsafe to repair incomplete values or a non-object response` + (runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) and + `isolated format repair failed: retained OpenCode session ses_…: context + deadline exceeded` (runs 55,81,174,179,183,221 — 6 runs). Run 85/142 add a + third: `unknown result field`. +- **harness_transport_execution_error** — run 73/74/75/77/80/149/150/152/195/202: + `model stage : HARNESS TRANSPORT: harness transport failure: OpenCode + agent reported an execution error`. +- **harness_transport_timeout** — run 33: `model stage implementer: HARNESS + TRANSPORT: context deadline exceeded \n context deadline exceeded` (also + 36,38,59,101,138). +- **review_loop_exhausted** — run 31: `Pipelines stopped reviewer after 3 visits + on outcome rejected` (also 70,96,97,127). +- **cancelled_by_operator** — `context canceled` with status `canceled`. +- **interrupted_by_deploy** — run 13/24: `Server stopped; run was not resumed` + with status `interrupted`. +- **invalid_agent_messages** — run 2/5: `model stage planner: invalid agent + messages`. +- **duplicate_json_field** — run 3: `invalid model stage result: duplicate JSON + field "performance_review"`. +- **harness_transport_readback** — run 161: `harness transport failure: final + assistant readback had no completed result`. +- **agent_quiescence_unconfirmed** — run 43: `agent quiescence not confirmed; + inspect retained session before further work`. +- **session_transport_command_failed** — run 4: `model stage implementer: + OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait + command failed: CLI exit code 1`. + +`server.log` contains none of these diagnostic strings (bounded search: +`FINAL RESULT VALIDATION` 0, `HARNESS TRANSPORT` 0, `invalid agent messages` 0, +`stopped reviewer after` 0); it is a request/response log +(`command received`/`response sent`, plus `summarise: harness=opencode …`). +Diagnostics live in the run records, not the server log. + +## 2. Stage cost + +821 steps across the 222 finished runs (3 steps of the 3 live runs have an empty +`finished_at` and were excluded, not fabricated: runs 223 seq1 `implementer`, +222 seq2 `implementer`, 220 seq3 `ui_reviewer`). + +Total wall-hours by stage key (sum of `finished_at - started_at`), with the share +spent on repeat visits (`control.visit > 1`, i.e. review/retry loops) split out: + +| stage key | total (h) | single-pass (h) | repeat (h) | steps | +|---|---|---|---|---| +| implementer | 41.13 | 33.51 | 7.62 | 268 | +| reviewer | 20.65 | 15.69 | 4.96 | 219 | +| ui_reviewer | 15.09 | 13.84 | 1.25 | 134 | +| planner | 3.55 | 3.55 | 0.00 | 79 | +| committer | 1.95 | 1.95 | 0.00 | 98 | +| (worker/research/review/probe/report/opencode/pi) | 0.15 | 0.15 | 0.00 | 23 | + +The dominant stage keys are **implementer (41.1 h), reviewer (20.7 h) and +ui_reviewer (15.1 h)** — together 76.9 of 82.53 h. Repeat visits of the same +stage cost **13.83 h (16.8 %)**, almost all in implementer (7.62 h) and reviewer +(4.96 h). + +Per (pipeline, stage key) distribution (count / median / p90 / max seconds / +total hours). Only the two high-volume pipelines are shown; the full table was +produced by the aggregation command. + +| pipeline | stage | n | median | p90 | max | total (h) | +|---|---|---|---|---|---|---| +| Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 | +| Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 | +| Focused tweak - implement review commit | ui_reviewer | 80 | 290 | 1030 | 2158 | 9.96 | +| Focused tweak - implement review commit | committer | 62 | 29 | 178 | 644 | 1.08 | +| Pipelines maintenance - plan implement review | implementer | 89 | 375 | 884 | 3601 | 11.98 | +| Pipelines maintenance - plan implement review | reviewer | 78 | 242 | 503 | 12613 | 10.47 | +| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 | +| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 | +| Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 | +| Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 | + +Notable long tails: one `reviewer` step in Pipelines maintenance ran 12613 s +(3.5 h); one Pipelines-maintenance `implementer` step hit 3601 s (≈ the 1 h +stage timeout), while the Focused-tweak `implementer` max is 1801 s and its p90 +1414 s. The `ui_reviewer` p90 is high (1030 s in Focused tweak). + +## 3. Reliability trend, split at 910a7aa + +Boundary: `2026-10-16T21:27:00Z`, the deploy of commit 910a7aa. `pipelines ping` +confirms the running `build.commit` is 910a7aa. `deploy-server.log` corroborates: +`22:27:45 quiescing the lanes …` / `22:27:45 building from main (910a7aa Merge +lane-app2: …)` / `22:27:57 server healthy; deployed 910a7aa` (its clock is local +UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by `created_at`: the last +"before" run is 193 (`2026-10-16T21:21:33Z`), the first "after" is 194 +(`2026-10-16T21:28:00Z`). + +| side | n | running | failed | failed/(failed+succeeded) | non-success/terminal | +|---|---|---|---|---|---| +| before | 193 | 0 | 68 | 37.4 % | 40.9 % | +| after | 30 | 3 | 5 | 20.0 % | 25.9 % | + +Failure-class mix, before → after (counts): + +| class | before | after | +|---|---|---| +| stage_outcome_blocked | 21 | 2 | +| final_result_validation | 18 | 1 | +| harness_transport_execution_error | 8 | 2 | +| harness_transport_timeout | 6 | 0 | +| cancelled_by_operator | 9 | 2 | +| review_loop_exhausted | 5 | 0 | +| stage_outcome_error | 4 | 0 | +| interrupted_by_deploy | 2 | 0 | +| invalid_agent_messages | 2 | 0 | +| harness_transport_readback | 1 | 0 | +| agent_quiescence_unconfirmed | 1 | 0 | +| session_transport_command_failed | 1 | 0 | +| duplicate_json_field | 1 | 0 | + +Read literally the change correlates with fewer malformed-result failures: +`final_result_validation + duplicate_json_field + invalid_agent_messages` +falls from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is +**confounded and under-powered**: the after side is only 30 runs (3 still +running), the pipeline mix differs (after is mostly Focused tweak 15, Pipelines +maintenance 7, Merge-resolution 3 — the one-off template/docker/research +pipelines appear only before), and several before-side classes +(transport timeout, review-loop, deploy interruption) simply never recurred in +the short after window rather than being fixed by 910a7aa. The data is +consistent with the format/repair change helping; it does not establish +causation. A larger after sample is needed. + +## 4. Throughput limits (measured) + +**Busy-workspace waits.** The lane logs record `workspace busy; waiting to +launch ` at a fixed 30 s cadence. 897 such lines were counted across the +lane logs. Grouping them into wait episodes (same lane+file, gaps ≤ 120 s) +gives 28 episodes, 23 of which end with a `launched ` line and 5 still +open at log end. For the 23 resolved episodes the wait is +`launch_time - first_busy`: + +- n=23, median **992 s (16.5 min)**, p90 **2293 s (38.2 min)**, max + **3666 s (61.1 min)**, total **7.34 h**. + +Longest episodes: laneGo `s02-tui-omni-search.txt` waited 15:47:22→16:48:28 +(3666 s, 122 busy lines) before `16:48:28 launched s02-tui-omni-search.txt as +run 149`; laneA `s02-tui-omni-search.txt` waited 2825 s; laneGo +`search-server-tui-brief.txt` waited 2293 s. + +**Server-restart holds.** `holding: another lane is restarting the server` +appears **769** times; `building server and cli from main` 13 times; `server +redeployed and healthy` 10 times. These are lanes idling while a deploy or +rebuild owns the server. + +**Merge/main-lane waits.** `merge deferred: main lane busy` appears **11** +times (e.g. `/tmp/laneB.log:243 04:14:33 [laneB] merge deferred: main lane +busy`). `merge requested for …` appears 23 times; `MERGE CONFLICT` recorded 15 +times; `SYNC CONFLICT: lane could not take main's latest; continuing on the +older base` 44 times. + +**Merge/resolver activity.** `merger.log` shows the queue draining through +pipeline resolvers. 12 resolver runs have measurable durations, total +**5153 s (1.43 h)**: `run 176 20 s`, `177 20 s` (these "completed the merge" +immediately), versus `run 180 661 s`, `182 602 s`, `183 1062 s` (handed to a +pipeline). Three did not land: `179 did NOT land lane-app2`, `183 did NOT land +lane-go`, `221 did NOT land lane-c`. The `merge-queue` marker in the worktree +currently holds `lane-go`, `lane-app2`. + +**Device/emulator churn.** `FAILED once: - recovering the emulator and +retrying` appears 50 times and `FAILED twice: - recording and +continuing` 35 times (e.g. `02:53:47 [laneA] FAILED once: +q03-pane-opaque-A.txt - recovering the emulator and retrying`). + +**Real concurrency.** Overlapping step intervals show a maximum of **6 +simultaneous executing steps**, at `2026-10-16T14:47:22Z`, in runs +[134, 136, 137, 138, 139, 140]. 156 of the 222 finished runs had at least one +step overlapping another run. Using run wall-spans (`created_at`..`finished_at`) +the maximum overlap is also **6 concurrent runs** at the same instant, with 154 +runs overlapping. So despite many queued runs, the observed executing +concurrency never exceeded 6 — consistent with the lane count and the 897 +workspace-busy lines, and not with a memory/harness-capacity limit. + +## 5. Ranked improvements + +Ranked by measured impact. Items marked **hypothesis** have no direct +measurement behind them. + +1. **Reduce busy-workspace waiting (measured 7.34 h + 769 restart holds).** + Problem: 23 resolved waits, median 992 s, total 7.34 h, plus 769 + server-restart holds. Smallest change: lower the lane's fixed 30 s + busy-poll to a few seconds and/or keep a lane resident while a workspace is + held, rather than relaunching the poll each cycle. Expected effect: cut the + median multi-minute launch latency. Risk: more frequent polling and + contention. Locus: orchestration. Evidence: lane log cadence and the 897 + `workspace busy` lines. + +2. **Finish or budget the malformed-result repair path.** + Problem: `final_result_validation` is 19 of 86 non-success runs; in 6 of them + (55,81,174,179,183,221) the repair itself failed with + `isolated format repair failed: retained OpenCode session ses_…: context + deadline exceeded`, and in 11 (8,12,29,94,118,120,126,148,157,164,170) the + result was judged `unsafe to repair`. Smallest change: give the isolated + format-repair session its own longer deadline (or a streaming repair), so a + repairable-but-slow result is recovered instead of failing the run. Expected + effect: recover part of the 6 timeout-repair runs; the 11 "unsafe to repair" + remain a separate problem. Risk: longer tail latency on already-slow runs. + Locus: server code (`internal/stage`). Evidence: the six quoted diagnostics + and the before/after drop (10.9 %→3.3 %) for this cluster. + +3. **Cut repeat-visit cost in the review loop.** + Problem: repeat visits of the same stage cost 13.83 h of 82.53 h (16.8 %), + concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs + (31,70,96,97,127) failed outright with `Pipelines stopped reviewer after 3 + visits on outcome rejected`. Smallest change: identify which rejections were + advisory-only (no must-fix finding) and stop looping on those, or reduce the + reviewer max-visits for advisory findings. Expected effect: remove ~14 h of + repeat work and the 5 exhausted-loop failures, if rejections are often + non-blocking. Risk: a real defect could pass. Locus: pipeline definition + (`reviewer` control). Evidence: repeat/single-pass split in §2 and the + `review_loop_exhausted` ids. + +4. **Make the ui_reviewer receive (or not require) captured artifacts.** + Problem: `ui_reviewer` blocked is a large part of the 23-run + `stage_outcome_blocked` class (run 146: `artifacts/146/3 is empty`), and + `ui_reviewer` p90 is 1030 s. Smallest change: whatever captures screenshots + for a UI run must hand them to the `ui_reviewer` stage, and the stage should + not block when no device is attached. Expected effect: fewer blocked UI runs. + Risk: silently skipping visual verification. Locus: pipeline definition + + orchestration (artifact handoff). Evidence: run 146 preview and the blocked + ids; **partly hypothesis** — the artifact-empty condition is quoted for one + run only, not measured across all 23. + +5. **Triage the `implementer`/`ui_reviewer` "blocked" outcomes.** + Problem: `stage_outcome_blocked` is the single largest class (23/86, 26.7 %). + Smallest change: have the stage distinguish "environment/device unavailable" + (retryable) from "acceptance genuinely unmeetable" (terminal), so the + retryable subset is retried instead of failing the run. Expected effect: a + portion of 23 runs recovered. Risk: retry storms. Locus: pipeline definition + / server code. Evidence: the 23 ids and their `result_preview`; the + retryable/terminal split is **hypothesis** (not measured). + +6. **Harden the harness transport path.** + Problem: 17 non-success runs are transport failures — 10 + `execution error`, 6 `context deadline exceeded`, 1 `final assistant readback + had no completed result`. Smallest change: add bounded retry/backoff around + the OpenCode `…/wait` call for the generic "agent reported an execution + error" case. Expected effect: recover the 10 execution-error runs if they are + transient. Risk: retrying a genuinely broken session. Locus: server code + (harness adapter). Evidence: the quoted diagnostics and ids. Whether the + errors are transient is **hypothesis**. + +7. **Stabilise merge resolution.** + Problem: merge-resolution pipelines contributed blocked implementers (176, + 177, 178), `final_result_validation` (179, 183, 221), and resolver runs cost + 1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver's + `implementer` blocks on a conflict workspace (e.g. missing conflict markers + or write-path scope). Locus: merge-resolution pipeline definition. Evidence: + `merger.log` lines and the run diagnostics; root cause **hypothesis**. + +## 6. What could not be verified + +- **server.log content.** It is a request/response log and contains none of the + stage diagnostics (0 matches for all diagnostic phrases). It could not be used + to re-derive failures; the run records were used instead. To settle whether + transport errors correlate with request latency, the server would need to log + per-request duration (not currently present). +- **3 non-terminal runs excluded.** Runs 220, 222, 223 were `running` when the + history and steps were captured; their incomplete steps + (223 seq1, 222 seq2, 220 seq3) have empty `finished_at` and were excluded from + all distributions. Settle by re-reading `pipelines inspect` after they finish. +- **5 unresolved busy-wait episodes.** Five `workspace busy` sequences were + still open at log end (no matching `launched` line), so their final wait is + unknown. Settle by re-parsing the lane logs once the lanes go idle. +- **Blocked-class cause distribution.** Only one blocked run (146) has a quoted + cause; the other 22 were classified by `outcome=blocked` plus stage name. A + per-run `result_preview` pass would quantify the retryable vs terminal split. +- **After-side sample size.** The 910a7aa comparison rests on 30 runs (3 + running). A larger after-window sample is required before attributing the + failure-rate drop to the change rather than to pipeline mix. +- **No system-changing measurements were attempted.** No run was started, + retried, cancelled or reconfigured; the server, ntfy service, lanes and devices + were not touched, and the phone/emulator was not used. Measurements that would + require such changes (e.g. injecting a busy workspace to time a wait, or + forcing a transport error to test retry) were deliberately not performed. + +## Verification and limits + +Commands and observed results: + +- `pipelines ping --json` → build/commit `910a7aa…`. +- `pipelines runs --json` (1 call) → 223 runs; the run-status counts reconcile + with `pipelines stats --json` (`total 223, succeeded 134, failed 73, canceled + 11, interrupted 2, active 3`). +- `pipelines inspect --json` → 223 calls (one per run id), used for all step + intervals and the taxonomy fallback. +- Go verification, run with the repository's normal invocation from the module + root on go1.27.1/darwin-arm64: + - `go build ./...` → exit 0, no output. + - `go test ./...` → all packages `ok` (cached), exit 0; the three `cmd/*` + packages and `internal/wstest` have no test files. + No Go source was changed; these commands confirm the worktree was left + buildable. + +Limits: the after-side sample is small; several classes did not recur after the +boundary for reasons the data cannot separate from the format change; the +blocked-class causes and the transience of transport errors are hypotheses, not +measurements. Raw captures are under +`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` (outside the repo). diff --git a/examples/README.md b/examples/README.md index 21c8cf23..30b4e478 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,15 +1,27 @@ # Simple workflow templates -Five optional, technology-neutral starters. Nothing is installed as a published -pipeline automatically; all templates are manual-only and have at most three stages. +Four optional, technology-neutral starters. Nothing is installed as a published +pipeline automatically; all templates are manual-only and have at most four stages. | Template key | Workflow | Tools | | --- | --- | --- | -| `task` | Complete a task and report actual results | Repository read/edit and shell, plus conditional web search/fetch | -| `summary` | Summarize supplied text | None | +| `task` | Complete a task, then commit it locally | Work: read/edit/shell and conditional web search/fetch; committer: read/shell | | `review` | Read repository content and report findings | Read/glob/grep and conditional web search/fetch; no edit/shell | | `research` | Research a topic, then write a sourced report | Web search/fetch, then none | -| `three-stage` | Plan → implement → review and verify | Planner read/glob/grep + conditional web; implementer read/glob/grep/edit/shell with no web (follows the planner's research); reviewer independent shell + conditional web | +| `three-stage` | Plan → implement → review → commit | Planner read/glob/grep + conditional web; implementer read/glob/grep/edit/shell with no web (follows the planner's research); reviewer independent shell + conditional web; committer read/shell | + +Every workflow that changes the repository ends in a **committer** stage that +commits locally and never pushes. The committer stages the union of the runner's +server-captured changed paths (`input.server_changed_files`) and the producer's +declared `changed_files` — a declared list can be incomplete — then runs the +repository's required checks before committing. `task` and `three-stage` follow this; +`review` and `research` change nothing and so do not commit. + +Model and agent stages return the line-oriented result: the first non-empty line is +the declared outcome, a `# output` section holds the JSON output map for the next +stage, and any further `# ` section is retained verbatim as evidence. The +legacy `{ "outcome": ..., "output": {...} }` envelope is still accepted. See +`docs/model-results.md`. Web search/fetch is conditional: stages use it only when a task genuinely needs external knowledge such as accepted industry approaches, framework or library @@ -43,11 +55,15 @@ contain a fallback model; direct file submission uses those configs as written. Provider inference and search availability must be verified on your execution server. Parsing a definition does not prove its model/tools can run. -Templates do not impose a language, framework, schedule, notification setup, -commit, push, or deployment. Add those only when required by your own workflow. +Templates do not impose a language, framework, schedule, notification setup or +deployment, and they never push to a remote. Work-producing templates commit +locally through their committer stage; add a schedule, notifications or a +deployment step only when your own workflow requires it. + Editing these source files does not change any installed or published definition; each starter becomes an immutable version only after explicit authoring and publication. + Removing a template from this catalog does not delete published versions, run history, retained drafts, or OpenCode sessions. Nested routing examples used for renderer regression tests live in `internal/pipeline/testdata`, not this catalog. diff --git a/examples/catalog.go b/examples/catalog.go index 9bad1fe6..5070bcda 100644 --- a/examples/catalog.go +++ b/examples/catalog.go @@ -12,9 +12,8 @@ import ( var sources embed.FS var starters = []protocol.Template{ - {Key: "task", Name: "Complete a task", Description: "One stage to complete a task in your repository and report actual results."}, - {Key: "summary", Name: "Summarize text", Description: "One tool-free stage to summarize supplied text without inventing facts."}, - {Key: "three-stage", Name: "Plan, implement, review", Description: "Plan a change, implement it, then review and verify it. Any language or file type."}, + {Key: "task", Name: "Complete a task", Description: "Complete a task in your repository, then commit the change locally."}, + {Key: "three-stage", Name: "Plan, implement, review", Description: "Plan a change, implement it, review and verify it, then commit it locally. Any language or file type."}, {Key: "review", Name: "Review a repository", Description: "One read-only stage to review repository content and report findings."}, {Key: "research", Name: "Research a topic", Description: "Research a topic on the web, then write a sourced report. Requires web search support."}, } diff --git a/examples/catalog_test.go b/examples/catalog_test.go index e0927998..ba0e8a23 100644 --- a/examples/catalog_test.go +++ b/examples/catalog_test.go @@ -20,18 +20,18 @@ func TestReadmeThreeStageExampleIsValid(t *testing.T) { if err != nil { t.Fatal(err) } - if len(graph.Stages) != 3 || graph.Settings != nil { - t.Fatal("example must be manual-only and three stages") + if len(graph.Stages) != 4 || graph.Settings != nil { + t.Fatal("example must be manual-only and four stages") } } func TestCatalogContainsValidUnpublishedTemplates(t *testing.T) { items := examples.Catalog() - if len(items) != 5 { + if len(items) != 4 { t.Fatal("unexpected catalog size", len(items)) } seen := map[string]bool{} - wantStages := map[string]int{"task": 1, "summary": 1, "review": 1, "research": 2, "three-stage": 3} + wantStages := map[string]int{"task": 2, "review": 1, "research": 2, "three-stage": 4} for _, item := range items { if seen[item.Key] || item.Description == "" || item.Name == "" { t.Fatal("invalid template metadata", item.Key) @@ -44,8 +44,8 @@ func TestCatalogContainsValidUnpublishedTemplates(t *testing.T) { if (graph.Settings != nil) != item.Scheduled { t.Fatal("missing schedule warning", item.Key) } - if graph.Settings != nil || len(graph.Stages) > 3 { - t.Fatal("starters must be simple and manual-only", item.Key) + if graph.Settings != nil || len(graph.Stages) > 4 { + t.Fatal("starters must be small and manual-only", item.Key) } if len(graph.Stages) != wantStages[item.Key] { t.Fatal("unexpected/non-generic catalog entry", item.Key) @@ -89,9 +89,11 @@ func stageTools(t *testing.T, graph pipeline.Graph, key string) ([]string, bool) // Every bundled starter declares a tools block; an empty list is the intentional // deny-all tool-free role. var starterTools = map[string]map[string][]string{ - "task": {"worker": {"read", "glob", "grep", "edit", "shell", "websearch", "webfetch"}}, - "summary": {"summarize": {}}, - "review": {"review": {"read", "glob", "grep", "websearch", "webfetch"}}, + "task": { + "worker": {"read", "glob", "grep", "edit", "shell", "websearch", "webfetch"}, + "committer": {"read", "glob", "grep", "shell"}, + }, + "review": {"review": {"read", "glob", "grep", "websearch", "webfetch"}}, "research": { "research": {"websearch", "webfetch"}, "report": {}, @@ -100,6 +102,7 @@ var starterTools = map[string]map[string][]string{ "planner": {"read", "glob", "grep", "websearch", "webfetch"}, "implementer": {"read", "glob", "grep", "edit", "shell"}, "reviewer": {"read", "glob", "grep", "shell", "websearch", "webfetch"}, + "committer": {"read", "glob", "grep", "shell"}, }, } @@ -171,3 +174,36 @@ func TestThreeStageReviewerMemoryKeepsPlannerAcceptance(t *testing.T) { } } } + +// TestWorkProducingStartersEndInACommitter pins the guideline that a workflow +// which changes the repository commits locally through its final committer stage, +// staging the server-captured change set rather than only a model's declaration. +func TestWorkProducingStartersEndInACommitter(t *testing.T) { + for _, tc := range []struct{ key, producer string }{ + {"task", "worker"}, + {"three-stage", "implementer"}, + } { + graph := parsedCatalog(t)[tc.key] + last := graph.Stages[len(graph.Stages)-1] + if last.Key != "committer" { + t.Fatalf("%s must end in a committer stage, got %q", tc.key, last.Key) + } + if got := last.Edges["success"]; got != protocol.SuccessEnd { + t.Fatalf("%s committer success route = %q, want %q", tc.key, got, protocol.SuccessEnd) + } + if got := last.Memory["server_changed_files"]; got != "input.server_changed_files" { + t.Fatalf("%s committer must select input.server_changed_files, got %q", tc.key, got) + } + if got := last.Memory["changed_files"]; got != tc.producer+".changed_files" { + t.Fatalf("%s committer changed_files = %q, want %q", tc.key, got, tc.producer+".changed_files") + } + // The producer hands its work on rather than ending the run itself. + for _, s := range graph.Stages { + if s.Key == tc.producer { + if protocol.IsTerminal(s.Edges["success"]) { + t.Fatalf("%s/%s must hand success to a later stage, not %q", tc.key, tc.producer, s.Edges["success"]) + } + } + } + } +} diff --git a/examples/research.md b/examples/research.md index 23441928..ddf9a438 100644 --- a/examples/research.md +++ b/examples/research.md @@ -17,7 +17,11 @@ gaps and source/version applicability, and do not invent citations. Return block when required reliable evidence cannot be obtained, including empty or unreliable results. A single failed, blocked or anti-bot search may be replaced by a successful official-source webfetch that fully supplies the required -evidence. Return success only with reliable evidence. +evidence. Return success only with reliable evidence. Return the line-oriented +result: the first non-empty line is the declared outcome, a `# output` section +holds one JSON object with the output fields named above, and any further +`# ` section is retained verbatim as evidence (for example the queries +run and source excerpts). ``` ```tools @@ -49,6 +53,9 @@ Answer input.question using only the collected evidence and sources. Write a concise report with citations and clearly marked uncertainties. Respect the original constraints. Do not turn missing evidence into a confident answer. Return success for a supported report or blocked when the evidence is insufficient. +Return the line-oriented result: the first non-empty line is the declared outcome, +a `# output` section holds one JSON object with the output fields named above, and +any further `# ` section is retained verbatim as evidence. ``` ```tools diff --git a/examples/review.md b/examples/review.md index 84fdd18c..13a52ca1 100644 --- a/examples/review.md +++ b/examples/review.md @@ -21,7 +21,11 @@ hypotheses. State explicitly that no commands were run and that runtime behavior and checks are not tested; source inspection is not proof of execution. Return success when the review is complete, even if it finds issues; return blocked if the requested content or required official evidence cannot be obtained rather than -fabricating it. Do not claim issues have been fixed. +fabricating it. Do not claim issues have been fixed. Return the line-oriented +result: the first non-empty line is the declared outcome, a `# output` section +holds one JSON object with the output fields named above, and any further +`# ` section is retained verbatim as evidence (for example the files read +and the sources checked, with their dates/versions). ``` ```tools diff --git a/examples/summary.md b/examples/summary.md deleted file mode 100644 index 309271ea..00000000 --- a/examples/summary.md +++ /dev/null @@ -1,28 +0,0 @@ -# stage summarize - -```output -{ summary, key_points, open_questions } -``` - -```prompt -Summarize the text supplied in input.prompt. Preserve its meaning, important -facts, decisions, and unresolved questions. Follow any requested length or -format. Use only the supplied information; do not invent sources or facts. -Return blocked if no text to summarize was supplied. Otherwise return success. -``` - -```tools -``` - -```configs -provider=opencode-go -model=deepseek-v4.1-flash -retries=2 -timeout=1h -``` - -```edges -success -> #success -blocked -> #failure -error -> #failure -``` diff --git a/examples/task.md b/examples/task.md index 5ce00be8..9395a751 100644 --- a/examples/task.md +++ b/examples/task.md @@ -1,7 +1,7 @@ # stage worker ```output -{ task, summary, changed_files, evidence, remaining_questions } +{ task, summary, changed_files, remaining_questions } ``` ```prompt @@ -16,20 +16,25 @@ existing dependencies/APIs/helpers over bespoke or copy-pasted code, and avoid unnecessary new dependencies rather than treating useful supported standard libraries as off-limits; justify any genuinely needed custom implementation, compatibility exception or dependency choice with its tradeoffs and do not force -coincidental -lookalikes into a shared abstraction. Research official primary sources with -websearch/webfetch only when the task genuinely needs external knowledge, -recording source URLs and applicability/version evidence or stating why none was -needed; keep queries free of private code and secrets. Return blocked only when -required reliable evidence cannot be obtained; a single failed, blocked or +coincidental lookalikes into a shared abstraction. Research official primary +sources with websearch/webfetch only when the task genuinely needs external +knowledge, recording source URLs and applicability/version evidence or stating why +none was needed; keep queries free of private code and secrets. Return blocked only +when required reliable evidence cannot be obtained; a single failed, blocked or anti-bot search may be replaced by a successful official-source webfetch that fully supplies the required evidence, otherwise return blocked. Fix causes rather than suppressing errors or weakening checks, and avoid speculative abstractions. -Preserve unrelated work. Do not commit, push, deploy, or perform destructive -actions. Verify changes with relevant checks and report actual commands, observed -results and untested limits in evidence. Never claim an unexecuted check passed. -Return blocked if the task or required tools are unavailable; otherwise return -success with actual results. +Preserve unrelated work and leave the commit to the committer stage: do not commit, +push, deploy, or perform destructive actions. Save any artifacts you produce +(reports, logs, screenshots) into the artifacts directory named in the runtime +context; they are attached to this stage. Verify the change with the repository's +relevant checks and report the exact commands and observed results in an +`# evidence` section; never claim an unexecuted check passed. Return the +line-oriented result: the first non-empty line is the declared outcome, a +`# output` section holds one JSON object with the output fields named above, and +any further `# ` section is retained verbatim as evidence. Return blocked +if the task or required tools are unavailable; otherwise return success with +actual results. ``` ```tools @@ -49,6 +54,54 @@ retries=2 timeout=1h ``` +```edges +success -> committer +blocked -> #failure +error -> #failure +``` + +# stage committer + +```output +{ committed, commit, changed_files, pre_commit_checks } +``` + +```prompt +Commit this run's change locally. Never push, tag, deploy or rewrite history. +Select the run's changed paths from input.server_changed_files (the runner's +captured evidence) and the worker's declared changed_files, and stage their union +of paths that still exist, keeping any exclusions. Inspect the staged diff before +committing; never commit unrelated work, secrets or generated artifacts. Pre-commit +checklist: confirm the staged paths are this run's change; run the repository's +required checks and record the exact commands and observed results in an +`# evidence` section; do not commit if a required check fails, return blocked +with the failing output instead. If nothing is staged, do not create an empty +commit; return success with a clear note. Otherwise commit with a concise message +naming this run's change and report the commit hash. Return the line-oriented +result: the first non-empty line is the declared outcome, a `# output` section +holds one JSON object with the output fields named above, and any further +`# ` section is retained verbatim as evidence. Never claim an unexecuted +check passed or fabricate a commit. +``` + +```memory +{"task":"input.prompt","changed_files":"worker.changed_files","server_changed_files":"input.server_changed_files"} +``` + +```tools +read +glob +grep +shell +``` + +```configs +provider=opencode-go +model=deepseek-v4.1-flash +retries=2 +timeout=1h +``` + ```edges success -> #success blocked -> #failure diff --git a/examples/three-stage.md b/examples/three-stage.md index acfbcfbc..f23e25f6 100644 --- a/examples/three-stage.md +++ b/examples/three-stage.md @@ -23,7 +23,10 @@ fully supplies the required evidence, otherwise return blocked. Limit repository inspection to relevant repository paths; never scan filesystem or mount roots, and use commands supported by the actual execution environment rather than guessed platform flags (a working instruction, not an OS sandbox). Preserve the -user's task and constraints; do not invent evidence. +user's task and constraints; do not invent evidence. Return the line-oriented +result: the first non-empty line is the declared outcome, a `# output` section +holds one JSON object with the output fields named above, and any further +`# ` section is retained verbatim as evidence. ``` ```tools @@ -50,7 +53,7 @@ error -> #failure # stage implementer ```output -{ task, plan, acceptance_checks, changed_files, implementation_summary, evidence } +{ task, plan, acceptance_checks, changed_files, implementation_summary } ``` ```prompt @@ -68,11 +71,17 @@ cancellation, error handling and bounded performance where applicable. Limit repository inspection and verification to relevant repository paths; never scan filesystem or mount roots, and use commands supported by the actual execution environment rather than guessed platform flags (a working instruction, not an OS -sandbox). Run the relevant checks with the stage tools and record the exact -commands, observed results and any untested limits in evidence. Preserve the -task, plan and acceptance checks and record actual changes in changed_files; do -not commit, push, deploy or destroy unrelated work. Return blocked when the -required edit scope or tools are unavailable. +sandbox). Save any artifacts you produce (reports, logs, screenshots) into the +artifacts directory named in the runtime context; they are attached to this stage. +Run the relevant checks with the stage tools and record the exact commands, +observed results and any untested limits in an `# evidence` section; never +claim an unexecuted check passed. Preserve the task, plan and acceptance checks and +record actual changes in changed_files. Do not commit, push, deploy or destroy +unrelated work; the committer stage commits this run's change. Return the +line-oriented result: the first non-empty line is the declared outcome, a +`# output` section holds one JSON object with the output fields named above, and +any further `# ` section is retained verbatim as evidence. Return blocked +when the required edit scope or tools are unavailable. ``` ```tools @@ -113,18 +122,23 @@ unnecessary bespoke solutions, not only overall architecture. Support each findi with a concrete location, evidence, severity and remedy; distinguish blocking evidence gaps from nonblocking personal-style nits and explain legitimate deviations. Run relevant checks with the stage tools yourself and report what was -not tested; do not accept a prior stage's summary as proof. Limit inspection and +not tested in an `# evidence` section; do not accept a prior stage's summary as +proof and never claim an unexecuted check passed. Limit inspection and verification to relevant repository paths; never scan filesystem or mount roots, and run commands supported by the actual execution environment rather than -guessed platform flags (working instructions, not an OS sandbox guarantee). -Return success only when the applicable acceptance checks were independently -observed passing and no unresolved substantive design, reuse, duplication, -workaround or correctness issue remains; return rejected with concrete findings -for failing checks, missing REQUIRED verification, or substantive code-quality -issues. Optional or +guessed platform flags (working instructions, not an OS sandbox guarantee). Save +any artifacts you produce (reports, logs, screenshots) into the artifacts +directory named in the runtime context; they are attached to this stage. Return +the line-oriented result: the first non-empty line is the declared outcome, a +`# output` section holds one JSON object with the output fields named above, and +any further `# ` section is retained verbatim as evidence. Return success +only when the applicable acceptance checks were independently observed passing and +no unresolved substantive design, reuse, duplication, workaround or correctness +issue remains; return rejected with concrete findings for failing checks, missing +REQUIRED verification, or substantive code-quality issues. Optional or out-of-scope checks may be recorded untested without forcing endless work. Use websearch/webfetch only when an official standard or documentation check is -needed. Do not edit files, claim unexecuted checks, commit, push or deploy. +needed. Do not edit files, commit, push or deploy. ``` ```memory @@ -148,7 +162,55 @@ timeout=1h ``` ```edges -success -> #success +success -> committer rejected -> #failure error -> #failure ``` + +# stage committer + +```output +{ committed, commit, changed_files, pre_commit_checks } +``` + +```prompt +Commit this run's change locally. Never push, tag, deploy or rewrite history. +Select the run's changed paths from input.server_changed_files (the runner's +captured evidence) and the implementer's declared changed_files, and stage their +union of paths that still exist, keeping any exclusions. Inspect the staged diff +before committing; never commit unrelated work, secrets or generated artifacts. +Pre-commit checklist: confirm the staged paths are this run's change; run the +repository's required checks and record the exact commands and observed results in +a `# evidence` section; do not commit if a required check fails, return +blocked with the failing output instead. If nothing is staged, do not create an +empty commit; return success with a clear note. Otherwise commit with a concise +message naming this run's change and report the commit hash. Return the +line-oriented result: the first non-empty line is the declared outcome, a +`# output` section holds one JSON object with the output fields named above, and +any further `# ` section is retained verbatim as evidence. Never claim an +unexecuted check passed or fabricate a commit. +``` + +```memory +{"task":"input.prompt","changed_files":"implementer.changed_files","server_changed_files":"input.server_changed_files"} +``` + +```tools +read +glob +grep +shell +``` + +```configs +provider=opencode-go +model=deepseek-v4.1-flash +retries=2 +timeout=1h +``` + +```edges +success -> #success +blocked -> #failure +error -> #failure +``` diff --git a/internal/author/skill/SKILL.md b/internal/author/skill/SKILL.md index 4eb05f32..b95a3731 100644 --- a/internal/author/skill/SKILL.md +++ b/internal/author/skill/SKILL.md @@ -63,7 +63,7 @@ and `references/format.md` (in parallel when possible); for notifications also r | Parallel lanes, worktrees, sync/merge, restart replay | `references/lanes.md` | | Diagnose a wedged, blocked or failed run | `references/debugging.md` | | Publish artefacts, LAN URLs, install boundaries | `references/delivery.md` | -| Definition format, tools, permissions, routing, schedules, budgets | `references/format.md` | +| Result format, tools, permissions, committer, routing, schedules, budgets | `references/format.md` | | Bounded loops, independent review, selected shared context | `references/control.md` | | OpenCode/Pi configuration, model defaults, harness limits | `references/harnesses.md` | | Phone alerts and report/file attachments | `references/notifications.md` | diff --git a/internal/author/skill/references/format.md b/internal/author/skill/references/format.md index c29573a9..0e101011 100644 --- a/internal/author/skill/references/format.md +++ b/internal/author/skill/references/format.md @@ -47,6 +47,39 @@ must be reachable from entry and have a possible path to a terminal. Use bounded runner-owned `limits` for feedback loops, never revision counters in model prompts/outputs. Read `references/control.md` for the exact format. +## Result format + +A model or agent stage returns a line-oriented document: the first non-empty line +is the declared outcome (one of the stage's non-error `edges` keys), sections begin +with `# `, and the required `# output` section holds one JSON object — the +output map that later stages and `memory` select from. Every other heading becomes +an evidence field keyed by its normalised heading (lowercased, spaces to +underscores) and is retained verbatim, so prose, code fences, quotes and embedded +JSON need no escaping. Extra sections are best-effort and never fail the stage. + +````markdown +success + +# output +{"task": "x", "changed_files": ["a.go"]} + +# evidence +go test ./... — PASS (observed output) +```` + +The legacy single JSON envelope `{ "outcome": "...", "output": {...} }` is still +accepted unchanged. The parser chooses by precedence: a response whose first +non-empty character is `{` (after an optional surrounding fence) uses the JSON path; +a `# ` heading that appears before any `{` uses the line path. A missing outcome, an +undeclared outcome, or a missing/invalid `# output` fails the stage with +`FINAL RESULT VALIDATION`; salvageable sections are retained and the parse status is +recorded under the reserved key `result_parse` (`parsed`, `partial`, `failed`). The +full raw response is saved as `raw-output.md` in the stage's artifacts directory. Use +the line form when the evidence is prose, logs or code; never fabricate a result, +never flatten the output fields alongside the outcome, and save any reports, logs or +screenshots you produce into the artifacts directory the runtime names +(`references/delivery.md`). + Model configs may set `harness=opencode` or `harness=pi`; omission preserves historical OpenCode execution. Read `references/harnesses.md` for setup/limits. Use the exact provider/model IDs validated during server setup. New drafts/templates @@ -89,7 +122,8 @@ may read/edit/shell and conditionally use `websearch`/`webfetch`; a planner may read plus conditional web search/fetch; a three-stage implementer has no web tools and works from the planner's research with local read/glob/grep/edit/shell; a reviewer may read plus conditional web; a source-only review stage has no -edit/shell; a research stage uses web search/fetch. Never force web access for +edit/shell; a research stage uses web search/fetch; a committer has read/glob/grep +and shell but no edit. Never force web access for trivial, fully understood local edits, and return blocked only when required reliable evidence cannot be obtained rather than whenever one search tool fails. Keep examples and prompts language-neutral unless the task itself asks for a @@ -145,6 +179,60 @@ a repeated decision for an already-decided visit does not double-apply. Cancel a retry keep working while waiting and after a decision, and a human stage is never killed by a stage timeout or `max_visits` (the shared run budget still applies). +## Committer stages + +A workflow that changes the repository ends in a committer: a normal model stage +whose prompt commits the run's change locally and never pushes, tags, deploys or +rewrites history. It is not a native type. Declare `shell` (and `read`/`glob`/`grep` +when it inspects content) and no `edit`, so a failed check is reported rather than +patched, and select the runner's captured change set alongside the producer's +declared list: + +````markdown +# stage committer + +```prompt +Commit this run's change locally; never push. +``` + +```memory +{"changed_files":"implementer.changed_files","server_changed_files":"input.server_changed_files"} +``` + +```tools +read +glob +grep +shell +``` + +```edges +success -> #success +blocked -> #failure +error -> #failure +``` +```` + +State this pre-commit checklist in the prompt: + +1. Stage the union of `input.server_changed_files` (the authoritative, + server-captured set) and the declared `changed_files`, restricted to paths that + still exist, keeping every exclusion. A declared list can be incomplete, and a + captured path it omitted is still this run's change (`references/control.md`). +2. Inspect the staged diff before committing; never commit unrelated work, secrets + or generated artifacts. +3. Run the repository's required checks and record the exact commands and observed + results as evidence. Do not commit when a required check fails; return `blocked` + with the failing output instead. +4. Create no empty commit: when nothing is staged, return `success` with a clear + note. +5. Commit with a concise message naming the run's change and report the commit hash; + never push, tag, rebase or amend. + +Merging is the operator's or a lane's job, not the committer's: it commits locally +and stops (`references/orchestration.md`, `references/lanes.md`). The server does +not create the commit and cannot enforce the message; a reviewer checks the rules. + Malformed final JSON has bounded recovery even with `retries=0`; it never changes verdict/evidence. Recovery tries a deterministic local pass first (append missing closing delimiters, drop one trailing comma, or insert the single comma/colon the @@ -162,9 +250,10 @@ retained deny-all, no-inference compatibility probe identifies known format incompatibility. Incompatible native backends and V2 require strict local envelope validation. Do not promise recovery of truncated strings, missing evidence or invalid outcomes. -Return the strict `{ "outcome": "declared-edge-key", "output": { ... } }` -envelope. Do not treat tool-call/DSML markup, plain prose or flattened output as -success, or rerun tool work to fix a response's formatting. +Return the line-oriented result format above, or the strict +`{ "outcome": "declared-edge-key", "output": { ... } }` envelope. Do not treat +tool-call/DSML markup, plain prose or flattened output as success, or rerun tool +work to fix a response's formatting. ## Scheduling diff --git a/internal/author/skill/references/operations.md b/internal/author/skill/references/operations.md index c44f6673..b538fa58 100644 --- a/internal/author/skill/references/operations.md +++ b/internal/author/skill/references/operations.md @@ -10,7 +10,7 @@ parallel/distributed task engine or a guarantee that model reports are correct. - Drafts are editable Markdown plus JSON metadata. Publishing validates and creates immutable versions; drafts remain retained. Keep context create-only/base-version - guards. Five optional generic starters are task, summary, review, research and + guards. Four optional generic starters are task, review, research and three-stage; a fresh database starts empty. Templates can be created directly with an editable prefilled name or customized first. Editing a bundled repository template changes no installed or published definition; only explicit authoring diff --git a/internal/author/skill/references/orchestration.md b/internal/author/skill/references/orchestration.md index 8a7db3d0..2431c369 100644 --- a/internal/author/skill/references/orchestration.md +++ b/internal/author/skill/references/orchestration.md @@ -119,7 +119,11 @@ Adapt the scope and publishing lines per item; keep the rest. files. It must select `input.server_changed_files` (the runner's captured evidence, `references/control.md`) alongside the implementer's declared `changed_files` and stage their union: an implementer's declared list can be incomplete, and a captured - path it omitted is still this run's change. + path it omitted is still this run's change. Its prompt carries the pre-commit + checklist in `references/format.md`; it commits locally and stops, never pushing. +- A run cannot approve its own work. When an item genuinely needs an operator gate + before it continues or commits, insert a `type=human` decision stage + (`references/format.md`); never invent a model stage that pretends to approve. - Choose per item: a logic, server or format change usually wants the planner (a plan/implement/review pipeline); a small, fully understood tweak may use the leaner implementer-first variant. Both keep all review stages. diff --git a/internal/cli/help.go b/internal/cli/help.go index ff7a510e..f23b011f 100644 --- a/internal/cli/help.go +++ b/internal/cli/help.go @@ -58,7 +58,7 @@ Use pipelines help for its flags. Names with spaces must be quoted. Examples: pipelines edit "bug fix" "Add a review step" pipelines new "release notes" "Summarize changes and notify my phone" - pipelines new "Text summary" --template summary --no-open --json --yes + pipelines new "Add a feature" --template task --no-open --json --yes pipelines validate /path/to/draft.md --json pipelines submit /path/to/draft.md pipelines run "bug fix" "Fix the layout" --json diff --git a/internal/pipeline/examples_test.go b/internal/pipeline/examples_test.go index 3a521329..a8da3401 100644 --- a/internal/pipeline/examples_test.go +++ b/internal/pipeline/examples_test.go @@ -10,7 +10,7 @@ import ( ) func TestShowcaseDefinitions(t *testing.T) { - for _, name := range []string{"task", "summary", "review", "research", "three-stage"} { + for _, name := range []string{"task", "review", "research", "three-stage"} { t.Run(name, func(t *testing.T) { source, err := os.ReadFile(filepath.Join("..", "..", "examples", name+".md")) if err != nil { diff --git a/internal/server/templates_test.go b/internal/server/templates_test.go index 75d51f25..6d7910e8 100644 --- a/internal/server/templates_test.go +++ b/internal/server/templates_test.go @@ -44,7 +44,7 @@ func TestTemplatesDoNotSeedDatabaseAndCreateOnlyPreservesExisting(t *testing.T) if err := wsjson.Read(ctx, conn, &response); err != nil { t.Fatal(err) } - if response.Type != protocol.Templates || len(response.Templates) != 5 { + if response.Type != protocol.Templates || len(response.Templates) != 4 { t.Fatal("missing catalog", response.Type, len(response.Templates)) } items, err := store.List(ctx) diff --git a/internal/tui/examples_test.go b/internal/tui/examples_test.go index d2865160..6a99b562 100644 --- a/internal/tui/examples_test.go +++ b/internal/tui/examples_test.go @@ -11,7 +11,7 @@ import ( ) func TestShowcaseGraphsRender(t *testing.T) { - for _, name := range []string{"task", "summary", "review", "research", "three-stage"} { + for _, name := range []string{"task", "review", "research", "three-stage"} { t.Run(name, func(t *testing.T) { source, err := os.ReadFile(filepath.Join("..", "..", "examples", name+".md")) if err != nil { diff --git a/internal/tui/template_submit_test.go b/internal/tui/template_submit_test.go index fbabc0a8..c57a0816 100644 --- a/internal/tui/template_submit_test.go +++ b/internal/tui/template_submit_test.go @@ -38,7 +38,7 @@ func TestTemplateSubmitNamesRetainsAndExplicitlyPublishesSelectedTemplate(t *tes m.templateOpen, m.keys.templates = true, true m.defaultModel = "fixture/model" for i, item := range m.templates { - if item.Key == "summary" { + if item.Key == "review" { m.templateIndex = i } } @@ -72,7 +72,7 @@ func TestTemplateSubmitNamesRetainsAndExplicitlyPublishesSelectedTemplate(t *tes m = updated.(Model) updated, _ = m.Update(tea.KeyMsg{Type: tea.KeyCtrlK}) m = updated.(Model) - updated, _ = m.Update(tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("Simple summary")}) + updated, _ = m.Update(tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("Simple review")}) m = updated.(Model) updated, cmd = m.Update(tea.KeyMsg{Type: tea.KeyEnter}) m = updated.(Model) @@ -81,7 +81,7 @@ func TestTemplateSubmitNamesRetainsAndExplicitlyPublishesSelectedTemplate(t *tes t.Fatal("single create confirmation did not send or failed to retain draft") } retained, err := draft.Load(m.pendingDraft.path) - if err != nil || retained.Name != "Simple summary" || retained.Server != url || !retained.CreateOnly || retained.BaseVersion != 0 { + if err != nil || retained.Name != "Simple review" || retained.Server != url || !retained.CreateOnly || retained.BaseVersion != 0 { t.Fatal("template draft metadata lost", retained, err) } text, err := draft.Read(retained.Path) diff --git a/scripts/smoke-first-install.py b/scripts/smoke-first-install.py index 49127603..e13ec5b3 100644 --- a/scripts/smoke-first-install.py +++ b/scripts/smoke-first-install.py @@ -78,8 +78,8 @@ def main(): assert pong["build"] == version, "Client/server build mismatch" assert not rpc("list").get("pipelines"), "Fresh database must be empty" assert not rpc("runs").get("runs"), "Fresh database must have no runs" - assert len(rpc("templates")["templates"]) == 5 - draft = rpc("new", "First test", "Summarize input", "--template", "summary", "--no-open", "--yes") + assert len(rpc("templates")["templates"]) == 4 + draft = rpc("new", "First test", "Review the repository", "--template", "review", "--no-open", "--yes") assert not rpc("list").get("pipelines"), "Draft creation must not publish" rpc("validate", draft["path"]) rpc("submit", draft["path"], "--yes") @@ -100,7 +100,7 @@ def main(): assert not rpc("runs").get("runs"), "Smoke check must not execute a workflow" finally: stop(process) - print("PASS developer lifecycle fixture: isolated source binaries/skill, offline version, empty DB, five templates, draft/validate/publish, ZIP identical import, restart persistence. Not an installation option; no inference or notifications.") + print("PASS developer lifecycle fixture: isolated source binaries/skill, offline version, empty DB, four templates, draft/validate/publish, ZIP identical import, restart persistence. Not an installation option; no inference or notifications.") if __name__ == "__main__":