diff --git a/docs/pipeline-performance-review.md b/docs/pipeline-performance-review.md index 3ee44733..ec203403 100644 --- a/docs/pipeline-performance-review.md +++ b/docs/pipeline-performance-review.md @@ -6,43 +6,52 @@ orchestration logs, on 2026-10-17. The only change is this document. ## Sources and method All numbers come from commands run in this session against the running server -(`pipelines ping` reports `build.version v0.0.0-20261016212730-910a7aa798af`, -`commit 910a7aa798af3eecff72fcde28b9eafde470286c`, `dirty false`). +(`pipelines ping --json` reports `build.version +v0.0.0-20261016212730-910a7aa798af`, `commit +910a7aa798af3eecff72fcde28b9eafde470286c`, `dirty false`). -- `pipelines runs --json` was captured once into +- `pipelines runs --json` was captured twice: once into `/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json` - (223 entries, 201939 bytes) and reused for the taxonomy and trend. -- `pipelines stats --json` was captured once - (`/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json`): `total - 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3`. - `134+73+11+2+3 = 223`, so the run inventory reconciles with the totals. + (223 entries, 201939 bytes) and once into + `/Users/alan/.local/share/pipelines/data/artifacts/222/4/runs.json` + (223 entries). Both captures were reused for the taxonomy, trend and + concurrency. +- `pipelines stats --json` was captured twice: + `/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json` reported + `total 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3`, + and `/Users/alan/.local/share/pipelines/data/artifacts/222/4/stats.json` + reported `total 223, succeeded 135, failed 73, canceled 11, interrupted 2, + active 2`. Both inventories reconcile with the run totals + (`134+73+11+2+3 = 223` and `135+73+11+2+2 = 223`). - `pipelines inspect --json` was called exactly once per run id (223 calls, - saved under `artifacts/222/2/inspect/`); step intervals were aggregated in one + saved under `artifacts/222/4/inspect/`); step intervals were aggregated in one pass from those files. -- Orchestration logs: `/tmp/lane{A,B,C,D,E,Go,T2}.log`, `/tmp/merger.log`, - `/tmp/deploy-server.log`, and - `.../T/opencode/lane-{hg,hs}.log`, plus - `~/.local/share/pipelines/logs/server.log` (16.2 MB, 143080 lines). +- Orchestration logs: `/tmp/lane{A,B,C,D,E,Go,T2,Chat}.log`, `/tmp/merger.log`, + `/tmp/deploy-server.log`, and the `lane-{hg,hs}.log` files in + `/private/var/folders/jp/q0qvjyg54fl_c5vhhk1ylz2m0000gn/T/opencode/`, plus + `~/.local/share/pipelines/logs/server.log` (16,282,789 bytes, 143,788 lines). Quantile convention: **nearest-rank**, i.e. the p-quantile is element `ceil(p*n)` (1-based) of the sorted list; the median is the p=0.5 element. Durations are `finished_at - started_at` parsed with RFC3339/RFC3339Nano rules -(`Z` normalised), as in `internal/pipeline/search.go`. +(`Z` normalised, empty string = absent), as in `internal/pipeline/search.go`. ## 1. Failure taxonomy over the whole history -86 runs have a terminal non-success status: `failed 73`, `canceled 11`, -`interrupted 2`. Every one is classified below from the run `diagnostic`, or — -where that is empty — from the failing step's outcome in its inspect record. -The class counts sum to 86; there is no unknown bucket. +The inventory holds **223 runs** (`running 2`, `succeeded 135`, `failed 73`, +`canceled 11`, `interrupted 2`). **86 runs have a terminal non-success status** +(`failed 73`, `canceled 11`, `interrupted 2`). Every one is classified below from +the run `diagnostic`, or — where that is empty (runs 30, 32, 45) — from the +failing step's `outcome` in its `pipelines inspect` record. The class counts sum +to 86; there is no unknown bucket. | class | count | run ids | |---|---|---| | `stage_outcome_blocked` | 23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 | | `final_result_validation` | 19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 | -| `cancelled_by_operator` | 11 | 28,39,57,63,78,108,113,114,191,211,219 | | `harness_transport_execution_error` | 10 | 73,74,75,77,80,149,150,152,195,202 | | `harness_transport_timeout` | 6 | 33,36,38,59,101,138 | +| `cancelled_by_operator` | 11 | 28,39,57,63,78,108,113,114,191,211,219 | | `review_loop_exhausted` | 5 | 31,70,96,97,127 | | `stage_outcome_error` | 4 | 30,137,139,190 | | `interrupted_by_deploy` | 2 | 13,24 | @@ -54,41 +63,44 @@ The class counts sum to 86; there is no unknown bucket. Classifying diagnostics (quoted verbatim from the run records): -- **stage_outcome_blocked** — a stage returns `blocked`, e.g. run 32/45 - (`implementer` final step `outcome='blocked'`, run `diagnostic` empty) and - run 105: `stage review returned outcome "blocked"`; run 115/141/145/151/154/ - 156/158/159/160/167/173/175/176/177/178/188/189/207/210: - `stage implementer returned outcome "blocked"` or - `stage ui_reviewer returned outcome "blocked"`. The blocked stages are - dominated by `implementer` and `ui_reviewer`, and their `result_preview` - shows the stage stopping deliberately — e.g. run 146: *"No screenshots/frames - exist for this run (artifacts/146/1 contains only the APK; artifacts/146/3 is - empty), and the task's device rules are mutually ex[c]lusive"*. -- **final_result_validation** — malformed/schema-rejected model result. Example, - run 8: `model stage implementer: FINAL RESULT VALIDATION: invalid model stage - result: invalid character 'I' looking for beginning of value at byte 1; unsafe - to repair incomplete values or a non-object response; completed edits and tool - side effects are retained and not undone; a full retry repeats work`. Two - sub-shapes: `unsafe to repair incomplete values or a non-object response` +- **stage_outcome_blocked** — a stage ends with `outcome=blocked`. The split by + blocked stage is **implementer 18, ui_reviewer 4, review 1**. Run 105: + `stage review returned outcome "blocked"`; runs 32/45 have an empty run + `diagnostic` and an `implementer` final step `outcome='blocked'`; the rest + carry `stage implementer returned outcome "blocked"` or `stage ui_reviewer + returned outcome "blocked"`. Run 146 shows a deliberate stop: *"No + screenshots/frames exist for this run (artifacts/146/1 contains only the APK; + artifacts/146/3 is empty), and the task's device rules are mutually + exclusive"*. +- **final_result_validation** — malformed/schema-rejected model result. Run 8: + `model stage implementer: FINAL RESULT VALIDATION: invalid model stage result: + invalid character 'I' looking for beginning of value at byte 1; unsafe to + repair incomplete values or a non-object response; completed edits and tool + side effects are retained and not undone; a full retry repeats work`. Run 221: + `model stage reviewer: FINAL RESULT VALIDATION: invalid model stage result: + invalid character '}' after array element at byte 2929; isolated format repair + failed: retained OpenCode session ses_eb8c5ab8affeAeLm9sFdG8uimz: context + deadline exceeded; …`. Two sub-shapes were observed: + `unsafe to repair incomplete values or a non-object response` (runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) and `isolated format repair failed: retained OpenCode session ses_…: context - deadline exceeded` (runs 55,81,174,179,183,221 — 6 runs). Run 85/142 add a - third: `unknown result field`. -- **harness_transport_execution_error** — run 73/74/75/77/80/149/150/152/195/202: - `model stage : HARNESS TRANSPORT: harness transport failure: OpenCode - agent reported an execution error`. + deadline exceeded` (runs 55,81,174,179,183,221 — 6 runs). Runs 85/142 add a + third: `unknown result field`; run 170's diagnostic is `invalid character '›' + after object key:value pair at byte 780`. +- **harness_transport_execution_error** — e.g. run 73: `model stage ui_reviewer: + HARNESS TRANSPORT: harness transport failure: OpenCode agent reported an + execution error`. - **harness_transport_timeout** — run 33: `model stage implementer: HARNESS - TRANSPORT: context deadline exceeded \n context deadline exceeded` (also - 36,38,59,101,138). + TRANSPORT: context deadline exceeded\ncontext deadline exceeded`. +- **cancelled_by_operator** — `context canceled`, status `canceled`. - **review_loop_exhausted** — run 31: `Pipelines stopped reviewer after 3 visits - on outcome rejected` (also 70,96,97,127). -- **cancelled_by_operator** — `context canceled` with status `canceled`. -- **interrupted_by_deploy** — run 13/24: `Server stopped; run was not resumed` - with status `interrupted`. + on outcome rejected`. +- **stage_outcome_error** — run 190: `stage implementer returned outcome + "error"`; runs 30/137/139 have a final step `outcome='error'`. +- **interrupted_by_deploy** — run 13/24: `Server stopped; run was not resumed`, + status `interrupted`. - **invalid_agent_messages** — run 2/5: `model stage planner: invalid agent messages`. -- **duplicate_json_field** — run 3: `invalid model stage result: duplicate JSON - field "performance_review"`. - **harness_transport_readback** — run 161: `harness transport failure: final assistant readback had no completed result`. - **agent_quiescence_unconfirmed** — run 43: `agent quiescence not confirmed; @@ -96,52 +108,62 @@ Classifying diagnostics (quoted verbatim from the run records): - **session_transport_command_failed** — run 4: `model stage implementer: OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait command failed: CLI exit code 1`. +- **duplicate_json_field** — run 3: `invalid model stage result: duplicate JSON + field "performance_review"`. `server.log` contains none of these diagnostic strings (bounded search: `FINAL RESULT VALIDATION` 0, `HARNESS TRANSPORT` 0, `invalid agent messages` 0, -`stopped reviewer after` 0); it is a request/response log +`stopped reviewer after` 0, `quiescence` 0); it is a request/response log (`command received`/`response sent`, plus `summarise: harness=opencode …`). Diagnostics live in the run records, not the server log. ## 2. Stage cost -821 steps across the 222 finished runs (3 steps of the 3 live runs have an empty -`finished_at` and were excluded, not fabricated: runs 223 seq1 `implementer`, -222 seq2 `implementer`, 220 seq3 `ui_reviewer`). +The 223 runs produced **828 steps**; **826 have both `started_at` and +`finished_at`** and were used. The 2 without a `finished_at` belong to the 2 +still-running runs and were excluded, not fabricated: run 223 seq2 `reviewer` +and run 222 seq4 `implementer`. By run: 221 runs have a non-empty `finished_at` +and all 223 runs contribute at least one complete step. This is the population +at the 2026-10-17 capture, when run 220 had already finished — an earlier working +note that called the population "222 finished" was stale and is corrected here. Total wall-hours by stage key (sum of `finished_at - started_at`), with the share spent on repeat visits (`control.visit > 1`, i.e. review/retry loops) split out: | stage key | total (h) | single-pass (h) | repeat (h) | steps | |---|---|---|---|---| -| implementer | 41.13 | 33.51 | 7.62 | 268 | -| reviewer | 20.65 | 15.69 | 4.96 | 219 | -| ui_reviewer | 15.09 | 13.84 | 1.25 | 134 | +| implementer | 41.36 | 33.73 | 7.62 | 270 | +| reviewer | 20.72 | 15.76 | 4.96 | 220 | +| ui_reviewer | 15.18 | 13.93 | 1.25 | 135 | | planner | 3.55 | 3.55 | 0.00 | 79 | -| committer | 1.95 | 1.95 | 0.00 | 98 | -| (worker/research/review/probe/report/opencode/pi) | 0.15 | 0.15 | 0.00 | 23 | +| committer | 2.00 | 2.00 | 0.00 | 99 | +| worker/research/review/probe/report/opencode/pi | 0.16 | 0.16 | 0.00 | 23 | -The dominant stage keys are **implementer (41.1 h), reviewer (20.7 h) and -ui_reviewer (15.1 h)** — together 76.9 of 82.53 h. Repeat visits of the same -stage cost **13.83 h (16.8 %)**, almost all in implementer (7.62 h) and reviewer +The dominant stage keys are **implementer (41.36 h), reviewer (20.72 h) and +ui_reviewer (15.18 h)** — together 77.26 of 82.97 h. Repeat visits of the same +stage cost **13.83 h (16.7 %)**, almost all in implementer (7.62 h) and reviewer (4.96 h). -Per (pipeline, stage key) distribution (count / median / p90 / max seconds / -total hours). Only the two high-volume pipelines are shown; the full table was -produced by the aggregation command. +Per (pipeline, stage key) distribution (count / median / p90 / max in seconds / +total hours), for the highest-volume keys across the three highest-volume +pipelines: | pipeline | stage | n | median | p90 | max | total (h) | |---|---|---|---|---|---|---| | Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 | +| Pipelines maintenance - plan implement review | implementer | 90 | 370 | 847 | 3601 | 12.07 | +| Pipelines maintenance - plan implement review | reviewer | 79 | 245 | 503 | 12613 | 10.54 | +| Focused tweak - implement review commit | ui_reviewer | 81 | 293 | 1030 | 2158 | 10.05 | | Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 | -| Focused tweak - implement review commit | ui_reviewer | 80 | 290 | 1030 | 2158 | 9.96 | -| Focused tweak - implement review commit | committer | 62 | 29 | 178 | 644 | 1.08 | -| Pipelines maintenance - plan implement review | implementer | 89 | 375 | 884 | 3601 | 11.98 | -| Pipelines maintenance - plan implement review | reviewer | 78 | 242 | 503 | 12613 | 10.47 | -| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 | -| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 | | Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 | +| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 | | Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 | +| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 | +| Pipelines quick - plan implement review | reviewer | 13 | 324 | 896 | 1800 | 1.69 | +| Focused tweak - implement review commit | committer | 63 | 31 | 180 | 644 | 1.13 | + +Three pipelines are shown (Focused tweak, Pipelines maintenance, Pipelines +quick); the full per-pipeline table was produced by the aggregation command. Notable long tails: one `reviewer` step in Pipelines maintenance ran 12613 s (3.5 h); one Pipelines-maintenance `implementer` step hit 3601 s (≈ the 1 h @@ -150,18 +172,18 @@ stage timeout), while the Focused-tweak `implementer` max is 1801 s and its p90 ## 3. Reliability trend, split at 910a7aa -Boundary: `2026-10-16T21:27:00Z`, the deploy of commit 910a7aa. `pipelines ping` -confirms the running `build.commit` is 910a7aa. `deploy-server.log` corroborates: -`22:27:45 quiescing the lanes …` / `22:27:45 building from main (910a7aa Merge -lane-app2: …)` / `22:27:57 server healthy; deployed 910a7aa` (its clock is local -UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by `created_at`: the last -"before" run is 193 (`2026-10-16T21:21:33Z`), the first "after" is 194 -(`2026-10-16T21:28:00Z`). +Boundary: `2026-10-16T21:27:00Z`, the deploy of commit 910a7aa. `pipelines ping +--json` confirms the running `build.commit` is 910a7aa. `deploy-server.log` +corroborates: `22:27:45 quiescing the lanes …` / `22:27:45 building from main +(910a7aa Merge lane-app2: …)` / `22:27:57 server healthy; deployed 910a7aa` +(its clock is local UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by +`created_at`: the last "before" run is 193 (`2026-10-16T21:21:33Z`), the first +"after" is 194 (`2026-10-16T21:28:00Z`). | side | n | running | failed | failed/(failed+succeeded) | non-success/terminal | |---|---|---|---|---|---| | before | 193 | 0 | 68 | 37.4 % | 40.9 % | -| after | 30 | 3 | 5 | 20.0 % | 25.9 % | +| after | 30 | 2 | 5 | 19.2 % | 25.0 % | Failure-class mix, before → after (counts): @@ -182,164 +204,195 @@ Failure-class mix, before → after (counts): | duplicate_json_field | 1 | 0 | Read literally the change correlates with fewer malformed-result failures: -`final_result_validation + duplicate_json_field + invalid_agent_messages` -falls from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is -**confounded and under-powered**: the after side is only 30 runs (3 still -running), the pipeline mix differs (after is mostly Focused tweak 15, Pipelines -maintenance 7, Merge-resolution 3 — the one-off template/docker/research -pipelines appear only before), and several before-side classes -(transport timeout, review-loop, deploy interruption) simply never recurred in -the short after window rather than being fixed by 910a7aa. The data is -consistent with the format/repair change helping; it does not establish -causation. A larger after sample is needed. +`final_result_validation + duplicate_json_field + invalid_agent_messages` falls +from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is **confounded +and under-powered**: the after side is only 30 runs (2 still running), the +pipeline mix differs (after is mostly Focused tweak 15, Pipelines maintenance 7, +Merge-resolution 3 — several one-off template/docker/research pipelines appear +only before), and several before-side classes (transport timeout, review-loop, +deploy interruption) simply never recurred in the short after window rather than +being fixed by 910a7aa. The data is consistent with the format/repair change +helping; it does not establish causation. A larger after sample is needed. ## 4. Throughput limits (measured) **Busy-workspace waits.** The lane logs record `workspace busy; waiting to -launch ` at a fixed 30 s cadence. 897 such lines were counted across the -lane logs. Grouping them into wait episodes (same lane+file, gaps ≤ 120 s) -gives 28 episodes, 23 of which end with a `launched ` line and 5 still -open at log end. For the 23 resolved episodes the wait is -`launch_time - first_busy`: +launch ` at a fixed 30 s cadence: **897 such lines**. Grouping them into +episodes (same lane+file, ending at the `launched ` line) gives **27 +episodes, 23 resolved and 4 still open at log end**. For the 23 resolved +episodes the wait is `launch_time - first_busy`: - n=23, median **992 s (16.5 min)**, p90 **2293 s (38.2 min)**, max - **3666 s (61.1 min)**, total **7.34 h**. + **9070 s (2.5 h)**, total **9.07 h**. This independent reimplementation of the + rule includes the laneGo `malformed-result-brief.txt` episode (2170 s); an + earlier working note that dropped it reported n=22 / 8.47 h and is corrected + here. -Longest episodes: laneGo `s02-tui-omni-search.txt` waited 15:47:22→16:48:28 -(3666 s, 122 busy lines) before `16:48:28 launched s02-tui-omni-search.txt as -run 149`; laneA `s02-tui-omni-search.txt` waited 2825 s; laneGo -`search-server-tui-brief.txt` waited 2293 s. +Longest episodes: laneA `s02-tui-omni-search.txt` waited 13:12:33 → 15:43:43 +(9070 s) before `15:43:43 launched s02-tui-omni-search.txt as run 137`; laneGo +`s02-tui-omni-search.txt` waited 15:47:22 → 16:48:28 (3666 s). + +**Server-side rejection of a second run in a busy workspace.** `server.log` +records **910** `error="run is already active in workspace ; wait for +it to finish or start this run in a different workspace …"` responses, from +**29 distinct blocking runs**, concentrated on 2026/10/16 (905) vs 2026/10/15 +(5). By workspace: `lane-go` 387, `/Users/alan/repos/pipelines` 307, `lane-app2` +166, `lane-c` 49, `ws-c` 1. So the dominant limit is one-run-per-workspace, and +the lanes retry it hundreds of times. **Server-restart holds.** `holding: another lane is restarting the server` -appears **769** times; `building server and cli from main` 13 times; `server -redeployed and healthy` 10 times. These are lanes idling while a deploy or -rebuild owns the server. +appears **800** times; grouping holds per lane while consecutive hold lines are +≤60 s apart (the 30 s poll cadence plus jitter; a wider gap starts a new +episode) gives 29 episodes totalling **6.42 h**, median 480 s, max 3572 s. (A +strict consecutive-line regroup, which splits an episode on any interleaved +line, gives 32 episodes; the 29 figure is the gap-based grouping.) `building +server and cli from main` 13 times; `server redeployed and healthy` 10 times. -**Merge/main-lane waits.** `merge deferred: main lane busy` appears **11** -times (e.g. `/tmp/laneB.log:243 04:14:33 [laneB] merge deferred: main lane -busy`). `merge requested for …` appears 23 times; `MERGE CONFLICT` recorded 15 -times; `SYNC CONFLICT: lane could not take main's latest; continuing on the -older base` 44 times. +**Merge/main-lane waits.** `merge deferred: main lane busy` appears **11** times; +`merge requested for …` 24 times; `MERGE CONFLICT` 15 times; `SYNC CONFLICT: lane +could not take main's latest; continuing on the older base` 45 times. **Merge/resolver activity.** `merger.log` shows the queue draining through pipeline resolvers. 12 resolver runs have measurable durations, total -**5153 s (1.43 h)**: `run 176 20 s`, `177 20 s` (these "completed the merge" -immediately), versus `run 180 661 s`, `182 602 s`, `183 1062 s` (handed to a -pipeline). Three did not land: `179 did NOT land lane-app2`, `183 did NOT land -lane-go`, `221 did NOT land lane-c`. The `merge-queue` marker in the worktree -currently holds `lane-go`, `lane-app2`. +**5153 s (1.43 h)**, median 421 s, max 1062 s: `run 176 20 s` and `177 20 s` +("completed the merge" immediately) versus `run 180 661 s`, `182 602 s`, +`183 1062 s` (handed to a pipeline). Four events record `could not launch a +resolver; aborting the merge for `; three resolvers `did NOT land` their +lane (`179 lane-app2`, `183 lane-go`, `221 lane-c`). The `merge-queue` marker in +the worktree currently holds `lane-go` and `lane-app2`. + +**Deploy waits.** `deploy-server.log` shows one deploy waiting for quiescence: +`13:39:07 quiescing the lanes so no new run starts` with `waiting: N run(s) +active` every ~20 s from 3 down to 1, until `14:18:18 building from main +(dce624b …)` — a **2351 s (39.2 min)** wait. The 910a7aa deploy at `22:27:45` +waited 12 s (0 active runs). **Device/emulator churn.** `FAILED once: - recovering the emulator and -retrying` appears 50 times and `FAILED twice: - recording and -continuing` 35 times (e.g. `02:53:47 [laneA] FAILED once: -q03-pane-opaque-A.txt - recovering the emulator and retrying`). +retrying` appears 50 times and `FAILED twice: - recording and continuing` +35 times (e.g. `02:53:47 [laneA] FAILED once: q03-pane-opaque-A.txt - recovering +the emulator and retrying`). No `device lock` event string was found in the lane +logs or `server.log`. -**Real concurrency.** Overlapping step intervals show a maximum of **6 -simultaneous executing steps**, at `2026-10-16T14:47:22Z`, in runs -[134, 136, 137, 138, 139, 140]. 156 of the 222 finished runs had at least one -step overlapping another run. Using run wall-spans (`created_at`..`finished_at`) -the maximum overlap is also **6 concurrent runs** at the same instant, with 154 -runs overlapping. So despite many queued runs, the observed executing -concurrency never exceeded 6 — consistent with the lane count and the 897 -workspace-busy lines, and not with a memory/harness-capacity limit. +**Real concurrency.** Overlapping step intervals show a maximum of **7 +simultaneous executing steps**, at `2026-10-16T14:47:35Z`, spanning the 6 runs +[134, 136, 137, 138, 139, 140] (one run had two steps overlapping at that +instant). Step intervals are counted as **closed** (`started_at ≤ t ≤ +finished_at`), so run 134's `reviewer` (ending 14:47:35Z) and its `implementer` +(starting 14:47:35Z) both count at the touching boundary; with half-open +intervals (`started_at ≤ t < finished_at`) the maximum is **6 steps at +14:47:22Z**, the same six runs. Using run wall-spans (`created_at`..`finished_at`) the maximum is **6 +concurrent runs** at `2026-10-16T14:47:22Z`, the same six runs. 157 runs had at +least one step interval overlapping another run. So despite many queued runs, +the observed executing concurrency never exceeded 7 steps / 6 runs — consistent +with the lane count and the 897 workspace-busy lines, and not with a +memory/harness-capacity limit. ## 5. Ranked improvements Ranked by measured impact. Items marked **hypothesis** have no direct measurement behind them. -1. **Reduce busy-workspace waiting (measured 7.34 h + 769 restart holds).** - Problem: 23 resolved waits, median 992 s, total 7.34 h, plus 769 - server-restart holds. Smallest change: lower the lane's fixed 30 s - busy-poll to a few seconds and/or keep a lane resident while a workspace is - held, rather than relaunching the poll each cycle. Expected effect: cut the - median multi-minute launch latency. Risk: more frequent polling and - contention. Locus: orchestration. Evidence: lane log cadence and the 897 - `workspace busy` lines. +1. **Remove the one-run-per-workspace contention (measured 9.07 h waiting + 910 + server rejections + 6.42 h restart holds).** Problem: 23 resolved bus waits + total 9.07 h (median 992 s, max 9070 s); the server rejected a second run in a + busy workspace 910 times across 29 blocking runs; restart holds cost 6.42 h. + Smallest change: give each lane its own worktree/workspace so a lane is never + blocked by a run already active in it (or lower the fixed 30 s busy-poll). + Expected effect: cut the multi-minute launch latency and the wasted poll + lines. Risk: more concurrent worktrees consume more disk and parallel model + capacity. Locus: orchestration. Evidence: the busy/rejection counts and the + per-workspace breakdown above. 2. **Finish or budget the malformed-result repair path.** - Problem: `final_result_validation` is 19 of 86 non-success runs; in 6 of them - (55,81,174,179,183,221) the repair itself failed with - `isolated format repair failed: retained OpenCode session ses_…: context - deadline exceeded`, and in 11 (8,12,29,94,118,120,126,148,157,164,170) the - result was judged `unsafe to repair`. Smallest change: give the isolated - format-repair session its own longer deadline (or a streaming repair), so a - repairable-but-slow result is recovered instead of failing the run. Expected - effect: recover part of the 6 timeout-repair runs; the 11 "unsafe to repair" - remain a separate problem. Risk: longer tail latency on already-slow runs. - Locus: server code (`internal/stage`). Evidence: the six quoted diagnostics - and the before/after drop (10.9 %→3.3 %) for this cluster. + Problem: `final_result_validation` is 19 of 86 non-success runs; in several + (e.g. 55,81,174,179,183,221) the repair itself failed with `isolated format + repair failed: retained OpenCode session ses_…: context deadline exceeded`, + while others were judged `unsafe to repair`. Smallest change: give the + isolated format-repair session its own longer deadline (or a streaming + repair), so a repairable-but-slow result is recovered instead of failing the + run. Expected effect: recover part of the timeout-repair runs; the "unsafe to + repair" set remains a separate problem. Risk: longer tail latency on + already-slow runs. Locus: server code (`internal/stage`). Evidence: the quoted + diagnostics and the before/after drop (10.9 %→3.3 %) for this cluster. 3. **Cut repeat-visit cost in the review loop.** - Problem: repeat visits of the same stage cost 13.83 h of 82.53 h (16.8 %), + Problem: repeat visits of the same stage cost 13.83 h of 82.97 h (16.7 %), concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs (31,70,96,97,127) failed outright with `Pipelines stopped reviewer after 3 - visits on outcome rejected`. Smallest change: identify which rejections were - advisory-only (no must-fix finding) and stop looping on those, or reduce the - reviewer max-visits for advisory findings. Expected effect: remove ~14 h of - repeat work and the 5 exhausted-loop failures, if rejections are often - non-blocking. Risk: a real defect could pass. Locus: pipeline definition - (`reviewer` control). Evidence: repeat/single-pass split in §2 and the - `review_loop_exhausted` ids. + visits on outcome rejected`. Smallest change: stop looping on advisory-only + rejections (no must-fix finding), or reduce the reviewer max-visits for + advisory findings. Expected effect: remove up to ~14 h of repeat work and the + 5 exhausted-loop failures, if rejections are often non-blocking. Risk: a real + defect could pass. Locus: pipeline definition (`reviewer` control). Evidence: + the repeat/single-pass split in §2 and the `review_loop_exhausted` ids. -4. **Make the ui_reviewer receive (or not require) captured artifacts.** - Problem: `ui_reviewer` blocked is a large part of the 23-run - `stage_outcome_blocked` class (run 146: `artifacts/146/3 is empty`), and - `ui_reviewer` p90 is 1030 s. Smallest change: whatever captures screenshots - for a UI run must hand them to the `ui_reviewer` stage, and the stage should - not block when no device is attached. Expected effect: fewer blocked UI runs. - Risk: silently skipping visual verification. Locus: pipeline definition + - orchestration (artifact handoff). Evidence: run 146 preview and the blocked - ids; **partly hypothesis** — the artifact-empty condition is quoted for one - run only, not measured across all 23. +4. **Make the `implementer` blocked outcome actionable (measured 18/23 of the + blocked class).** Problem: `stage_outcome_blocked` is the single largest class + (23/86, 26.7 %), and it is dominated by **implementer (18)** — not + `ui_reviewer`, which is only 4 (and `review` 1). Smallest change: have the + implementer stage distinguish "environment/device unavailable" (retryable) + from "acceptance genuinely unmeetable" (terminal), so the retryable subset is + retried instead of failing the run. Expected effect: a portion of the 18 + implementer-blocked runs recovered. Risk: retry storms. Locus: pipeline + definition / server code. Evidence: the 23 ids and the measured blocked-stage + split; the retryable/terminal split is **hypothesis** (not measured). -5. **Triage the `implementer`/`ui_reviewer` "blocked" outcomes.** - Problem: `stage_outcome_blocked` is the single largest class (23/86, 26.7 %). - Smallest change: have the stage distinguish "environment/device unavailable" - (retryable) from "acceptance genuinely unmeetable" (terminal), so the - retryable subset is retried instead of failing the run. Expected effect: a - portion of 23 runs recovered. Risk: retry storms. Locus: pipeline definition - / server code. Evidence: the 23 ids and their `result_preview`; the - retryable/terminal split is **hypothesis** (not measured). +5. **Hand the `ui_reviewer` captured artifacts (measured 4/23 blocked).** + Problem: `ui_reviewer` blocked in 4 runs (146,167,207,210) and its p90 is + 1030 s in Focused tweak; run 146's preview reports `artifacts/146/3 is + empty`. Smallest change: whatever captures screenshots for a UI run must hand + them to the `ui_reviewer` stage, and the stage should not block when no device + is attached. Expected effect: fewer blocked UI runs. Risk: silently skipping + visual verification. Locus: pipeline definition + orchestration (artifact + handoff). Evidence: run 146 preview and the 4 blocked ids; **partly + hypothesis** — the empty-artifact cause is quoted for one run only, not + measured across all 4. 6. **Harden the harness transport path.** - Problem: 17 non-success runs are transport failures — 10 - `execution error`, 6 `context deadline exceeded`, 1 `final assistant readback - had no completed result`. Smallest change: add bounded retry/backoff around - the OpenCode `…/wait` call for the generic "agent reported an execution - error" case. Expected effect: recover the 10 execution-error runs if they are - transient. Risk: retrying a genuinely broken session. Locus: server code - (harness adapter). Evidence: the quoted diagnostics and ids. Whether the - errors are transient is **hypothesis**. + Problem: 18 non-success runs are transport failures — 10 `execution error`, 6 + `context deadline exceeded`, 1 `final assistant readback had no completed + result`, 1 `OpenCode post … command failed`. Smallest change: add bounded + retry/backoff around the OpenCode `…/wait` call for the generic "agent + reported an execution error" case. Expected effect: recover the 10 + execution-error runs if they are transient. Risk: retrying a genuinely broken + session. Locus: server code (harness adapter). Evidence: the quoted + diagnostics and ids. Whether the errors are transient is **hypothesis**. 7. **Stabilise merge resolution.** Problem: merge-resolution pipelines contributed blocked implementers (176, 177, 178), `final_result_validation` (179, 183, 221), and resolver runs cost 1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver's - `implementer` blocks on a conflict workspace (e.g. missing conflict markers - or write-path scope). Locus: merge-resolution pipeline definition. Evidence: + `implementer` blocks on a conflict workspace (e.g. missing conflict markers or + write-path scope). Locus: merge-resolution pipeline definition. Evidence: `merger.log` lines and the run diagnostics; root cause **hypothesis**. ## 6. What could not be verified - **server.log content.** It is a request/response log and contains none of the - stage diagnostics (0 matches for all diagnostic phrases). It could not be used - to re-derive failures; the run records were used instead. To settle whether - transport errors correlate with request latency, the server would need to log - per-request duration (not currently present). -- **3 non-terminal runs excluded.** Runs 220, 222, 223 were `running` when the - history and steps were captured; their incomplete steps - (223 seq1, 222 seq2, 220 seq3) have empty `finished_at` and were excluded from - all distributions. Settle by re-reading `pipelines inspect` after they finish. -- **5 unresolved busy-wait episodes.** Five `workspace busy` sequences were - still open at log end (no matching `launched` line), so their final wait is - unknown. Settle by re-parsing the lane logs once the lanes go idle. -- **Blocked-class cause distribution.** Only one blocked run (146) has a quoted - cause; the other 22 were classified by `outcome=blocked` plus stage name. A + stage diagnostics (0 matches for every diagnostic phrase above). It could not + be used to re-derive failures; the run records were used instead. To settle + whether transport errors correlate with request latency, the server would need + to log per-request duration (not currently present). +- **2 non-terminal runs excluded.** Runs 222 and 223 were `running` when the + history and steps were captured; their incomplete steps (223 seq2 `reviewer`, + 222 seq4 `implementer`) have an empty `finished_at` and were excluded from all + distributions. Settle by re-reading `pipelines inspect` after they finish. + (An earlier capture also excluded run 220 seq3 `ui_reviewer` when it was still + running; it had finished by the later capture and is included above.) +- **4 unresolved busy-wait episodes.** Four `workspace busy` sequences were still + open at log end (no matching `launched` line), so their final wait is unknown. + (An earlier working note reported 5 unresolved episodes; the re-counted + population gives 4.) Settle by re-parsing the lane logs once the lanes go idle. +- **Blocked-class cause distribution.** Only run 146 has a quoted cause; the + other blocked runs were classified by `outcome=blocked` plus stage name. A per-run `result_preview` pass would quantify the retryable vs terminal split. -- **After-side sample size.** The 910a7aa comparison rests on 30 runs (3 +- **After-side sample size.** The 910a7aa comparison rests on 30 runs (2 still running). A larger after-window sample is required before attributing the failure-rate drop to the change rather than to pipeline mix. +- **Device-lock waits.** No `device lock` event string exists in the lane logs or + `server.log`; emulator trouble appears only as `recovering the emulator` + retries. A device-lock measurement would need the orchestration to log an + explicit acquire/wait/release. - **No system-changing measurements were attempted.** No run was started, retried, cancelled or reconfigured; the server, ntfy service, lanes and devices were not touched, and the phone/emulator was not used. Measurements that would @@ -348,24 +401,30 @@ measurement behind them. ## Verification and limits -Commands and observed results: +Commands and observed results (all run in this session): -- `pipelines ping --json` → build/commit `910a7aa…`. -- `pipelines runs --json` (1 call) → 223 runs; the run-status counts reconcile - with `pipelines stats --json` (`total 223, succeeded 134, failed 73, canceled - 11, interrupted 2, active 3`). +- `pipelines ping --json` → build/commit `910a7aa…`, `dirty false`. +- `pipelines runs --json` (2 calls, at different capture times) → 223 runs; the + two run-status counts reconcile with `pipelines stats --json` + (`total 223, succeeded 134/135, failed 73, canceled 11, interrupted 2, active + 3/2`). - `pipelines inspect --json` → 223 calls (one per run id), used for all step intervals and the taxonomy fallback. +- `server.log` bounded searches: 910 `is already active in workspace` + rejections; 0 diagnostic strings. +- Lane/orchestration logs (streaming `grep`/`awk`): 897 `workspace busy` lines, + 800 restart holds, 11 `merge deferred`, 12 resolver durations, 50/35 emulator + retries. - Go verification, run with the repository's normal invocation from the module - root on go1.27.1/darwin-arm64: + root: - `go build ./...` → exit 0, no output. - - `go test ./...` → all packages `ok` (cached), exit 0; the three `cmd/*` - packages and `internal/wstest` have no test files. - No Go source was changed; these commands confirm the worktree was left - buildable. + - `go test ./...` → exit 0; 20 packages `ok`, 3 `cmd/*` and `internal/wstest` + have no test files. No Go source was changed; these confirm the worktree was + left buildable. Limits: the after-side sample is small; several classes did not recur after the boundary for reasons the data cannot separate from the format change; the -blocked-class causes and the transience of transport errors are hypotheses, not -measurements. Raw captures are under -`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` (outside the repo). +blocked-class cause distribution and the transience of transport errors are +hypotheses, not measurements. Raw captures are under +`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and +`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).