From 3997e1bbc14e343f3605a24a8329518611d75926 Mon Sep 17 00:00:00 2001 From: Alan Silva Date: Sat, 17 Oct 2026 01:26:35 +0100 Subject: [PATCH] docs: pipeline performance and reliability review --- docs/pipeline-performance-review.md | 371 ++++++++++++++++++++++++++++ 1 file changed, 371 insertions(+) create mode 100644 docs/pipeline-performance-review.md diff --git a/docs/pipeline-performance-review.md b/docs/pipeline-performance-review.md new file mode 100644 index 00000000..3ee44733 --- /dev/null +++ b/docs/pipeline-performance-review.md @@ -0,0 +1,371 @@ +# Pipeline performance and reliability review + +Read-only analysis of the live Pipelines run history, step records and +orchestration logs, on 2026-10-17. The only change is this document. + +## Sources and method + +All numbers come from commands run in this session against the running server +(`pipelines ping` reports `build.version v0.0.0-20261016212730-910a7aa798af`, +`commit 910a7aa798af3eecff72fcde28b9eafde470286c`, `dirty false`). + +- `pipelines runs --json` was captured once into + `/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json` + (223 entries, 201939 bytes) and reused for the taxonomy and trend. +- `pipelines stats --json` was captured once + (`/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json`): `total + 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3`. + `134+73+11+2+3 = 223`, so the run inventory reconciles with the totals. +- `pipelines inspect --json` was called exactly once per run id (223 calls, + saved under `artifacts/222/2/inspect/`); step intervals were aggregated in one + pass from those files. +- Orchestration logs: `/tmp/lane{A,B,C,D,E,Go,T2}.log`, `/tmp/merger.log`, + `/tmp/deploy-server.log`, and + `.../T/opencode/lane-{hg,hs}.log`, plus + `~/.local/share/pipelines/logs/server.log` (16.2 MB, 143080 lines). + +Quantile convention: **nearest-rank**, i.e. the p-quantile is element +`ceil(p*n)` (1-based) of the sorted list; the median is the p=0.5 element. +Durations are `finished_at - started_at` parsed with RFC3339/RFC3339Nano rules +(`Z` normalised), as in `internal/pipeline/search.go`. + +## 1. Failure taxonomy over the whole history + +86 runs have a terminal non-success status: `failed 73`, `canceled 11`, +`interrupted 2`. Every one is classified below from the run `diagnostic`, or — +where that is empty — from the failing step's outcome in its inspect record. +The class counts sum to 86; there is no unknown bucket. + +| class | count | run ids | +|---|---|---| +| `stage_outcome_blocked` | 23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 | +| `final_result_validation` | 19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 | +| `cancelled_by_operator` | 11 | 28,39,57,63,78,108,113,114,191,211,219 | +| `harness_transport_execution_error` | 10 | 73,74,75,77,80,149,150,152,195,202 | +| `harness_transport_timeout` | 6 | 33,36,38,59,101,138 | +| `review_loop_exhausted` | 5 | 31,70,96,97,127 | +| `stage_outcome_error` | 4 | 30,137,139,190 | +| `interrupted_by_deploy` | 2 | 13,24 | +| `invalid_agent_messages` | 2 | 2,5 | +| `harness_transport_readback` | 1 | 161 | +| `agent_quiescence_unconfirmed` | 1 | 43 | +| `session_transport_command_failed` | 1 | 4 | +| `duplicate_json_field` | 1 | 3 | + +Classifying diagnostics (quoted verbatim from the run records): + +- **stage_outcome_blocked** — a stage returns `blocked`, e.g. run 32/45 + (`implementer` final step `outcome='blocked'`, run `diagnostic` empty) and + run 105: `stage review returned outcome "blocked"`; run 115/141/145/151/154/ + 156/158/159/160/167/173/175/176/177/178/188/189/207/210: + `stage implementer returned outcome "blocked"` or + `stage ui_reviewer returned outcome "blocked"`. The blocked stages are + dominated by `implementer` and `ui_reviewer`, and their `result_preview` + shows the stage stopping deliberately — e.g. run 146: *"No screenshots/frames + exist for this run (artifacts/146/1 contains only the APK; artifacts/146/3 is + empty), and the task's device rules are mutually ex[c]lusive"*. +- **final_result_validation** — malformed/schema-rejected model result. Example, + run 8: `model stage implementer: FINAL RESULT VALIDATION: invalid model stage + result: invalid character 'I' looking for beginning of value at byte 1; unsafe + to repair incomplete values or a non-object response; completed edits and tool + side effects are retained and not undone; a full retry repeats work`. Two + sub-shapes: `unsafe to repair incomplete values or a non-object response` + (runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) and + `isolated format repair failed: retained OpenCode session ses_…: context + deadline exceeded` (runs 55,81,174,179,183,221 — 6 runs). Run 85/142 add a + third: `unknown result field`. +- **harness_transport_execution_error** — run 73/74/75/77/80/149/150/152/195/202: + `model stage : HARNESS TRANSPORT: harness transport failure: OpenCode + agent reported an execution error`. +- **harness_transport_timeout** — run 33: `model stage implementer: HARNESS + TRANSPORT: context deadline exceeded \n context deadline exceeded` (also + 36,38,59,101,138). +- **review_loop_exhausted** — run 31: `Pipelines stopped reviewer after 3 visits + on outcome rejected` (also 70,96,97,127). +- **cancelled_by_operator** — `context canceled` with status `canceled`. +- **interrupted_by_deploy** — run 13/24: `Server stopped; run was not resumed` + with status `interrupted`. +- **invalid_agent_messages** — run 2/5: `model stage planner: invalid agent + messages`. +- **duplicate_json_field** — run 3: `invalid model stage result: duplicate JSON + field "performance_review"`. +- **harness_transport_readback** — run 161: `harness transport failure: final + assistant readback had no completed result`. +- **agent_quiescence_unconfirmed** — run 43: `agent quiescence not confirmed; + inspect retained session before further work`. +- **session_transport_command_failed** — run 4: `model stage implementer: + OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait + command failed: CLI exit code 1`. + +`server.log` contains none of these diagnostic strings (bounded search: +`FINAL RESULT VALIDATION` 0, `HARNESS TRANSPORT` 0, `invalid agent messages` 0, +`stopped reviewer after` 0); it is a request/response log +(`command received`/`response sent`, plus `summarise: harness=opencode …`). +Diagnostics live in the run records, not the server log. + +## 2. Stage cost + +821 steps across the 222 finished runs (3 steps of the 3 live runs have an empty +`finished_at` and were excluded, not fabricated: runs 223 seq1 `implementer`, +222 seq2 `implementer`, 220 seq3 `ui_reviewer`). + +Total wall-hours by stage key (sum of `finished_at - started_at`), with the share +spent on repeat visits (`control.visit > 1`, i.e. review/retry loops) split out: + +| stage key | total (h) | single-pass (h) | repeat (h) | steps | +|---|---|---|---|---| +| implementer | 41.13 | 33.51 | 7.62 | 268 | +| reviewer | 20.65 | 15.69 | 4.96 | 219 | +| ui_reviewer | 15.09 | 13.84 | 1.25 | 134 | +| planner | 3.55 | 3.55 | 0.00 | 79 | +| committer | 1.95 | 1.95 | 0.00 | 98 | +| (worker/research/review/probe/report/opencode/pi) | 0.15 | 0.15 | 0.00 | 23 | + +The dominant stage keys are **implementer (41.1 h), reviewer (20.7 h) and +ui_reviewer (15.1 h)** — together 76.9 of 82.53 h. Repeat visits of the same +stage cost **13.83 h (16.8 %)**, almost all in implementer (7.62 h) and reviewer +(4.96 h). + +Per (pipeline, stage key) distribution (count / median / p90 / max seconds / +total hours). Only the two high-volume pipelines are shown; the full table was +produced by the aggregation command. + +| pipeline | stage | n | median | p90 | max | total (h) | +|---|---|---|---|---|---|---| +| Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 | +| Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 | +| Focused tweak - implement review commit | ui_reviewer | 80 | 290 | 1030 | 2158 | 9.96 | +| Focused tweak - implement review commit | committer | 62 | 29 | 178 | 644 | 1.08 | +| Pipelines maintenance - plan implement review | implementer | 89 | 375 | 884 | 3601 | 11.98 | +| Pipelines maintenance - plan implement review | reviewer | 78 | 242 | 503 | 12613 | 10.47 | +| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 | +| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 | +| Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 | +| Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 | + +Notable long tails: one `reviewer` step in Pipelines maintenance ran 12613 s +(3.5 h); one Pipelines-maintenance `implementer` step hit 3601 s (≈ the 1 h +stage timeout), while the Focused-tweak `implementer` max is 1801 s and its p90 +1414 s. The `ui_reviewer` p90 is high (1030 s in Focused tweak). + +## 3. Reliability trend, split at 910a7aa + +Boundary: `2026-10-16T21:27:00Z`, the deploy of commit 910a7aa. `pipelines ping` +confirms the running `build.commit` is 910a7aa. `deploy-server.log` corroborates: +`22:27:45 quiescing the lanes …` / `22:27:45 building from main (910a7aa Merge +lane-app2: …)` / `22:27:57 server healthy; deployed 910a7aa` (its clock is local +UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by `created_at`: the last +"before" run is 193 (`2026-10-16T21:21:33Z`), the first "after" is 194 +(`2026-10-16T21:28:00Z`). + +| side | n | running | failed | failed/(failed+succeeded) | non-success/terminal | +|---|---|---|---|---|---| +| before | 193 | 0 | 68 | 37.4 % | 40.9 % | +| after | 30 | 3 | 5 | 20.0 % | 25.9 % | + +Failure-class mix, before → after (counts): + +| class | before | after | +|---|---|---| +| stage_outcome_blocked | 21 | 2 | +| final_result_validation | 18 | 1 | +| harness_transport_execution_error | 8 | 2 | +| harness_transport_timeout | 6 | 0 | +| cancelled_by_operator | 9 | 2 | +| review_loop_exhausted | 5 | 0 | +| stage_outcome_error | 4 | 0 | +| interrupted_by_deploy | 2 | 0 | +| invalid_agent_messages | 2 | 0 | +| harness_transport_readback | 1 | 0 | +| agent_quiescence_unconfirmed | 1 | 0 | +| session_transport_command_failed | 1 | 0 | +| duplicate_json_field | 1 | 0 | + +Read literally the change correlates with fewer malformed-result failures: +`final_result_validation + duplicate_json_field + invalid_agent_messages` +falls from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is +**confounded and under-powered**: the after side is only 30 runs (3 still +running), the pipeline mix differs (after is mostly Focused tweak 15, Pipelines +maintenance 7, Merge-resolution 3 — the one-off template/docker/research +pipelines appear only before), and several before-side classes +(transport timeout, review-loop, deploy interruption) simply never recurred in +the short after window rather than being fixed by 910a7aa. The data is +consistent with the format/repair change helping; it does not establish +causation. A larger after sample is needed. + +## 4. Throughput limits (measured) + +**Busy-workspace waits.** The lane logs record `workspace busy; waiting to +launch ` at a fixed 30 s cadence. 897 such lines were counted across the +lane logs. Grouping them into wait episodes (same lane+file, gaps ≤ 120 s) +gives 28 episodes, 23 of which end with a `launched ` line and 5 still +open at log end. For the 23 resolved episodes the wait is +`launch_time - first_busy`: + +- n=23, median **992 s (16.5 min)**, p90 **2293 s (38.2 min)**, max + **3666 s (61.1 min)**, total **7.34 h**. + +Longest episodes: laneGo `s02-tui-omni-search.txt` waited 15:47:22→16:48:28 +(3666 s, 122 busy lines) before `16:48:28 launched s02-tui-omni-search.txt as +run 149`; laneA `s02-tui-omni-search.txt` waited 2825 s; laneGo +`search-server-tui-brief.txt` waited 2293 s. + +**Server-restart holds.** `holding: another lane is restarting the server` +appears **769** times; `building server and cli from main` 13 times; `server +redeployed and healthy` 10 times. These are lanes idling while a deploy or +rebuild owns the server. + +**Merge/main-lane waits.** `merge deferred: main lane busy` appears **11** +times (e.g. `/tmp/laneB.log:243 04:14:33 [laneB] merge deferred: main lane +busy`). `merge requested for …` appears 23 times; `MERGE CONFLICT` recorded 15 +times; `SYNC CONFLICT: lane could not take main's latest; continuing on the +older base` 44 times. + +**Merge/resolver activity.** `merger.log` shows the queue draining through +pipeline resolvers. 12 resolver runs have measurable durations, total +**5153 s (1.43 h)**: `run 176 20 s`, `177 20 s` (these "completed the merge" +immediately), versus `run 180 661 s`, `182 602 s`, `183 1062 s` (handed to a +pipeline). Three did not land: `179 did NOT land lane-app2`, `183 did NOT land +lane-go`, `221 did NOT land lane-c`. The `merge-queue` marker in the worktree +currently holds `lane-go`, `lane-app2`. + +**Device/emulator churn.** `FAILED once: - recovering the emulator and +retrying` appears 50 times and `FAILED twice: - recording and +continuing` 35 times (e.g. `02:53:47 [laneA] FAILED once: +q03-pane-opaque-A.txt - recovering the emulator and retrying`). + +**Real concurrency.** Overlapping step intervals show a maximum of **6 +simultaneous executing steps**, at `2026-10-16T14:47:22Z`, in runs +[134, 136, 137, 138, 139, 140]. 156 of the 222 finished runs had at least one +step overlapping another run. Using run wall-spans (`created_at`..`finished_at`) +the maximum overlap is also **6 concurrent runs** at the same instant, with 154 +runs overlapping. So despite many queued runs, the observed executing +concurrency never exceeded 6 — consistent with the lane count and the 897 +workspace-busy lines, and not with a memory/harness-capacity limit. + +## 5. Ranked improvements + +Ranked by measured impact. Items marked **hypothesis** have no direct +measurement behind them. + +1. **Reduce busy-workspace waiting (measured 7.34 h + 769 restart holds).** + Problem: 23 resolved waits, median 992 s, total 7.34 h, plus 769 + server-restart holds. Smallest change: lower the lane's fixed 30 s + busy-poll to a few seconds and/or keep a lane resident while a workspace is + held, rather than relaunching the poll each cycle. Expected effect: cut the + median multi-minute launch latency. Risk: more frequent polling and + contention. Locus: orchestration. Evidence: lane log cadence and the 897 + `workspace busy` lines. + +2. **Finish or budget the malformed-result repair path.** + Problem: `final_result_validation` is 19 of 86 non-success runs; in 6 of them + (55,81,174,179,183,221) the repair itself failed with + `isolated format repair failed: retained OpenCode session ses_…: context + deadline exceeded`, and in 11 (8,12,29,94,118,120,126,148,157,164,170) the + result was judged `unsafe to repair`. Smallest change: give the isolated + format-repair session its own longer deadline (or a streaming repair), so a + repairable-but-slow result is recovered instead of failing the run. Expected + effect: recover part of the 6 timeout-repair runs; the 11 "unsafe to repair" + remain a separate problem. Risk: longer tail latency on already-slow runs. + Locus: server code (`internal/stage`). Evidence: the six quoted diagnostics + and the before/after drop (10.9 %→3.3 %) for this cluster. + +3. **Cut repeat-visit cost in the review loop.** + Problem: repeat visits of the same stage cost 13.83 h of 82.53 h (16.8 %), + concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs + (31,70,96,97,127) failed outright with `Pipelines stopped reviewer after 3 + visits on outcome rejected`. Smallest change: identify which rejections were + advisory-only (no must-fix finding) and stop looping on those, or reduce the + reviewer max-visits for advisory findings. Expected effect: remove ~14 h of + repeat work and the 5 exhausted-loop failures, if rejections are often + non-blocking. Risk: a real defect could pass. Locus: pipeline definition + (`reviewer` control). Evidence: repeat/single-pass split in §2 and the + `review_loop_exhausted` ids. + +4. **Make the ui_reviewer receive (or not require) captured artifacts.** + Problem: `ui_reviewer` blocked is a large part of the 23-run + `stage_outcome_blocked` class (run 146: `artifacts/146/3 is empty`), and + `ui_reviewer` p90 is 1030 s. Smallest change: whatever captures screenshots + for a UI run must hand them to the `ui_reviewer` stage, and the stage should + not block when no device is attached. Expected effect: fewer blocked UI runs. + Risk: silently skipping visual verification. Locus: pipeline definition + + orchestration (artifact handoff). Evidence: run 146 preview and the blocked + ids; **partly hypothesis** — the artifact-empty condition is quoted for one + run only, not measured across all 23. + +5. **Triage the `implementer`/`ui_reviewer` "blocked" outcomes.** + Problem: `stage_outcome_blocked` is the single largest class (23/86, 26.7 %). + Smallest change: have the stage distinguish "environment/device unavailable" + (retryable) from "acceptance genuinely unmeetable" (terminal), so the + retryable subset is retried instead of failing the run. Expected effect: a + portion of 23 runs recovered. Risk: retry storms. Locus: pipeline definition + / server code. Evidence: the 23 ids and their `result_preview`; the + retryable/terminal split is **hypothesis** (not measured). + +6. **Harden the harness transport path.** + Problem: 17 non-success runs are transport failures — 10 + `execution error`, 6 `context deadline exceeded`, 1 `final assistant readback + had no completed result`. Smallest change: add bounded retry/backoff around + the OpenCode `…/wait` call for the generic "agent reported an execution + error" case. Expected effect: recover the 10 execution-error runs if they are + transient. Risk: retrying a genuinely broken session. Locus: server code + (harness adapter). Evidence: the quoted diagnostics and ids. Whether the + errors are transient is **hypothesis**. + +7. **Stabilise merge resolution.** + Problem: merge-resolution pipelines contributed blocked implementers (176, + 177, 178), `final_result_validation` (179, 183, 221), and resolver runs cost + 1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver's + `implementer` blocks on a conflict workspace (e.g. missing conflict markers + or write-path scope). Locus: merge-resolution pipeline definition. Evidence: + `merger.log` lines and the run diagnostics; root cause **hypothesis**. + +## 6. What could not be verified + +- **server.log content.** It is a request/response log and contains none of the + stage diagnostics (0 matches for all diagnostic phrases). It could not be used + to re-derive failures; the run records were used instead. To settle whether + transport errors correlate with request latency, the server would need to log + per-request duration (not currently present). +- **3 non-terminal runs excluded.** Runs 220, 222, 223 were `running` when the + history and steps were captured; their incomplete steps + (223 seq1, 222 seq2, 220 seq3) have empty `finished_at` and were excluded from + all distributions. Settle by re-reading `pipelines inspect` after they finish. +- **5 unresolved busy-wait episodes.** Five `workspace busy` sequences were + still open at log end (no matching `launched` line), so their final wait is + unknown. Settle by re-parsing the lane logs once the lanes go idle. +- **Blocked-class cause distribution.** Only one blocked run (146) has a quoted + cause; the other 22 were classified by `outcome=blocked` plus stage name. A + per-run `result_preview` pass would quantify the retryable vs terminal split. +- **After-side sample size.** The 910a7aa comparison rests on 30 runs (3 + running). A larger after-window sample is required before attributing the + failure-rate drop to the change rather than to pipeline mix. +- **No system-changing measurements were attempted.** No run was started, + retried, cancelled or reconfigured; the server, ntfy service, lanes and devices + were not touched, and the phone/emulator was not used. Measurements that would + require such changes (e.g. injecting a busy workspace to time a wait, or + forcing a transport error to test retry) were deliberately not performed. + +## Verification and limits + +Commands and observed results: + +- `pipelines ping --json` → build/commit `910a7aa…`. +- `pipelines runs --json` (1 call) → 223 runs; the run-status counts reconcile + with `pipelines stats --json` (`total 223, succeeded 134, failed 73, canceled + 11, interrupted 2, active 3`). +- `pipelines inspect --json` → 223 calls (one per run id), used for all step + intervals and the taxonomy fallback. +- Go verification, run with the repository's normal invocation from the module + root on go1.27.1/darwin-arm64: + - `go build ./...` → exit 0, no output. + - `go test ./...` → all packages `ok` (cached), exit 0; the three `cmd/*` + packages and `internal/wstest` have no test files. + No Go source was changed; these commands confirm the worktree was left + buildable. + +Limits: the after-side sample is small; several classes did not recur after the +boundary for reasons the data cannot separate from the format change; the +blocked-class causes and the transience of transport errors are hypotheses, not +measurements. Raw captures are under +`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` (outside the repo).