f31f150e52
Combine the two reviewed versions of docs/pipeline-performance-review.md. - Local (main) review: the exact 201939-byte runs.json capture, the three final_result_validation sub-shapes, and the 5-to-4 unresolved busy-wait correction. - Incoming (lane-t2) review: both captures (artifacts/222/2 and artifacts/222/4), both stat snapshots (134 succeeded / 3 active and 135 succeeded / 2 active), plus the server-side rejection, deploy-wait, device-lock and refined improvement-ranking analysis. Checked: no conflict markers remain anywhere in the tree and the two sides are composed without duplicated headings or dead text. This is a documentation-only change to a dated analysis document; the packaged templates, the README and topic docs, and the packaged authoring skill references are not made stale by it, so nothing further needs updating.
431 lines
25 KiB
Markdown
431 lines
25 KiB
Markdown
# Pipeline performance and reliability review
|
||
|
||
Read-only analysis of the live Pipelines run history, step records and
|
||
orchestration logs, on 2026-10-17. The only change is this document.
|
||
|
||
## Sources and method
|
||
|
||
All numbers come from commands run in this session against the running server
|
||
(`pipelines ping --json` reports `build.version
|
||
v0.0.0-20261016212730-910a7aa798af`, `commit
|
||
910a7aa798af3eecff72fcde28b9eafde470286c`, `dirty false`).
|
||
|
||
- `pipelines runs --json` was captured twice: once into
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json`
|
||
(223 entries, 201939 bytes) and once into
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/runs.json`
|
||
(223 entries). Both captures were reused for the taxonomy, trend and
|
||
concurrency.
|
||
- `pipelines stats --json` was captured twice:
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json` reported
|
||
`total 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3`,
|
||
and `/Users/alan/.local/share/pipelines/data/artifacts/222/4/stats.json`
|
||
reported `total 223, succeeded 135, failed 73, canceled 11, interrupted 2,
|
||
active 2`. Both inventories reconcile with the run totals
|
||
(`134+73+11+2+3 = 223` and `135+73+11+2+2 = 223`).
|
||
- `pipelines inspect <id> --json` was called exactly once per run id (223 calls,
|
||
saved under `artifacts/222/4/inspect/`); step intervals were aggregated in one
|
||
pass from those files.
|
||
- Orchestration logs: `/tmp/lane{A,B,C,D,E,Go,T2,Chat}.log`, `/tmp/merger.log`,
|
||
`/tmp/deploy-server.log`, and the `lane-{hg,hs}.log` files in
|
||
`/private/var/folders/jp/q0qvjyg54fl_c5vhhk1ylz2m0000gn/T/opencode/`, plus
|
||
`~/.local/share/pipelines/logs/server.log` (16,282,789 bytes, 143,788 lines).
|
||
|
||
Quantile convention: **nearest-rank**, i.e. the p-quantile is element
|
||
`ceil(p*n)` (1-based) of the sorted list; the median is the p=0.5 element.
|
||
Durations are `finished_at - started_at` parsed with RFC3339/RFC3339Nano rules
|
||
(`Z` normalised, empty string = absent), as in `internal/pipeline/search.go`.
|
||
|
||
## 1. Failure taxonomy over the whole history
|
||
|
||
The inventory holds **223 runs** (`running 2`, `succeeded 135`, `failed 73`,
|
||
`canceled 11`, `interrupted 2`). **86 runs have a terminal non-success status**
|
||
(`failed 73`, `canceled 11`, `interrupted 2`). Every one is classified below from
|
||
the run `diagnostic`, or — where that is empty (runs 30, 32, 45) — from the
|
||
failing step's `outcome` in its `pipelines inspect` record. The class counts sum
|
||
to 86; there is no unknown bucket.
|
||
|
||
| class | count | run ids |
|
||
|---|---|---|
|
||
| `stage_outcome_blocked` | 23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 |
|
||
| `final_result_validation` | 19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 |
|
||
| `harness_transport_execution_error` | 10 | 73,74,75,77,80,149,150,152,195,202 |
|
||
| `harness_transport_timeout` | 6 | 33,36,38,59,101,138 |
|
||
| `cancelled_by_operator` | 11 | 28,39,57,63,78,108,113,114,191,211,219 |
|
||
| `review_loop_exhausted` | 5 | 31,70,96,97,127 |
|
||
| `stage_outcome_error` | 4 | 30,137,139,190 |
|
||
| `interrupted_by_deploy` | 2 | 13,24 |
|
||
| `invalid_agent_messages` | 2 | 2,5 |
|
||
| `harness_transport_readback` | 1 | 161 |
|
||
| `agent_quiescence_unconfirmed` | 1 | 43 |
|
||
| `session_transport_command_failed` | 1 | 4 |
|
||
| `duplicate_json_field` | 1 | 3 |
|
||
|
||
Classifying diagnostics (quoted verbatim from the run records):
|
||
|
||
- **stage_outcome_blocked** — a stage ends with `outcome=blocked`. The split by
|
||
blocked stage is **implementer 18, ui_reviewer 4, review 1**. Run 105:
|
||
`stage review returned outcome "blocked"`; runs 32/45 have an empty run
|
||
`diagnostic` and an `implementer` final step `outcome='blocked'`; the rest
|
||
carry `stage implementer returned outcome "blocked"` or `stage ui_reviewer
|
||
returned outcome "blocked"`. Run 146 shows a deliberate stop: *"No
|
||
screenshots/frames exist for this run (artifacts/146/1 contains only the APK;
|
||
artifacts/146/3 is empty), and the task's device rules are mutually
|
||
exclusive"*.
|
||
- **final_result_validation** — malformed/schema-rejected model result. Run 8:
|
||
`model stage implementer: FINAL RESULT VALIDATION: invalid model stage result:
|
||
invalid character 'I' looking for beginning of value at byte 1; unsafe to
|
||
repair incomplete values or a non-object response; completed edits and tool
|
||
side effects are retained and not undone; a full retry repeats work`. Run 221:
|
||
`model stage reviewer: FINAL RESULT VALIDATION: invalid model stage result:
|
||
invalid character '}' after array element at byte 2929; isolated format repair
|
||
failed: retained OpenCode session ses_eb8c5ab8affeAeLm9sFdG8uimz: context
|
||
deadline exceeded; …`. Two sub-shapes were observed:
|
||
`unsafe to repair incomplete values or a non-object response`
|
||
(runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) and
|
||
`isolated format repair failed: retained OpenCode session ses_…: context
|
||
deadline exceeded` (runs 55,81,174,179,183,221 — 6 runs). Runs 85/142 add a
|
||
third: `unknown result field`; run 170's diagnostic is `invalid character '›'
|
||
after object key:value pair at byte 780`.
|
||
- **harness_transport_execution_error** — e.g. run 73: `model stage ui_reviewer:
|
||
HARNESS TRANSPORT: harness transport failure: OpenCode agent reported an
|
||
execution error`.
|
||
- **harness_transport_timeout** — run 33: `model stage implementer: HARNESS
|
||
TRANSPORT: context deadline exceeded\ncontext deadline exceeded`.
|
||
- **cancelled_by_operator** — `context canceled`, status `canceled`.
|
||
- **review_loop_exhausted** — run 31: `Pipelines stopped reviewer after 3 visits
|
||
on outcome rejected`.
|
||
- **stage_outcome_error** — run 190: `stage implementer returned outcome
|
||
"error"`; runs 30/137/139 have a final step `outcome='error'`.
|
||
- **interrupted_by_deploy** — run 13/24: `Server stopped; run was not resumed`,
|
||
status `interrupted`.
|
||
- **invalid_agent_messages** — run 2/5: `model stage planner: invalid agent
|
||
messages`.
|
||
- **harness_transport_readback** — run 161: `harness transport failure: final
|
||
assistant readback had no completed result`.
|
||
- **agent_quiescence_unconfirmed** — run 43: `agent quiescence not confirmed;
|
||
inspect retained session before further work`.
|
||
- **session_transport_command_failed** — run 4: `model stage implementer:
|
||
OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait
|
||
command failed: CLI exit code 1`.
|
||
- **duplicate_json_field** — run 3: `invalid model stage result: duplicate JSON
|
||
field "performance_review"`.
|
||
|
||
`server.log` contains none of these diagnostic strings (bounded search:
|
||
`FINAL RESULT VALIDATION` 0, `HARNESS TRANSPORT` 0, `invalid agent messages` 0,
|
||
`stopped reviewer after` 0, `quiescence` 0); it is a request/response log
|
||
(`command received`/`response sent`, plus `summarise: harness=opencode …`).
|
||
Diagnostics live in the run records, not the server log.
|
||
|
||
## 2. Stage cost
|
||
|
||
The 223 runs produced **828 steps**; **826 have both `started_at` and
|
||
`finished_at`** and were used. The 2 without a `finished_at` belong to the 2
|
||
still-running runs and were excluded, not fabricated: run 223 seq2 `reviewer`
|
||
and run 222 seq4 `implementer`. By run: 221 runs have a non-empty `finished_at`
|
||
and all 223 runs contribute at least one complete step. This is the population
|
||
at the 2026-10-17 capture, when run 220 had already finished — an earlier working
|
||
note that called the population "222 finished" was stale and is corrected here.
|
||
|
||
Total wall-hours by stage key (sum of `finished_at - started_at`), with the share
|
||
spent on repeat visits (`control.visit > 1`, i.e. review/retry loops) split out:
|
||
|
||
| stage key | total (h) | single-pass (h) | repeat (h) | steps |
|
||
|---|---|---|---|---|
|
||
| implementer | 41.36 | 33.73 | 7.62 | 270 |
|
||
| reviewer | 20.72 | 15.76 | 4.96 | 220 |
|
||
| ui_reviewer | 15.18 | 13.93 | 1.25 | 135 |
|
||
| planner | 3.55 | 3.55 | 0.00 | 79 |
|
||
| committer | 2.00 | 2.00 | 0.00 | 99 |
|
||
| worker/research/review/probe/report/opencode/pi | 0.16 | 0.16 | 0.00 | 23 |
|
||
|
||
The dominant stage keys are **implementer (41.36 h), reviewer (20.72 h) and
|
||
ui_reviewer (15.18 h)** — together 77.26 of 82.97 h. Repeat visits of the same
|
||
stage cost **13.83 h (16.7 %)**, almost all in implementer (7.62 h) and reviewer
|
||
(4.96 h).
|
||
|
||
Per (pipeline, stage key) distribution (count / median / p90 / max in seconds /
|
||
total hours), for the highest-volume keys across the three highest-volume
|
||
pipelines:
|
||
|
||
| pipeline | stage | n | median | p90 | max | total (h) |
|
||
|---|---|---|---|---|---|---|
|
||
| Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 |
|
||
| Pipelines maintenance - plan implement review | implementer | 90 | 370 | 847 | 3601 | 12.07 |
|
||
| Pipelines maintenance - plan implement review | reviewer | 79 | 245 | 503 | 12613 | 10.54 |
|
||
| Focused tweak - implement review commit | ui_reviewer | 81 | 293 | 1030 | 2158 | 10.05 |
|
||
| Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 |
|
||
| Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 |
|
||
| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 |
|
||
| Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 |
|
||
| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 |
|
||
| Pipelines quick - plan implement review | reviewer | 13 | 324 | 896 | 1800 | 1.69 |
|
||
| Focused tweak - implement review commit | committer | 63 | 31 | 180 | 644 | 1.13 |
|
||
|
||
Three pipelines are shown (Focused tweak, Pipelines maintenance, Pipelines
|
||
quick); the full per-pipeline table was produced by the aggregation command.
|
||
|
||
Notable long tails: one `reviewer` step in Pipelines maintenance ran 12613 s
|
||
(3.5 h); one Pipelines-maintenance `implementer` step hit 3601 s (≈ the 1 h
|
||
stage timeout), while the Focused-tweak `implementer` max is 1801 s and its p90
|
||
1414 s. The `ui_reviewer` p90 is high (1030 s in Focused tweak).
|
||
|
||
## 3. Reliability trend, split at 910a7aa
|
||
|
||
Boundary: `2026-10-16T21:27:00Z`, the deploy of commit 910a7aa. `pipelines ping
|
||
--json` confirms the running `build.commit` is 910a7aa. `deploy-server.log`
|
||
corroborates: `22:27:45 quiescing the lanes …` / `22:27:45 building from main
|
||
(910a7aa Merge lane-app2: …)` / `22:27:57 server healthy; deployed 910a7aa`
|
||
(its clock is local UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by
|
||
`created_at`: the last "before" run is 193 (`2026-10-16T21:21:33Z`), the first
|
||
"after" is 194 (`2026-10-16T21:28:00Z`).
|
||
|
||
| side | n | running | failed | failed/(failed+succeeded) | non-success/terminal |
|
||
|---|---|---|---|---|---|
|
||
| before | 193 | 0 | 68 | 37.4 % | 40.9 % |
|
||
| after | 30 | 2 | 5 | 19.2 % | 25.0 % |
|
||
|
||
Failure-class mix, before → after (counts):
|
||
|
||
| class | before | after |
|
||
|---|---|---|
|
||
| stage_outcome_blocked | 21 | 2 |
|
||
| final_result_validation | 18 | 1 |
|
||
| harness_transport_execution_error | 8 | 2 |
|
||
| harness_transport_timeout | 6 | 0 |
|
||
| cancelled_by_operator | 9 | 2 |
|
||
| review_loop_exhausted | 5 | 0 |
|
||
| stage_outcome_error | 4 | 0 |
|
||
| interrupted_by_deploy | 2 | 0 |
|
||
| invalid_agent_messages | 2 | 0 |
|
||
| harness_transport_readback | 1 | 0 |
|
||
| agent_quiescence_unconfirmed | 1 | 0 |
|
||
| session_transport_command_failed | 1 | 0 |
|
||
| duplicate_json_field | 1 | 0 |
|
||
|
||
Read literally the change correlates with fewer malformed-result failures:
|
||
`final_result_validation + duplicate_json_field + invalid_agent_messages` falls
|
||
from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is **confounded
|
||
and under-powered**: the after side is only 30 runs (2 still running), the
|
||
pipeline mix differs (after is mostly Focused tweak 15, Pipelines maintenance 7,
|
||
Merge-resolution 3 — several one-off template/docker/research pipelines appear
|
||
only before), and several before-side classes (transport timeout, review-loop,
|
||
deploy interruption) simply never recurred in the short after window rather than
|
||
being fixed by 910a7aa. The data is consistent with the format/repair change
|
||
helping; it does not establish causation. A larger after sample is needed.
|
||
|
||
## 4. Throughput limits (measured)
|
||
|
||
**Busy-workspace waits.** The lane logs record `workspace busy; waiting to
|
||
launch <file>` at a fixed 30 s cadence: **897 such lines**. Grouping them into
|
||
episodes (same lane+file, ending at the `launched <file>` line) gives **27
|
||
episodes, 23 resolved and 4 still open at log end**. For the 23 resolved
|
||
episodes the wait is `launch_time - first_busy`:
|
||
|
||
- n=23, median **992 s (16.5 min)**, p90 **2293 s (38.2 min)**, max
|
||
**9070 s (2.5 h)**, total **9.07 h**. This independent reimplementation of the
|
||
rule includes the laneGo `malformed-result-brief.txt` episode (2170 s); an
|
||
earlier working note that dropped it reported n=22 / 8.47 h and is corrected
|
||
here.
|
||
|
||
Longest episodes: laneA `s02-tui-omni-search.txt` waited 13:12:33 → 15:43:43
|
||
(9070 s) before `15:43:43 launched s02-tui-omni-search.txt as run 137`; laneGo
|
||
`s02-tui-omni-search.txt` waited 15:47:22 → 16:48:28 (3666 s).
|
||
|
||
**Server-side rejection of a second run in a busy workspace.** `server.log`
|
||
records **910** `error="run <id> is already active in workspace <path>; wait for
|
||
it to finish or start this run in a different workspace …"` responses, from
|
||
**29 distinct blocking runs**, concentrated on 2026/10/16 (905) vs 2026/10/15
|
||
(5). By workspace: `lane-go` 387, `/Users/alan/repos/pipelines` 307, `lane-app2`
|
||
166, `lane-c` 49, `ws-c` 1. So the dominant limit is one-run-per-workspace, and
|
||
the lanes retry it hundreds of times.
|
||
|
||
**Server-restart holds.** `holding: another lane is restarting the server`
|
||
appears **800** times; grouping holds per lane while consecutive hold lines are
|
||
≤60 s apart (the 30 s poll cadence plus jitter; a wider gap starts a new
|
||
episode) gives 29 episodes totalling **6.42 h**, median 480 s, max 3572 s. (A
|
||
strict consecutive-line regroup, which splits an episode on any interleaved
|
||
line, gives 32 episodes; the 29 figure is the gap-based grouping.) `building
|
||
server and cli from main` 13 times; `server redeployed and healthy` 10 times.
|
||
|
||
**Merge/main-lane waits.** `merge deferred: main lane busy` appears **11** times;
|
||
`merge requested for …` 24 times; `MERGE CONFLICT` 15 times; `SYNC CONFLICT: lane
|
||
could not take main's latest; continuing on the older base` 45 times.
|
||
|
||
**Merge/resolver activity.** `merger.log` shows the queue draining through
|
||
pipeline resolvers. 12 resolver runs have measurable durations, total
|
||
**5153 s (1.43 h)**, median 421 s, max 1062 s: `run 176 20 s` and `177 20 s`
|
||
("completed the merge" immediately) versus `run 180 661 s`, `182 602 s`,
|
||
`183 1062 s` (handed to a pipeline). Four events record `could not launch a
|
||
resolver; aborting the merge for <lane>`; three resolvers `did NOT land` their
|
||
lane (`179 lane-app2`, `183 lane-go`, `221 lane-c`). The `merge-queue` marker in
|
||
the worktree currently holds `lane-go` and `lane-app2`.
|
||
|
||
**Deploy waits.** `deploy-server.log` shows one deploy waiting for quiescence:
|
||
`13:39:07 quiescing the lanes so no new run starts` with `waiting: N run(s)
|
||
active` every ~20 s from 3 down to 1, until `14:18:18 building from main
|
||
(dce624b …)` — a **2351 s (39.2 min)** wait. The 910a7aa deploy at `22:27:45`
|
||
waited 12 s (0 active runs).
|
||
|
||
**Device/emulator churn.** `FAILED once: <file> - recovering the emulator and
|
||
retrying` appears 50 times and `FAILED twice: <file> - recording and continuing`
|
||
35 times (e.g. `02:53:47 [laneA] FAILED once: q03-pane-opaque-A.txt - recovering
|
||
the emulator and retrying`). No `device lock` event string was found in the lane
|
||
logs or `server.log`.
|
||
|
||
**Real concurrency.** Overlapping step intervals show a maximum of **7
|
||
simultaneous executing steps**, at `2026-10-16T14:47:35Z`, spanning the 6 runs
|
||
[134, 136, 137, 138, 139, 140] (one run had two steps overlapping at that
|
||
instant). Step intervals are counted as **closed** (`started_at ≤ t ≤
|
||
finished_at`), so run 134's `reviewer` (ending 14:47:35Z) and its `implementer`
|
||
(starting 14:47:35Z) both count at the touching boundary; with half-open
|
||
intervals (`started_at ≤ t < finished_at`) the maximum is **6 steps at
|
||
14:47:22Z**, the same six runs. Using run wall-spans (`created_at`..`finished_at`) the maximum is **6
|
||
concurrent runs** at `2026-10-16T14:47:22Z`, the same six runs. 157 runs had at
|
||
least one step interval overlapping another run. So despite many queued runs,
|
||
the observed executing concurrency never exceeded 7 steps / 6 runs — consistent
|
||
with the lane count and the 897 workspace-busy lines, and not with a
|
||
memory/harness-capacity limit.
|
||
|
||
## 5. Ranked improvements
|
||
|
||
Ranked by measured impact. Items marked **hypothesis** have no direct
|
||
measurement behind them.
|
||
|
||
1. **Remove the one-run-per-workspace contention (measured 9.07 h waiting + 910
|
||
server rejections + 6.42 h restart holds).** Problem: 23 resolved bus waits
|
||
total 9.07 h (median 992 s, max 9070 s); the server rejected a second run in a
|
||
busy workspace 910 times across 29 blocking runs; restart holds cost 6.42 h.
|
||
Smallest change: give each lane its own worktree/workspace so a lane is never
|
||
blocked by a run already active in it (or lower the fixed 30 s busy-poll).
|
||
Expected effect: cut the multi-minute launch latency and the wasted poll
|
||
lines. Risk: more concurrent worktrees consume more disk and parallel model
|
||
capacity. Locus: orchestration. Evidence: the busy/rejection counts and the
|
||
per-workspace breakdown above.
|
||
|
||
2. **Finish or budget the malformed-result repair path.**
|
||
Problem: `final_result_validation` is 19 of 86 non-success runs; in several
|
||
(e.g. 55,81,174,179,183,221) the repair itself failed with `isolated format
|
||
repair failed: retained OpenCode session ses_…: context deadline exceeded`,
|
||
while others were judged `unsafe to repair`. Smallest change: give the
|
||
isolated format-repair session its own longer deadline (or a streaming
|
||
repair), so a repairable-but-slow result is recovered instead of failing the
|
||
run. Expected effect: recover part of the timeout-repair runs; the "unsafe to
|
||
repair" set remains a separate problem. Risk: longer tail latency on
|
||
already-slow runs. Locus: server code (`internal/stage`). Evidence: the quoted
|
||
diagnostics and the before/after drop (10.9 %→3.3 %) for this cluster.
|
||
|
||
3. **Cut repeat-visit cost in the review loop.**
|
||
Problem: repeat visits of the same stage cost 13.83 h of 82.97 h (16.7 %),
|
||
concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs
|
||
(31,70,96,97,127) failed outright with `Pipelines stopped reviewer after 3
|
||
visits on outcome rejected`. Smallest change: stop looping on advisory-only
|
||
rejections (no must-fix finding), or reduce the reviewer max-visits for
|
||
advisory findings. Expected effect: remove up to ~14 h of repeat work and the
|
||
5 exhausted-loop failures, if rejections are often non-blocking. Risk: a real
|
||
defect could pass. Locus: pipeline definition (`reviewer` control). Evidence:
|
||
the repeat/single-pass split in §2 and the `review_loop_exhausted` ids.
|
||
|
||
4. **Make the `implementer` blocked outcome actionable (measured 18/23 of the
|
||
blocked class).** Problem: `stage_outcome_blocked` is the single largest class
|
||
(23/86, 26.7 %), and it is dominated by **implementer (18)** — not
|
||
`ui_reviewer`, which is only 4 (and `review` 1). Smallest change: have the
|
||
implementer stage distinguish "environment/device unavailable" (retryable)
|
||
from "acceptance genuinely unmeetable" (terminal), so the retryable subset is
|
||
retried instead of failing the run. Expected effect: a portion of the 18
|
||
implementer-blocked runs recovered. Risk: retry storms. Locus: pipeline
|
||
definition / server code. Evidence: the 23 ids and the measured blocked-stage
|
||
split; the retryable/terminal split is **hypothesis** (not measured).
|
||
|
||
5. **Hand the `ui_reviewer` captured artifacts (measured 4/23 blocked).**
|
||
Problem: `ui_reviewer` blocked in 4 runs (146,167,207,210) and its p90 is
|
||
1030 s in Focused tweak; run 146's preview reports `artifacts/146/3 is
|
||
empty`. Smallest change: whatever captures screenshots for a UI run must hand
|
||
them to the `ui_reviewer` stage, and the stage should not block when no device
|
||
is attached. Expected effect: fewer blocked UI runs. Risk: silently skipping
|
||
visual verification. Locus: pipeline definition + orchestration (artifact
|
||
handoff). Evidence: run 146 preview and the 4 blocked ids; **partly
|
||
hypothesis** — the empty-artifact cause is quoted for one run only, not
|
||
measured across all 4.
|
||
|
||
6. **Harden the harness transport path.**
|
||
Problem: 18 non-success runs are transport failures — 10 `execution error`, 6
|
||
`context deadline exceeded`, 1 `final assistant readback had no completed
|
||
result`, 1 `OpenCode post … command failed`. Smallest change: add bounded
|
||
retry/backoff around the OpenCode `…/wait` call for the generic "agent
|
||
reported an execution error" case. Expected effect: recover the 10
|
||
execution-error runs if they are transient. Risk: retrying a genuinely broken
|
||
session. Locus: server code (harness adapter). Evidence: the quoted
|
||
diagnostics and ids. Whether the errors are transient is **hypothesis**.
|
||
|
||
7. **Stabilise merge resolution.**
|
||
Problem: merge-resolution pipelines contributed blocked implementers (176,
|
||
177, 178), `final_result_validation` (179, 183, 221), and resolver runs cost
|
||
1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver's
|
||
`implementer` blocks on a conflict workspace (e.g. missing conflict markers or
|
||
write-path scope). Locus: merge-resolution pipeline definition. Evidence:
|
||
`merger.log` lines and the run diagnostics; root cause **hypothesis**.
|
||
|
||
## 6. What could not be verified
|
||
|
||
- **server.log content.** It is a request/response log and contains none of the
|
||
stage diagnostics (0 matches for every diagnostic phrase above). It could not
|
||
be used to re-derive failures; the run records were used instead. To settle
|
||
whether transport errors correlate with request latency, the server would need
|
||
to log per-request duration (not currently present).
|
||
- **2 non-terminal runs excluded.** Runs 222 and 223 were `running` when the
|
||
history and steps were captured; their incomplete steps (223 seq2 `reviewer`,
|
||
222 seq4 `implementer`) have an empty `finished_at` and were excluded from all
|
||
distributions. Settle by re-reading `pipelines inspect` after they finish.
|
||
(An earlier capture also excluded run 220 seq3 `ui_reviewer` when it was still
|
||
running; it had finished by the later capture and is included above.)
|
||
- **4 unresolved busy-wait episodes.** Four `workspace busy` sequences were still
|
||
open at log end (no matching `launched` line), so their final wait is unknown.
|
||
(An earlier working note reported 5 unresolved episodes; the re-counted
|
||
population gives 4.) Settle by re-parsing the lane logs once the lanes go idle.
|
||
- **Blocked-class cause distribution.** Only run 146 has a quoted cause; the
|
||
other blocked runs were classified by `outcome=blocked` plus stage name. A
|
||
per-run `result_preview` pass would quantify the retryable vs terminal split.
|
||
- **After-side sample size.** The 910a7aa comparison rests on 30 runs (2 still
|
||
running). A larger after-window sample is required before attributing the
|
||
failure-rate drop to the change rather than to pipeline mix.
|
||
- **Device-lock waits.** No `device lock` event string exists in the lane logs or
|
||
`server.log`; emulator trouble appears only as `recovering the emulator`
|
||
retries. A device-lock measurement would need the orchestration to log an
|
||
explicit acquire/wait/release.
|
||
- **No system-changing measurements were attempted.** No run was started,
|
||
retried, cancelled or reconfigured; the server, ntfy service, lanes and devices
|
||
were not touched, and the phone/emulator was not used. Measurements that would
|
||
require such changes (e.g. injecting a busy workspace to time a wait, or
|
||
forcing a transport error to test retry) were deliberately not performed.
|
||
|
||
## Verification and limits
|
||
|
||
Commands and observed results (all run in this session):
|
||
|
||
- `pipelines ping --json` → build/commit `910a7aa…`, `dirty false`.
|
||
- `pipelines runs --json` (2 calls, at different capture times) → 223 runs; the
|
||
two run-status counts reconcile with `pipelines stats --json`
|
||
(`total 223, succeeded 134/135, failed 73, canceled 11, interrupted 2, active
|
||
3/2`).
|
||
- `pipelines inspect <id> --json` → 223 calls (one per run id), used for all step
|
||
intervals and the taxonomy fallback.
|
||
- `server.log` bounded searches: 910 `is already active in workspace`
|
||
rejections; 0 diagnostic strings.
|
||
- Lane/orchestration logs (streaming `grep`/`awk`): 897 `workspace busy` lines,
|
||
800 restart holds, 11 `merge deferred`, 12 resolver durations, 50/35 emulator
|
||
retries.
|
||
- Go verification, run with the repository's normal invocation from the module
|
||
root:
|
||
- `go build ./...` → exit 0, no output.
|
||
- `go test ./...` → exit 0; 20 packages `ok`, 3 `cmd/*` and `internal/wstest`
|
||
have no test files. No Go source was changed; these confirm the worktree was
|
||
left buildable.
|
||
|
||
Limits: the after-side sample is small; several classes did not recur after the
|
||
boundary for reasons the data cannot separate from the format change; the
|
||
blocked-class cause distribution and the transience of transport errors are
|
||
hypotheses, not measurements. Raw captures are under
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).
|