7b72b8f20f
Treat a failure to reach the harness before any prompt or tool call is admitted (connecting or creating the session) as a distinct retryable reach failure, so a transport blip at the first stage no longer ends a mutating run; keep post-admission readback/poll/agent failures retried only on an explicit read-only stage and never replay admitted tool work. Preserve the true cause when the context is already done (join ctx.Err with the typed transport error) and report a sanitized unknown provider error name/status instead of a blanket summary. Make the isolated format-repair deadline proportional to the stage timeout (floored, capped and clamped to remaining minus a margin) so a slow-but-repairable result is recovered without ever extending the stage deadline; the bounded ladder, value guard, deny-all formatter session, cumulative usage accounting and no-tool-replay rule are unchanged. Update docs/model-results.md, docs/pipeline-performance-review.md and the packaged skill references; add stage, opencode and server fixtures.
459 lines
28 KiB
Markdown
459 lines
28 KiB
Markdown
# Pipeline performance and reliability review
|
||
|
||
Read-only analysis of the live Pipelines run history, step records and
|
||
orchestration logs, on 2026-10-17. The only change is this document.
|
||
|
||
## Sources and method
|
||
|
||
All numbers come from commands run in this session against the running server
|
||
(`pipelines ping --json` reports `build.version
|
||
v0.0.0-20261016212730-910a7aa798af`, `commit
|
||
910a7aa798af3eecff72fcde28b9eafde470286c`, `dirty false`).
|
||
|
||
- `pipelines runs --json` was captured twice: once into
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json`
|
||
(223 entries, 201939 bytes) and once into
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/runs.json`
|
||
(223 entries). Both captures were reused for the taxonomy, trend and
|
||
concurrency.
|
||
- `pipelines stats --json` was captured twice:
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json` reported
|
||
`total 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3`,
|
||
and `/Users/alan/.local/share/pipelines/data/artifacts/222/4/stats.json`
|
||
reported `total 223, succeeded 135, failed 73, canceled 11, interrupted 2,
|
||
active 2`. Both inventories reconcile with the run totals
|
||
(`134+73+11+2+3 = 223` and `135+73+11+2+2 = 223`).
|
||
- `pipelines inspect <id> --json` was called exactly once per run id (223 calls,
|
||
saved under `artifacts/222/4/inspect/`); step intervals were aggregated in one
|
||
pass from those files.
|
||
- Orchestration logs: `/tmp/lane{A,B,C,D,E,Go,T2,Chat}.log`, `/tmp/merger.log`,
|
||
`/tmp/deploy-server.log`, and the `lane-{hg,hs}.log` files in
|
||
`/private/var/folders/jp/q0qvjyg54fl_c5vhhk1ylz2m0000gn/T/opencode/`, plus
|
||
`~/.local/share/pipelines/logs/server.log` (16,282,789 bytes, 143,788 lines).
|
||
|
||
Quantile convention: **nearest-rank**, i.e. the p-quantile is element
|
||
`ceil(p*n)` (1-based) of the sorted list; the median is the p=0.5 element.
|
||
Durations are `finished_at - started_at` parsed with RFC3339/RFC3339Nano rules
|
||
(`Z` normalised, empty string = absent), as in `internal/pipeline/search.go`.
|
||
|
||
## 1. Failure taxonomy over the whole history
|
||
|
||
The inventory holds **223 runs** (`running 2`, `succeeded 135`, `failed 73`,
|
||
`canceled 11`, `interrupted 2`). **86 runs have a terminal non-success status**
|
||
(`failed 73`, `canceled 11`, `interrupted 2`). Every one is classified below from
|
||
the run `diagnostic`, or — where that is empty (runs 30, 32, 45) — from the
|
||
failing step's `outcome` in its `pipelines inspect` record. The class counts sum
|
||
to 86; there is no unknown bucket.
|
||
|
||
| class | count | run ids |
|
||
|---|---|---|
|
||
| `stage_outcome_blocked` | 23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 |
|
||
| `final_result_validation` | 19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 |
|
||
| `harness_transport_execution_error` | 10 | 73,74,75,77,80,149,150,152,195,202 |
|
||
| `harness_transport_timeout` | 6 | 33,36,38,59,101,138 |
|
||
| `cancelled_by_operator` | 11 | 28,39,57,63,78,108,113,114,191,211,219 |
|
||
| `review_loop_exhausted` | 5 | 31,70,96,97,127 |
|
||
| `stage_outcome_error` | 4 | 30,137,139,190 |
|
||
| `interrupted_by_deploy` | 2 | 13,24 |
|
||
| `invalid_agent_messages` | 2 | 2,5 |
|
||
| `harness_transport_readback` | 1 | 161 |
|
||
| `agent_quiescence_unconfirmed` | 1 | 43 |
|
||
| `session_transport_command_failed` | 1 | 4 |
|
||
| `duplicate_json_field` | 1 | 3 |
|
||
|
||
Classifying diagnostics (quoted verbatim from the run records):
|
||
|
||
- **stage_outcome_blocked** — a stage ends with `outcome=blocked`. The split by
|
||
blocked stage is **implementer 18, ui_reviewer 4, review 1**. Run 105:
|
||
`stage review returned outcome "blocked"`; runs 32/45 have an empty run
|
||
`diagnostic` and an `implementer` final step `outcome='blocked'`; the rest
|
||
carry `stage implementer returned outcome "blocked"` or `stage ui_reviewer
|
||
returned outcome "blocked"`. Run 146 shows a deliberate stop: *"No
|
||
screenshots/frames exist for this run (artifacts/146/1 contains only the APK;
|
||
artifacts/146/3 is empty), and the task's device rules are mutually
|
||
exclusive"*.
|
||
- **final_result_validation** — malformed/schema-rejected model result. Run 8:
|
||
`model stage implementer: FINAL RESULT VALIDATION: invalid model stage result:
|
||
invalid character 'I' looking for beginning of value at byte 1; unsafe to
|
||
repair incomplete values or a non-object response; completed edits and tool
|
||
side effects are retained and not undone; a full retry repeats work`. Run 221:
|
||
`model stage reviewer: FINAL RESULT VALIDATION: invalid model stage result:
|
||
invalid character '}' after array element at byte 2929; isolated format repair
|
||
failed: retained OpenCode session ses_eb8c5ab8affeAeLm9sFdG8uimz: context
|
||
deadline exceeded; …`. Two sub-shapes were observed:
|
||
`unsafe to repair incomplete values or a non-object response`
|
||
(runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) and
|
||
`isolated format repair failed: retained OpenCode session ses_…: context
|
||
deadline exceeded` (runs 55,81,174,179,183,221 — 6 runs). Runs 85/142 add a
|
||
third: `unknown result field`; run 170's diagnostic is `invalid character '›'
|
||
after object key:value pair at byte 780`.
|
||
- **harness_transport_execution_error** — e.g. run 73: `model stage ui_reviewer:
|
||
HARNESS TRANSPORT: harness transport failure: OpenCode agent reported an
|
||
execution error`.
|
||
- **harness_transport_timeout** — run 33: `model stage implementer: HARNESS
|
||
TRANSPORT: context deadline exceeded\ncontext deadline exceeded`.
|
||
- **cancelled_by_operator** — `context canceled`, status `canceled`.
|
||
- **review_loop_exhausted** — run 31: `Pipelines stopped reviewer after 3 visits
|
||
on outcome rejected`.
|
||
- **stage_outcome_error** — run 190: `stage implementer returned outcome
|
||
"error"`; runs 30/137/139 have a final step `outcome='error'`.
|
||
- **interrupted_by_deploy** — run 13/24: `Server stopped; run was not resumed`,
|
||
status `interrupted`.
|
||
- **invalid_agent_messages** — run 2/5: `model stage planner: invalid agent
|
||
messages`.
|
||
- **harness_transport_readback** — run 161: `harness transport failure: final
|
||
assistant readback had no completed result`.
|
||
- **agent_quiescence_unconfirmed** — run 43: `agent quiescence not confirmed;
|
||
inspect retained session before further work`.
|
||
- **session_transport_command_failed** — run 4: `model stage implementer:
|
||
OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait
|
||
command failed: CLI exit code 1`.
|
||
- **duplicate_json_field** — run 3: `invalid model stage result: duplicate JSON
|
||
field "performance_review"`.
|
||
|
||
`server.log` contains none of these diagnostic strings (bounded search:
|
||
`FINAL RESULT VALIDATION` 0, `HARNESS TRANSPORT` 0, `invalid agent messages` 0,
|
||
`stopped reviewer after` 0, `quiescence` 0); it is a request/response log
|
||
(`command received`/`response sent`, plus `summarise: harness=opencode …`).
|
||
Diagnostics live in the run records, not the server log.
|
||
|
||
## 2. Stage cost
|
||
|
||
The 223 runs produced **828 steps**; **826 have both `started_at` and
|
||
`finished_at`** and were used. The 2 without a `finished_at` belong to the 2
|
||
still-running runs and were excluded, not fabricated: run 223 seq2 `reviewer`
|
||
and run 222 seq4 `implementer`. By run: 221 runs have a non-empty `finished_at`
|
||
and all 223 runs contribute at least one complete step. This is the population
|
||
at the 2026-10-17 capture, when run 220 had already finished — an earlier working
|
||
note that called the population "222 finished" was stale and is corrected here.
|
||
|
||
Total wall-hours by stage key (sum of `finished_at - started_at`), with the share
|
||
spent on repeat visits (`control.visit > 1`, i.e. review/retry loops) split out:
|
||
|
||
| stage key | total (h) | single-pass (h) | repeat (h) | steps |
|
||
|---|---|---|---|---|
|
||
| implementer | 41.36 | 33.73 | 7.62 | 270 |
|
||
| reviewer | 20.72 | 15.76 | 4.96 | 220 |
|
||
| ui_reviewer | 15.18 | 13.93 | 1.25 | 135 |
|
||
| planner | 3.55 | 3.55 | 0.00 | 79 |
|
||
| committer | 2.00 | 2.00 | 0.00 | 99 |
|
||
| worker/research/review/probe/report/opencode/pi | 0.16 | 0.16 | 0.00 | 23 |
|
||
|
||
The dominant stage keys are **implementer (41.36 h), reviewer (20.72 h) and
|
||
ui_reviewer (15.18 h)** — together 77.26 of 82.97 h. Repeat visits of the same
|
||
stage cost **13.83 h (16.7 %)**, almost all in implementer (7.62 h) and reviewer
|
||
(4.96 h).
|
||
|
||
Per (pipeline, stage key) distribution (count / median / p90 / max in seconds /
|
||
total hours), for the highest-volume keys across the three highest-volume
|
||
pipelines:
|
||
|
||
| pipeline | stage | n | median | p90 | max | total (h) |
|
||
|---|---|---|---|---|---|---|
|
||
| Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 |
|
||
| Pipelines maintenance - plan implement review | implementer | 90 | 370 | 847 | 3601 | 12.07 |
|
||
| Pipelines maintenance - plan implement review | reviewer | 79 | 245 | 503 | 12613 | 10.54 |
|
||
| Focused tweak - implement review commit | ui_reviewer | 81 | 293 | 1030 | 2158 | 10.05 |
|
||
| Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 |
|
||
| Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 |
|
||
| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 |
|
||
| Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 |
|
||
| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 |
|
||
| Pipelines quick - plan implement review | reviewer | 13 | 324 | 896 | 1800 | 1.69 |
|
||
| Focused tweak - implement review commit | committer | 63 | 31 | 180 | 644 | 1.13 |
|
||
|
||
Three pipelines are shown (Focused tweak, Pipelines maintenance, Pipelines
|
||
quick); the full per-pipeline table was produced by the aggregation command.
|
||
|
||
Notable long tails: one `reviewer` step in Pipelines maintenance ran 12613 s
|
||
(3.5 h); one Pipelines-maintenance `implementer` step hit 3601 s (≈ the 1 h
|
||
stage timeout), while the Focused-tweak `implementer` max is 1801 s and its p90
|
||
1414 s. The `ui_reviewer` p90 is high (1030 s in Focused tweak).
|
||
|
||
## 3. Reliability trend, split at 910a7aa
|
||
|
||
Boundary: `2026-10-16T21:27:00Z`, the deploy of commit 910a7aa. `pipelines ping
|
||
--json` confirms the running `build.commit` is 910a7aa. `deploy-server.log`
|
||
corroborates: `22:27:45 quiescing the lanes …` / `22:27:45 building from main
|
||
(910a7aa Merge lane-app2: …)` / `22:27:57 server healthy; deployed 910a7aa`
|
||
(its clock is local UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by
|
||
`created_at`: the last "before" run is 193 (`2026-10-16T21:21:33Z`), the first
|
||
"after" is 194 (`2026-10-16T21:28:00Z`).
|
||
|
||
| side | n | running | failed | failed/(failed+succeeded) | non-success/terminal |
|
||
|---|---|---|---|---|---|
|
||
| before | 193 | 0 | 68 | 37.4 % | 40.9 % |
|
||
| after | 30 | 2 | 5 | 19.2 % | 25.0 % |
|
||
|
||
Failure-class mix, before → after (counts):
|
||
|
||
| class | before | after |
|
||
|---|---|---|
|
||
| stage_outcome_blocked | 21 | 2 |
|
||
| final_result_validation | 18 | 1 |
|
||
| harness_transport_execution_error | 8 | 2 |
|
||
| harness_transport_timeout | 6 | 0 |
|
||
| cancelled_by_operator | 9 | 2 |
|
||
| review_loop_exhausted | 5 | 0 |
|
||
| stage_outcome_error | 4 | 0 |
|
||
| interrupted_by_deploy | 2 | 0 |
|
||
| invalid_agent_messages | 2 | 0 |
|
||
| harness_transport_readback | 1 | 0 |
|
||
| agent_quiescence_unconfirmed | 1 | 0 |
|
||
| session_transport_command_failed | 1 | 0 |
|
||
| duplicate_json_field | 1 | 0 |
|
||
|
||
Read literally the change correlates with fewer malformed-result failures:
|
||
`final_result_validation + duplicate_json_field + invalid_agent_messages` falls
|
||
from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is **confounded
|
||
and under-powered**: the after side is only 30 runs (2 still running), the
|
||
pipeline mix differs (after is mostly Focused tweak 15, Pipelines maintenance 7,
|
||
Merge-resolution 3 — several one-off template/docker/research pipelines appear
|
||
only before), and several before-side classes (transport timeout, review-loop,
|
||
deploy interruption) simply never recurred in the short after window rather than
|
||
being fixed by 910a7aa. The data is consistent with the format/repair change
|
||
helping; it does not establish causation. A larger after sample is needed.
|
||
|
||
## 4. Throughput limits (measured)
|
||
|
||
**Busy-workspace waits.** The lane logs record `workspace busy; waiting to
|
||
launch <file>` at a fixed 30 s cadence: **897 such lines**. Grouping them into
|
||
episodes (same lane+file, ending at the `launched <file>` line) gives **27
|
||
episodes, 23 resolved and 4 still open at log end**. For the 23 resolved
|
||
episodes the wait is `launch_time - first_busy`:
|
||
|
||
- n=23, median **992 s (16.5 min)**, p90 **2293 s (38.2 min)**, max
|
||
**9070 s (2.5 h)**, total **9.07 h**. This independent reimplementation of the
|
||
rule includes the laneGo `malformed-result-brief.txt` episode (2170 s); an
|
||
earlier working note that dropped it reported n=22 / 8.47 h and is corrected
|
||
here.
|
||
|
||
Longest episodes: laneA `s02-tui-omni-search.txt` waited 13:12:33 → 15:43:43
|
||
(9070 s) before `15:43:43 launched s02-tui-omni-search.txt as run 137`; laneGo
|
||
`s02-tui-omni-search.txt` waited 15:47:22 → 16:48:28 (3666 s).
|
||
|
||
**Server-side rejection of a second run in a busy workspace.** `server.log`
|
||
records **910** `error="run <id> is already active in workspace <path>; wait for
|
||
it to finish or start this run in a different workspace …"` responses, from
|
||
**29 distinct blocking runs**, concentrated on 2026/10/16 (905) vs 2026/10/15
|
||
(5). By workspace: `lane-go` 387, `/Users/alan/repos/pipelines` 307, `lane-app2`
|
||
166, `lane-c` 49, `ws-c` 1. So the dominant limit is one-run-per-workspace, and
|
||
the lanes retry it hundreds of times.
|
||
|
||
**Server-restart holds.** `holding: another lane is restarting the server`
|
||
appears **800** times; grouping holds per lane while consecutive hold lines are
|
||
≤60 s apart (the 30 s poll cadence plus jitter; a wider gap starts a new
|
||
episode) gives 29 episodes totalling **6.42 h**, median 480 s, max 3572 s. (A
|
||
strict consecutive-line regroup, which splits an episode on any interleaved
|
||
line, gives 32 episodes; the 29 figure is the gap-based grouping.) `building
|
||
server and cli from main` 13 times; `server redeployed and healthy` 10 times.
|
||
|
||
**Merge/main-lane waits.** `merge deferred: main lane busy` appears **11** times;
|
||
`merge requested for …` 24 times; `MERGE CONFLICT` 15 times; `SYNC CONFLICT: lane
|
||
could not take main's latest; continuing on the older base` 45 times.
|
||
|
||
**Merge/resolver activity.** `merger.log` shows the queue draining through
|
||
pipeline resolvers. 12 resolver runs have measurable durations, total
|
||
**5153 s (1.43 h)**, median 421 s, max 1062 s: `run 176 20 s` and `177 20 s`
|
||
("completed the merge" immediately) versus `run 180 661 s`, `182 602 s`,
|
||
`183 1062 s` (handed to a pipeline). Four events record `could not launch a
|
||
resolver; aborting the merge for <lane>`; three resolvers `did NOT land` their
|
||
lane (`179 lane-app2`, `183 lane-go`, `221 lane-c`). The `merge-queue` marker in
|
||
the worktree currently holds `lane-go` and `lane-app2`.
|
||
|
||
**Deploy waits.** `deploy-server.log` shows one deploy waiting for quiescence:
|
||
`13:39:07 quiescing the lanes so no new run starts` with `waiting: N run(s)
|
||
active` every ~20 s from 3 down to 1, until `14:18:18 building from main
|
||
(dce624b …)` — a **2351 s (39.2 min)** wait. The 910a7aa deploy at `22:27:45`
|
||
waited 12 s (0 active runs).
|
||
|
||
**Device/emulator churn.** `FAILED once: <file> - recovering the emulator and
|
||
retrying` appears 50 times and `FAILED twice: <file> - recording and continuing`
|
||
35 times (e.g. `02:53:47 [laneA] FAILED once: q03-pane-opaque-A.txt - recovering
|
||
the emulator and retrying`). No `device lock` event string was found in the lane
|
||
logs or `server.log`.
|
||
|
||
**Real concurrency.** Overlapping step intervals show a maximum of **7
|
||
simultaneous executing steps**, at `2026-10-16T14:47:35Z`, spanning the 6 runs
|
||
[134, 136, 137, 138, 139, 140] (one run had two steps overlapping at that
|
||
instant). Step intervals are counted as **closed** (`started_at ≤ t ≤
|
||
finished_at`), so run 134's `reviewer` (ending 14:47:35Z) and its `implementer`
|
||
(starting 14:47:35Z) both count at the touching boundary; with half-open
|
||
intervals (`started_at ≤ t < finished_at`) the maximum is **6 steps at
|
||
14:47:22Z**, the same six runs. Using run wall-spans (`created_at`..`finished_at`) the maximum is **6
|
||
concurrent runs** at `2026-10-16T14:47:22Z`, the same six runs. 157 runs had at
|
||
least one step interval overlapping another run. So despite many queued runs,
|
||
the observed executing concurrency never exceeded 7 steps / 6 runs — consistent
|
||
with the lane count and the 897 workspace-busy lines, and not with a
|
||
memory/harness-capacity limit.
|
||
|
||
## 5. Ranked improvements
|
||
|
||
Ranked by measured impact. Items marked **hypothesis** have no direct
|
||
measurement behind them.
|
||
|
||
1. **Remove the one-run-per-workspace contention (measured 9.07 h waiting + 910
|
||
server rejections + 6.42 h restart holds).** Problem: 23 resolved bus waits
|
||
total 9.07 h (median 992 s, max 9070 s); the server rejected a second run in a
|
||
busy workspace 910 times across 29 blocking runs; restart holds cost 6.42 h.
|
||
Smallest change: give each lane its own worktree/workspace so a lane is never
|
||
blocked by a run already active in it (or lower the fixed 30 s busy-poll).
|
||
Expected effect: cut the multi-minute launch latency and the wasted poll
|
||
lines. Risk: more concurrent worktrees consume more disk and parallel model
|
||
capacity. Locus: orchestration. Evidence: the busy/rejection counts and the
|
||
per-workspace breakdown above.
|
||
|
||
2. **Finish or budget the malformed-result repair path.**
|
||
Problem: `final_result_validation` is 19 of 86 non-success runs; in several
|
||
(e.g. 55,81,174,179,183,221) the repair itself failed with `isolated format
|
||
repair failed: retained OpenCode session ses_…: context deadline exceeded`,
|
||
while others were judged `unsafe to repair`. Smallest change: give the
|
||
isolated format-repair session its own longer deadline (or a streaming
|
||
repair), so a repairable-but-slow result is recovered instead of failing the
|
||
run. Expected effect: recover part of the timeout-repair runs; the "unsafe to
|
||
repair" set remains a separate problem. Risk: longer tail latency on
|
||
already-slow runs. Locus: server code (`internal/stage`). Evidence: the quoted
|
||
diagnostics and the before/after drop (10.9 %→3.3 %) for this cluster.
|
||
|
||
3. **Cut repeat-visit cost in the review loop.**
|
||
Problem: repeat visits of the same stage cost 13.83 h of 82.97 h (16.7 %),
|
||
concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs
|
||
(31,70,96,97,127) failed outright with `Pipelines stopped reviewer after 3
|
||
visits on outcome rejected`. Smallest change: stop looping on advisory-only
|
||
rejections (no must-fix finding), or reduce the reviewer max-visits for
|
||
advisory findings. Expected effect: remove up to ~14 h of repeat work and the
|
||
5 exhausted-loop failures, if rejections are often non-blocking. Risk: a real
|
||
defect could pass. Locus: pipeline definition (`reviewer` control). Evidence:
|
||
the repeat/single-pass split in §2 and the `review_loop_exhausted` ids.
|
||
|
||
4. **Make the `implementer` blocked outcome actionable (measured 18/23 of the
|
||
blocked class).** Problem: `stage_outcome_blocked` is the single largest class
|
||
(23/86, 26.7 %), and it is dominated by **implementer (18)** — not
|
||
`ui_reviewer`, which is only 4 (and `review` 1). Smallest change: have the
|
||
implementer stage distinguish "environment/device unavailable" (retryable)
|
||
from "acceptance genuinely unmeetable" (terminal), so the retryable subset is
|
||
retried instead of failing the run. Expected effect: a portion of the 18
|
||
implementer-blocked runs recovered. Risk: retry storms. Locus: pipeline
|
||
definition / server code. Evidence: the 23 ids and the measured blocked-stage
|
||
split; the retryable/terminal split is **hypothesis** (not measured).
|
||
|
||
5. **Hand the `ui_reviewer` captured artifacts (measured 4/23 blocked).**
|
||
Problem: `ui_reviewer` blocked in 4 runs (146,167,207,210) and its p90 is
|
||
1030 s in Focused tweak; run 146's preview reports `artifacts/146/3 is
|
||
empty`. Smallest change: whatever captures screenshots for a UI run must hand
|
||
them to the `ui_reviewer` stage, and the stage should not block when no device
|
||
is attached. Expected effect: fewer blocked UI runs. Risk: silently skipping
|
||
visual verification. Locus: pipeline definition + orchestration (artifact
|
||
handoff). Evidence: run 146 preview and the 4 blocked ids; **partly
|
||
hypothesis** — the empty-artifact cause is quoted for one run only, not
|
||
measured across all 4.
|
||
|
||
6. **Harden the harness transport path.**
|
||
Problem: 18 non-success runs are transport failures — 10 `execution error`, 6
|
||
`context deadline exceeded`, 1 `final assistant readback had no completed
|
||
result`, 1 `OpenCode post … command failed`. Smallest change: add bounded
|
||
retry/backoff around the OpenCode `…/wait` call for the generic "agent
|
||
reported an execution error" case. Expected effect: recover the 10
|
||
execution-error runs if they are transient. Risk: retrying a genuinely broken
|
||
session. Locus: server code (harness adapter). Evidence: the quoted
|
||
diagnostics and ids. Whether the errors are transient is **hypothesis**.
|
||
|
||
7. **Stabilise merge resolution.**
|
||
Problem: merge-resolution pipelines contributed blocked implementers (176,
|
||
177, 178), `final_result_validation` (179, 183, 221), and resolver runs cost
|
||
1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver's
|
||
`implementer` blocks on a conflict workspace (e.g. missing conflict markers or
|
||
write-path scope). Locus: merge-resolution pipeline definition. Evidence:
|
||
`merger.log` lines and the run diagnostics; root cause **hypothesis**.
|
||
|
||
## 6. What could not be verified
|
||
|
||
- **server.log content.** It is a request/response log and contains none of the
|
||
stage diagnostics (0 matches for every diagnostic phrase above). It could not
|
||
be used to re-derive failures; the run records were used instead. To settle
|
||
whether transport errors correlate with request latency, the server would need
|
||
to log per-request duration (not currently present).
|
||
- **2 non-terminal runs excluded.** Runs 222 and 223 were `running` when the
|
||
history and steps were captured; their incomplete steps (223 seq2 `reviewer`,
|
||
222 seq4 `implementer`) have an empty `finished_at` and were excluded from all
|
||
distributions. Settle by re-reading `pipelines inspect` after they finish.
|
||
(An earlier capture also excluded run 220 seq3 `ui_reviewer` when it was still
|
||
running; it had finished by the later capture and is included above.)
|
||
- **4 unresolved busy-wait episodes.** Four `workspace busy` sequences were still
|
||
open at log end (no matching `launched` line), so their final wait is unknown.
|
||
(An earlier working note reported 5 unresolved episodes; the re-counted
|
||
population gives 4.) Settle by re-parsing the lane logs once the lanes go idle.
|
||
- **Blocked-class cause distribution.** Only run 146 has a quoted cause; the
|
||
other blocked runs were classified by `outcome=blocked` plus stage name. A
|
||
per-run `result_preview` pass would quantify the retryable vs terminal split.
|
||
- **After-side sample size.** The 910a7aa comparison rests on 30 runs (2 still
|
||
running). A larger after-window sample is required before attributing the
|
||
failure-rate drop to the change rather than to pipeline mix.
|
||
- **Device-lock waits.** No `device lock` event string exists in the lane logs or
|
||
`server.log`; emulator trouble appears only as `recovering the emulator`
|
||
retries. A device-lock measurement would need the orchestration to log an
|
||
explicit acquire/wait/release.
|
||
- **No system-changing measurements were attempted.** No run was started,
|
||
retried, cancelled or reconfigured; the server, ntfy service, lanes and devices
|
||
were not touched, and the phone/emulator was not used. Measurements that would
|
||
require such changes (e.g. injecting a busy workspace to time a wait, or
|
||
forcing a transport error to test retry) were deliberately not performed.
|
||
|
||
## Verification and limits
|
||
|
||
Commands and observed results (all run in this session):
|
||
|
||
- `pipelines ping --json` → build/commit `910a7aa…`, `dirty false`.
|
||
- `pipelines runs --json` (2 calls, at different capture times) → 223 runs; the
|
||
two run-status counts reconcile with `pipelines stats --json`
|
||
(`total 223, succeeded 134/135, failed 73, canceled 11, interrupted 2, active
|
||
3/2`).
|
||
- `pipelines inspect <id> --json` → 223 calls (one per run id), used for all step
|
||
intervals and the taxonomy fallback.
|
||
- `server.log` bounded searches: 910 `is already active in workspace`
|
||
rejections; 0 diagnostic strings.
|
||
- Lane/orchestration logs (streaming `grep`/`awk`): 897 `workspace busy` lines,
|
||
800 restart holds, 11 `merge deferred`, 12 resolver durations, 50/35 emulator
|
||
retries.
|
||
- Go verification, run with the repository's normal invocation from the module
|
||
root:
|
||
- `go build ./...` → exit 0, no output.
|
||
- `go test ./...` → exit 0; 20 packages `ok`, 3 `cmd/*` and `internal/wstest`
|
||
have no test files. No Go source was changed; these confirm the worktree was
|
||
left buildable.
|
||
|
||
Limits: the after-side sample is small; several classes did not recur after the
|
||
boundary for reasons the data cannot separate from the format change; the
|
||
blocked-class cause distribution and the transience of transport errors are
|
||
hypotheses, not measurements. Raw captures are under
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and
|
||
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).
|
||
|
||
## 7. Transport and repair recovery (source-level, 2026-10-17)
|
||
|
||
Read-only pass over the §1 diagnostics after the harness-transport and
|
||
format-repair change. No run was started, retried or cancelled; no server, ntfy,
|
||
lane or device was restarted or redeployed; no provider inference was performed.
|
||
Each shape is mapped to the code path that produces it. The per-run
|
||
pre-admission vs post-admission and read-only vs mutating split was **not**
|
||
re-inspected with `pipelines inspect`, so every per-run row is labelled *not
|
||
verified* and the counts are not summed into a "recovered" total.
|
||
|
||
| failure shape (§1 class, quoted diagnostic) | run ids | code path | can recover | remains unrecoverable |
|
||
|---|---|---|---|---|
|
||
| execution error at readback (`OpenCode agent reported an execution error`) | 73,74,75,77,80,149,150,152,195,202 | post-admission `completedAgentResult` / `opencode.ExecutionError` | a stage that is explicit read-only may retry within `retries`(*not verified per run*) | a mutating/legacy stage never replays admitted work; a non-transient provider error stays terminal |
|
||
| `context deadline exceeded` | 33,36,38,59,101,138 | attempt-boundary deadline, or a transport call that expired | a deadline during an in-flight request now keeps the transport cause and, when pre-admission, retries for any stage; a read-only stage retries a post-admission one | a genuine stage overrun on a mutating/legacy stage stays terminal; whether each run was a hung call or a real overrun is *not verified* |
|
||
| readback failure (`final assistant readback had no completed result`) | 161 | post-admission `completedAgentResult` | explicit read-only stage may retry | mutating/legacy stage never replays |
|
||
| native V1 `/wait` command failed (`CLI exit code 1`) | 4 | post-admission `openCodeError` on `/wait` | explicit read-only stage may retry | mutating/legacy stage never replays |
|
||
| repair timeout (`isolated format repair failed … context deadline exceeded`) | 55,81,174,179,183,221 | `repairDeadline` (payload bytes + stage timeout, floored, capped, clamped to remaining − margin) | a slow-but-repairable payload now gets a longer bounded budget | run 55's payload is the ambiguous unescaped-quote/colon class (byte offset 7837) → unrecoverable; any payload past the 32 KiB bound stays unrecoverable |
|
||
|
||
Mapping notes: the execution-error and readback shapes are read back *after* the
|
||
prompt was admitted, so they are post-admission transport failures and the
|
||
no-tool-replay rule keeps them terminal on a mutating stage. The
|
||
`session_transport_command_failed` shape (run 4) is the native V1 `/wait` call
|
||
issued after the prompt, not a session-create reach failure. The repair-timeout
|
||
shapes are `FINAL RESULT VALIDATION` (repair-stage), not transport. The new code
|
||
also reports an unknown provider execution-error name as a bounded sanitized
|
||
identifier plus any HTTP status, so future occurrences of runs 73–202 are more
|
||
precisely diagnostic even when they are not retried.
|