Files
pipelines/docs/pipeline-performance-review.md
T
alan 7b72b8f20f stage: retry pre-admission harness reach failures and widen repair deadline
Treat a failure to reach the harness before any prompt or tool call is
admitted (connecting or creating the session) as a distinct retryable
reach failure, so a transport blip at the first stage no longer ends a
mutating run; keep post-admission readback/poll/agent failures retried
only on an explicit read-only stage and never replay admitted tool work.
Preserve the true cause when the context is already done (join ctx.Err
with the typed transport error) and report a sanitized unknown provider
error name/status instead of a blanket summary.

Make the isolated format-repair deadline proportional to the stage
timeout (floored, capped and clamped to remaining minus a margin) so a
slow-but-repairable result is recovered without ever extending the stage
deadline; the bounded ladder, value guard, deny-all formatter session,
cumulative usage accounting and no-tool-replay rule are unchanged.

Update docs/model-results.md, docs/pipeline-performance-review.md and the
packaged skill references; add stage, opencode and server fixtures.
2026-10-17 14:51:09 +01:00

459 lines
28 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Pipeline performance and reliability review
Read-only analysis of the live Pipelines run history, step records and
orchestration logs, on 2026-10-17. The only change is this document.
## Sources and method
All numbers come from commands run in this session against the running server
(`pipelines ping --json` reports `build.version
v0.0.0-20261016212730-910a7aa798af`, `commit
910a7aa798af3eecff72fcde28b9eafde470286c`, `dirty false`).
- `pipelines runs --json` was captured twice: once into
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json`
(223 entries, 201939 bytes) and once into
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/runs.json`
(223 entries). Both captures were reused for the taxonomy, trend and
concurrency.
- `pipelines stats --json` was captured twice:
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json` reported
`total 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3`,
and `/Users/alan/.local/share/pipelines/data/artifacts/222/4/stats.json`
reported `total 223, succeeded 135, failed 73, canceled 11, interrupted 2,
active 2`. Both inventories reconcile with the run totals
(`134+73+11+2+3 = 223` and `135+73+11+2+2 = 223`).
- `pipelines inspect <id> --json` was called exactly once per run id (223 calls,
saved under `artifacts/222/4/inspect/`); step intervals were aggregated in one
pass from those files.
- Orchestration logs: `/tmp/lane{A,B,C,D,E,Go,T2,Chat}.log`, `/tmp/merger.log`,
`/tmp/deploy-server.log`, and the `lane-{hg,hs}.log` files in
`/private/var/folders/jp/q0qvjyg54fl_c5vhhk1ylz2m0000gn/T/opencode/`, plus
`~/.local/share/pipelines/logs/server.log` (16,282,789 bytes, 143,788 lines).
Quantile convention: **nearest-rank**, i.e. the p-quantile is element
`ceil(p*n)` (1-based) of the sorted list; the median is the p=0.5 element.
Durations are `finished_at - started_at` parsed with RFC3339/RFC3339Nano rules
(`Z` normalised, empty string = absent), as in `internal/pipeline/search.go`.
## 1. Failure taxonomy over the whole history
The inventory holds **223 runs** (`running 2`, `succeeded 135`, `failed 73`,
`canceled 11`, `interrupted 2`). **86 runs have a terminal non-success status**
(`failed 73`, `canceled 11`, `interrupted 2`). Every one is classified below from
the run `diagnostic`, or — where that is empty (runs 30, 32, 45) — from the
failing step's `outcome` in its `pipelines inspect` record. The class counts sum
to 86; there is no unknown bucket.
| class | count | run ids |
|---|---|---|
| `stage_outcome_blocked` | 23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 |
| `final_result_validation` | 19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 |
| `harness_transport_execution_error` | 10 | 73,74,75,77,80,149,150,152,195,202 |
| `harness_transport_timeout` | 6 | 33,36,38,59,101,138 |
| `cancelled_by_operator` | 11 | 28,39,57,63,78,108,113,114,191,211,219 |
| `review_loop_exhausted` | 5 | 31,70,96,97,127 |
| `stage_outcome_error` | 4 | 30,137,139,190 |
| `interrupted_by_deploy` | 2 | 13,24 |
| `invalid_agent_messages` | 2 | 2,5 |
| `harness_transport_readback` | 1 | 161 |
| `agent_quiescence_unconfirmed` | 1 | 43 |
| `session_transport_command_failed` | 1 | 4 |
| `duplicate_json_field` | 1 | 3 |
Classifying diagnostics (quoted verbatim from the run records):
- **stage_outcome_blocked** — a stage ends with `outcome=blocked`. The split by
blocked stage is **implementer 18, ui_reviewer 4, review 1**. Run 105:
`stage review returned outcome "blocked"`; runs 32/45 have an empty run
`diagnostic` and an `implementer` final step `outcome='blocked'`; the rest
carry `stage implementer returned outcome "blocked"` or `stage ui_reviewer
returned outcome "blocked"`. Run 146 shows a deliberate stop: *"No
screenshots/frames exist for this run (artifacts/146/1 contains only the APK;
artifacts/146/3 is empty), and the task's device rules are mutually
exclusive"*.
- **final_result_validation** — malformed/schema-rejected model result. Run 8:
`model stage implementer: FINAL RESULT VALIDATION: invalid model stage result:
invalid character 'I' looking for beginning of value at byte 1; unsafe to
repair incomplete values or a non-object response; completed edits and tool
side effects are retained and not undone; a full retry repeats work`. Run 221:
`model stage reviewer: FINAL RESULT VALIDATION: invalid model stage result:
invalid character '}' after array element at byte 2929; isolated format repair
failed: retained OpenCode session ses_eb8c5ab8affeAeLm9sFdG8uimz: context
deadline exceeded; …`. Two sub-shapes were observed:
`unsafe to repair incomplete values or a non-object response`
(runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) and
`isolated format repair failed: retained OpenCode session ses_…: context
deadline exceeded` (runs 55,81,174,179,183,221 — 6 runs). Runs 85/142 add a
third: `unknown result field`; run 170's diagnostic is `invalid character '›'
after object key:value pair at byte 780`.
- **harness_transport_execution_error** — e.g. run 73: `model stage ui_reviewer:
HARNESS TRANSPORT: harness transport failure: OpenCode agent reported an
execution error`.
- **harness_transport_timeout** — run 33: `model stage implementer: HARNESS
TRANSPORT: context deadline exceeded\ncontext deadline exceeded`.
- **cancelled_by_operator** — `context canceled`, status `canceled`.
- **review_loop_exhausted** — run 31: `Pipelines stopped reviewer after 3 visits
on outcome rejected`.
- **stage_outcome_error** — run 190: `stage implementer returned outcome
"error"`; runs 30/137/139 have a final step `outcome='error'`.
- **interrupted_by_deploy** — run 13/24: `Server stopped; run was not resumed`,
status `interrupted`.
- **invalid_agent_messages** — run 2/5: `model stage planner: invalid agent
messages`.
- **harness_transport_readback** — run 161: `harness transport failure: final
assistant readback had no completed result`.
- **agent_quiescence_unconfirmed** — run 43: `agent quiescence not confirmed;
inspect retained session before further work`.
- **session_transport_command_failed** — run 4: `model stage implementer:
OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait
command failed: CLI exit code 1`.
- **duplicate_json_field** — run 3: `invalid model stage result: duplicate JSON
field "performance_review"`.
`server.log` contains none of these diagnostic strings (bounded search:
`FINAL RESULT VALIDATION` 0, `HARNESS TRANSPORT` 0, `invalid agent messages` 0,
`stopped reviewer after` 0, `quiescence` 0); it is a request/response log
(`command received`/`response sent`, plus `summarise: harness=opencode …`).
Diagnostics live in the run records, not the server log.
## 2. Stage cost
The 223 runs produced **828 steps**; **826 have both `started_at` and
`finished_at`** and were used. The 2 without a `finished_at` belong to the 2
still-running runs and were excluded, not fabricated: run 223 seq2 `reviewer`
and run 222 seq4 `implementer`. By run: 221 runs have a non-empty `finished_at`
and all 223 runs contribute at least one complete step. This is the population
at the 2026-10-17 capture, when run 220 had already finished — an earlier working
note that called the population "222 finished" was stale and is corrected here.
Total wall-hours by stage key (sum of `finished_at - started_at`), with the share
spent on repeat visits (`control.visit > 1`, i.e. review/retry loops) split out:
| stage key | total (h) | single-pass (h) | repeat (h) | steps |
|---|---|---|---|---|
| implementer | 41.36 | 33.73 | 7.62 | 270 |
| reviewer | 20.72 | 15.76 | 4.96 | 220 |
| ui_reviewer | 15.18 | 13.93 | 1.25 | 135 |
| planner | 3.55 | 3.55 | 0.00 | 79 |
| committer | 2.00 | 2.00 | 0.00 | 99 |
| worker/research/review/probe/report/opencode/pi | 0.16 | 0.16 | 0.00 | 23 |
The dominant stage keys are **implementer (41.36 h), reviewer (20.72 h) and
ui_reviewer (15.18 h)** — together 77.26 of 82.97 h. Repeat visits of the same
stage cost **13.83 h (16.7 %)**, almost all in implementer (7.62 h) and reviewer
(4.96 h).
Per (pipeline, stage key) distribution (count / median / p90 / max in seconds /
total hours), for the highest-volume keys across the three highest-volume
pipelines:
| pipeline | stage | n | median | p90 | max | total (h) |
|---|---|---|---|---|---|---|
| Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 |
| Pipelines maintenance - plan implement review | implementer | 90 | 370 | 847 | 3601 | 12.07 |
| Pipelines maintenance - plan implement review | reviewer | 79 | 245 | 503 | 12613 | 10.54 |
| Focused tweak - implement review commit | ui_reviewer | 81 | 293 | 1030 | 2158 | 10.05 |
| Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 |
| Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 |
| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 |
| Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 |
| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 |
| Pipelines quick - plan implement review | reviewer | 13 | 324 | 896 | 1800 | 1.69 |
| Focused tweak - implement review commit | committer | 63 | 31 | 180 | 644 | 1.13 |
Three pipelines are shown (Focused tweak, Pipelines maintenance, Pipelines
quick); the full per-pipeline table was produced by the aggregation command.
Notable long tails: one `reviewer` step in Pipelines maintenance ran 12613 s
(3.5 h); one Pipelines-maintenance `implementer` step hit 3601 s (≈ the 1 h
stage timeout), while the Focused-tweak `implementer` max is 1801 s and its p90
1414 s. The `ui_reviewer` p90 is high (1030 s in Focused tweak).
## 3. Reliability trend, split at 910a7aa
Boundary: `2026-10-16T21:27:00Z`, the deploy of commit 910a7aa. `pipelines ping
--json` confirms the running `build.commit` is 910a7aa. `deploy-server.log`
corroborates: `22:27:45 quiescing the lanes …` / `22:27:45 building from main
(910a7aa Merge lane-app2: …)` / `22:27:57 server healthy; deployed 910a7aa`
(its clock is local UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by
`created_at`: the last "before" run is 193 (`2026-10-16T21:21:33Z`), the first
"after" is 194 (`2026-10-16T21:28:00Z`).
| side | n | running | failed | failed/(failed+succeeded) | non-success/terminal |
|---|---|---|---|---|---|
| before | 193 | 0 | 68 | 37.4 % | 40.9 % |
| after | 30 | 2 | 5 | 19.2 % | 25.0 % |
Failure-class mix, before → after (counts):
| class | before | after |
|---|---|---|
| stage_outcome_blocked | 21 | 2 |
| final_result_validation | 18 | 1 |
| harness_transport_execution_error | 8 | 2 |
| harness_transport_timeout | 6 | 0 |
| cancelled_by_operator | 9 | 2 |
| review_loop_exhausted | 5 | 0 |
| stage_outcome_error | 4 | 0 |
| interrupted_by_deploy | 2 | 0 |
| invalid_agent_messages | 2 | 0 |
| harness_transport_readback | 1 | 0 |
| agent_quiescence_unconfirmed | 1 | 0 |
| session_transport_command_failed | 1 | 0 |
| duplicate_json_field | 1 | 0 |
Read literally the change correlates with fewer malformed-result failures:
`final_result_validation + duplicate_json_field + invalid_agent_messages` falls
from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is **confounded
and under-powered**: the after side is only 30 runs (2 still running), the
pipeline mix differs (after is mostly Focused tweak 15, Pipelines maintenance 7,
Merge-resolution 3 — several one-off template/docker/research pipelines appear
only before), and several before-side classes (transport timeout, review-loop,
deploy interruption) simply never recurred in the short after window rather than
being fixed by 910a7aa. The data is consistent with the format/repair change
helping; it does not establish causation. A larger after sample is needed.
## 4. Throughput limits (measured)
**Busy-workspace waits.** The lane logs record `workspace busy; waiting to
launch <file>` at a fixed 30 s cadence: **897 such lines**. Grouping them into
episodes (same lane+file, ending at the `launched <file>` line) gives **27
episodes, 23 resolved and 4 still open at log end**. For the 23 resolved
episodes the wait is `launch_time - first_busy`:
- n=23, median **992 s (16.5 min)**, p90 **2293 s (38.2 min)**, max
**9070 s (2.5 h)**, total **9.07 h**. This independent reimplementation of the
rule includes the laneGo `malformed-result-brief.txt` episode (2170 s); an
earlier working note that dropped it reported n=22 / 8.47 h and is corrected
here.
Longest episodes: laneA `s02-tui-omni-search.txt` waited 13:12:33 → 15:43:43
(9070 s) before `15:43:43 launched s02-tui-omni-search.txt as run 137`; laneGo
`s02-tui-omni-search.txt` waited 15:47:22 → 16:48:28 (3666 s).
**Server-side rejection of a second run in a busy workspace.** `server.log`
records **910** `error="run <id> is already active in workspace <path>; wait for
it to finish or start this run in a different workspace …"` responses, from
**29 distinct blocking runs**, concentrated on 2026/10/16 (905) vs 2026/10/15
(5). By workspace: `lane-go` 387, `/Users/alan/repos/pipelines` 307, `lane-app2`
166, `lane-c` 49, `ws-c` 1. So the dominant limit is one-run-per-workspace, and
the lanes retry it hundreds of times.
**Server-restart holds.** `holding: another lane is restarting the server`
appears **800** times; grouping holds per lane while consecutive hold lines are
≤60 s apart (the 30 s poll cadence plus jitter; a wider gap starts a new
episode) gives 29 episodes totalling **6.42 h**, median 480 s, max 3572 s. (A
strict consecutive-line regroup, which splits an episode on any interleaved
line, gives 32 episodes; the 29 figure is the gap-based grouping.) `building
server and cli from main` 13 times; `server redeployed and healthy` 10 times.
**Merge/main-lane waits.** `merge deferred: main lane busy` appears **11** times;
`merge requested for …` 24 times; `MERGE CONFLICT` 15 times; `SYNC CONFLICT: lane
could not take main's latest; continuing on the older base` 45 times.
**Merge/resolver activity.** `merger.log` shows the queue draining through
pipeline resolvers. 12 resolver runs have measurable durations, total
**5153 s (1.43 h)**, median 421 s, max 1062 s: `run 176 20 s` and `177 20 s`
("completed the merge" immediately) versus `run 180 661 s`, `182 602 s`,
`183 1062 s` (handed to a pipeline). Four events record `could not launch a
resolver; aborting the merge for <lane>`; three resolvers `did NOT land` their
lane (`179 lane-app2`, `183 lane-go`, `221 lane-c`). The `merge-queue` marker in
the worktree currently holds `lane-go` and `lane-app2`.
**Deploy waits.** `deploy-server.log` shows one deploy waiting for quiescence:
`13:39:07 quiescing the lanes so no new run starts` with `waiting: N run(s)
active` every ~20 s from 3 down to 1, until `14:18:18 building from main
(dce624b …)` — a **2351 s (39.2 min)** wait. The 910a7aa deploy at `22:27:45`
waited 12 s (0 active runs).
**Device/emulator churn.** `FAILED once: <file> - recovering the emulator and
retrying` appears 50 times and `FAILED twice: <file> - recording and continuing`
35 times (e.g. `02:53:47 [laneA] FAILED once: q03-pane-opaque-A.txt - recovering
the emulator and retrying`). No `device lock` event string was found in the lane
logs or `server.log`.
**Real concurrency.** Overlapping step intervals show a maximum of **7
simultaneous executing steps**, at `2026-10-16T14:47:35Z`, spanning the 6 runs
[134, 136, 137, 138, 139, 140] (one run had two steps overlapping at that
instant). Step intervals are counted as **closed** (`started_at ≤ t ≤
finished_at`), so run 134's `reviewer` (ending 14:47:35Z) and its `implementer`
(starting 14:47:35Z) both count at the touching boundary; with half-open
intervals (`started_at ≤ t < finished_at`) the maximum is **6 steps at
14:47:22Z**, the same six runs. Using run wall-spans (`created_at`..`finished_at`) the maximum is **6
concurrent runs** at `2026-10-16T14:47:22Z`, the same six runs. 157 runs had at
least one step interval overlapping another run. So despite many queued runs,
the observed executing concurrency never exceeded 7 steps / 6 runs — consistent
with the lane count and the 897 workspace-busy lines, and not with a
memory/harness-capacity limit.
## 5. Ranked improvements
Ranked by measured impact. Items marked **hypothesis** have no direct
measurement behind them.
1. **Remove the one-run-per-workspace contention (measured 9.07 h waiting + 910
server rejections + 6.42 h restart holds).** Problem: 23 resolved bus waits
total 9.07 h (median 992 s, max 9070 s); the server rejected a second run in a
busy workspace 910 times across 29 blocking runs; restart holds cost 6.42 h.
Smallest change: give each lane its own worktree/workspace so a lane is never
blocked by a run already active in it (or lower the fixed 30 s busy-poll).
Expected effect: cut the multi-minute launch latency and the wasted poll
lines. Risk: more concurrent worktrees consume more disk and parallel model
capacity. Locus: orchestration. Evidence: the busy/rejection counts and the
per-workspace breakdown above.
2. **Finish or budget the malformed-result repair path.**
Problem: `final_result_validation` is 19 of 86 non-success runs; in several
(e.g. 55,81,174,179,183,221) the repair itself failed with `isolated format
repair failed: retained OpenCode session ses_…: context deadline exceeded`,
while others were judged `unsafe to repair`. Smallest change: give the
isolated format-repair session its own longer deadline (or a streaming
repair), so a repairable-but-slow result is recovered instead of failing the
run. Expected effect: recover part of the timeout-repair runs; the "unsafe to
repair" set remains a separate problem. Risk: longer tail latency on
already-slow runs. Locus: server code (`internal/stage`). Evidence: the quoted
diagnostics and the before/after drop (10.9 %→3.3 %) for this cluster.
3. **Cut repeat-visit cost in the review loop.**
Problem: repeat visits of the same stage cost 13.83 h of 82.97 h (16.7 %),
concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs
(31,70,96,97,127) failed outright with `Pipelines stopped reviewer after 3
visits on outcome rejected`. Smallest change: stop looping on advisory-only
rejections (no must-fix finding), or reduce the reviewer max-visits for
advisory findings. Expected effect: remove up to ~14 h of repeat work and the
5 exhausted-loop failures, if rejections are often non-blocking. Risk: a real
defect could pass. Locus: pipeline definition (`reviewer` control). Evidence:
the repeat/single-pass split in §2 and the `review_loop_exhausted` ids.
4. **Make the `implementer` blocked outcome actionable (measured 18/23 of the
blocked class).** Problem: `stage_outcome_blocked` is the single largest class
(23/86, 26.7 %), and it is dominated by **implementer (18)** — not
`ui_reviewer`, which is only 4 (and `review` 1). Smallest change: have the
implementer stage distinguish "environment/device unavailable" (retryable)
from "acceptance genuinely unmeetable" (terminal), so the retryable subset is
retried instead of failing the run. Expected effect: a portion of the 18
implementer-blocked runs recovered. Risk: retry storms. Locus: pipeline
definition / server code. Evidence: the 23 ids and the measured blocked-stage
split; the retryable/terminal split is **hypothesis** (not measured).
5. **Hand the `ui_reviewer` captured artifacts (measured 4/23 blocked).**
Problem: `ui_reviewer` blocked in 4 runs (146,167,207,210) and its p90 is
1030 s in Focused tweak; run 146's preview reports `artifacts/146/3 is
empty`. Smallest change: whatever captures screenshots for a UI run must hand
them to the `ui_reviewer` stage, and the stage should not block when no device
is attached. Expected effect: fewer blocked UI runs. Risk: silently skipping
visual verification. Locus: pipeline definition + orchestration (artifact
handoff). Evidence: run 146 preview and the 4 blocked ids; **partly
hypothesis** — the empty-artifact cause is quoted for one run only, not
measured across all 4.
6. **Harden the harness transport path.**
Problem: 18 non-success runs are transport failures — 10 `execution error`, 6
`context deadline exceeded`, 1 `final assistant readback had no completed
result`, 1 `OpenCode post … command failed`. Smallest change: add bounded
retry/backoff around the OpenCode `…/wait` call for the generic "agent
reported an execution error" case. Expected effect: recover the 10
execution-error runs if they are transient. Risk: retrying a genuinely broken
session. Locus: server code (harness adapter). Evidence: the quoted
diagnostics and ids. Whether the errors are transient is **hypothesis**.
7. **Stabilise merge resolution.**
Problem: merge-resolution pipelines contributed blocked implementers (176,
177, 178), `final_result_validation` (179, 183, 221), and resolver runs cost
1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver's
`implementer` blocks on a conflict workspace (e.g. missing conflict markers or
write-path scope). Locus: merge-resolution pipeline definition. Evidence:
`merger.log` lines and the run diagnostics; root cause **hypothesis**.
## 6. What could not be verified
- **server.log content.** It is a request/response log and contains none of the
stage diagnostics (0 matches for every diagnostic phrase above). It could not
be used to re-derive failures; the run records were used instead. To settle
whether transport errors correlate with request latency, the server would need
to log per-request duration (not currently present).
- **2 non-terminal runs excluded.** Runs 222 and 223 were `running` when the
history and steps were captured; their incomplete steps (223 seq2 `reviewer`,
222 seq4 `implementer`) have an empty `finished_at` and were excluded from all
distributions. Settle by re-reading `pipelines inspect` after they finish.
(An earlier capture also excluded run 220 seq3 `ui_reviewer` when it was still
running; it had finished by the later capture and is included above.)
- **4 unresolved busy-wait episodes.** Four `workspace busy` sequences were still
open at log end (no matching `launched` line), so their final wait is unknown.
(An earlier working note reported 5 unresolved episodes; the re-counted
population gives 4.) Settle by re-parsing the lane logs once the lanes go idle.
- **Blocked-class cause distribution.** Only run 146 has a quoted cause; the
other blocked runs were classified by `outcome=blocked` plus stage name. A
per-run `result_preview` pass would quantify the retryable vs terminal split.
- **After-side sample size.** The 910a7aa comparison rests on 30 runs (2 still
running). A larger after-window sample is required before attributing the
failure-rate drop to the change rather than to pipeline mix.
- **Device-lock waits.** No `device lock` event string exists in the lane logs or
`server.log`; emulator trouble appears only as `recovering the emulator`
retries. A device-lock measurement would need the orchestration to log an
explicit acquire/wait/release.
- **No system-changing measurements were attempted.** No run was started,
retried, cancelled or reconfigured; the server, ntfy service, lanes and devices
were not touched, and the phone/emulator was not used. Measurements that would
require such changes (e.g. injecting a busy workspace to time a wait, or
forcing a transport error to test retry) were deliberately not performed.
## Verification and limits
Commands and observed results (all run in this session):
- `pipelines ping --json` → build/commit `910a7aa…`, `dirty false`.
- `pipelines runs --json` (2 calls, at different capture times) → 223 runs; the
two run-status counts reconcile with `pipelines stats --json`
(`total 223, succeeded 134/135, failed 73, canceled 11, interrupted 2, active
3/2`).
- `pipelines inspect <id> --json` → 223 calls (one per run id), used for all step
intervals and the taxonomy fallback.
- `server.log` bounded searches: 910 `is already active in workspace`
rejections; 0 diagnostic strings.
- Lane/orchestration logs (streaming `grep`/`awk`): 897 `workspace busy` lines,
800 restart holds, 11 `merge deferred`, 12 resolver durations, 50/35 emulator
retries.
- Go verification, run with the repository's normal invocation from the module
root:
- `go build ./...` → exit 0, no output.
- `go test ./...` → exit 0; 20 packages `ok`, 3 `cmd/*` and `internal/wstest`
have no test files. No Go source was changed; these confirm the worktree was
left buildable.
Limits: the after-side sample is small; several classes did not recur after the
boundary for reasons the data cannot separate from the format change; the
blocked-class cause distribution and the transience of transport errors are
hypotheses, not measurements. Raw captures are under
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).
## 7. Transport and repair recovery (source-level, 2026-10-17)
Read-only pass over the §1 diagnostics after the harness-transport and
format-repair change. No run was started, retried or cancelled; no server, ntfy,
lane or device was restarted or redeployed; no provider inference was performed.
Each shape is mapped to the code path that produces it. The per-run
pre-admission vs post-admission and read-only vs mutating split was **not**
re-inspected with `pipelines inspect`, so every per-run row is labelled *not
verified* and the counts are not summed into a "recovered" total.
| failure shape (§1 class, quoted diagnostic) | run ids | code path | can recover | remains unrecoverable |
|---|---|---|---|---|
| execution error at readback (`OpenCode agent reported an execution error`) | 73,74,75,77,80,149,150,152,195,202 | post-admission `completedAgentResult` / `opencode.ExecutionError` | a stage that is explicit read-only may retry within `retries`(*not verified per run*) | a mutating/legacy stage never replays admitted work; a non-transient provider error stays terminal |
| `context deadline exceeded` | 33,36,38,59,101,138 | attempt-boundary deadline, or a transport call that expired | a deadline during an in-flight request now keeps the transport cause and, when pre-admission, retries for any stage; a read-only stage retries a post-admission one | a genuine stage overrun on a mutating/legacy stage stays terminal; whether each run was a hung call or a real overrun is *not verified* |
| readback failure (`final assistant readback had no completed result`) | 161 | post-admission `completedAgentResult` | explicit read-only stage may retry | mutating/legacy stage never replays |
| native V1 `/wait` command failed (`CLI exit code 1`) | 4 | post-admission `openCodeError` on `/wait` | explicit read-only stage may retry | mutating/legacy stage never replays |
| repair timeout (`isolated format repair failed … context deadline exceeded`) | 55,81,174,179,183,221 | `repairDeadline` (payload bytes + stage timeout, floored, capped, clamped to remaining − margin) | a slow-but-repairable payload now gets a longer bounded budget | run 55's payload is the ambiguous unescaped-quote/colon class (byte offset 7837) → unrecoverable; any payload past the 32 KiB bound stays unrecoverable |
Mapping notes: the execution-error and readback shapes are read back *after* the
prompt was admitted, so they are post-admission transport failures and the
no-tool-replay rule keeps them terminal on a mutating stage. The
`session_transport_command_failed` shape (run 4) is the native V1 `/wait` call
issued after the prompt, not a session-create reach failure. The repair-timeout
shapes are `FINAL RESULT VALIDATION` (repair-stage), not transport. The new code
also reports an unknown provider execution-error name as a bounded sanitized
identifier plus any HTTP status, so future occurrences of runs 73–202 are more
precisely diagnostic even when they are not retried.