Treat a failure to reach the harness before any prompt or tool call is admitted (connecting or creating the session) as a distinct retryable reach failure, so a transport blip at the first stage no longer ends a mutating run; keep post-admission readback/poll/agent failures retried only on an explicit read-only stage and never replay admitted tool work. Preserve the true cause when the context is already done (join ctx.Err with the typed transport error) and report a sanitized unknown provider error name/status instead of a blanket summary. Make the isolated format-repair deadline proportional to the stage timeout (floored, capped and clamped to remaining minus a margin) so a slow-but-repairable result is recovered without ever extending the stage deadline; the bounded ladder, value guard, deny-all formatter session, cumulative usage accounting and no-tool-replay rule are unchanged. Update docs/model-results.md, docs/pipeline-performance-review.md and the packaged skill references; add stage, opencode and server fixtures.
28 KiB
Pipeline performance and reliability review
Read-only analysis of the live Pipelines run history, step records and orchestration logs, on 2026-10-17. The only change is this document.
Sources and method
All numbers come from commands run in this session against the running server
(pipelines ping --json reports build.version v0.0.0-20261016212730-910a7aa798af, commit 910a7aa798af3eecff72fcde28b9eafde470286c, dirty false).
pipelines runs --jsonwas captured twice: once into/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json(223 entries, 201939 bytes) and once into/Users/alan/.local/share/pipelines/data/artifacts/222/4/runs.json(223 entries). Both captures were reused for the taxonomy, trend and concurrency.pipelines stats --jsonwas captured twice:/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.jsonreportedtotal 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3, and/Users/alan/.local/share/pipelines/data/artifacts/222/4/stats.jsonreportedtotal 223, succeeded 135, failed 73, canceled 11, interrupted 2, active 2. Both inventories reconcile with the run totals (134+73+11+2+3 = 223and135+73+11+2+2 = 223).pipelines inspect <id> --jsonwas called exactly once per run id (223 calls, saved underartifacts/222/4/inspect/); step intervals were aggregated in one pass from those files.- Orchestration logs:
/tmp/lane{A,B,C,D,E,Go,T2,Chat}.log,/tmp/merger.log,/tmp/deploy-server.log, and thelane-{hg,hs}.logfiles in/private/var/folders/jp/q0qvjyg54fl_c5vhhk1ylz2m0000gn/T/opencode/, plus~/.local/share/pipelines/logs/server.log(16,282,789 bytes, 143,788 lines).
Quantile convention: nearest-rank, i.e. the p-quantile is element
ceil(p*n) (1-based) of the sorted list; the median is the p=0.5 element.
Durations are finished_at - started_at parsed with RFC3339/RFC3339Nano rules
(Z normalised, empty string = absent), as in internal/pipeline/search.go.
1. Failure taxonomy over the whole history
The inventory holds 223 runs (running 2, succeeded 135, failed 73,
canceled 11, interrupted 2). 86 runs have a terminal non-success status
(failed 73, canceled 11, interrupted 2). Every one is classified below from
the run diagnostic, or — where that is empty (runs 30, 32, 45) — from the
failing step's outcome in its pipelines inspect record. The class counts sum
to 86; there is no unknown bucket.
| class | count | run ids |
|---|---|---|
stage_outcome_blocked |
23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 |
final_result_validation |
19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 |
harness_transport_execution_error |
10 | 73,74,75,77,80,149,150,152,195,202 |
harness_transport_timeout |
6 | 33,36,38,59,101,138 |
cancelled_by_operator |
11 | 28,39,57,63,78,108,113,114,191,211,219 |
review_loop_exhausted |
5 | 31,70,96,97,127 |
stage_outcome_error |
4 | 30,137,139,190 |
interrupted_by_deploy |
2 | 13,24 |
invalid_agent_messages |
2 | 2,5 |
harness_transport_readback |
1 | 161 |
agent_quiescence_unconfirmed |
1 | 43 |
session_transport_command_failed |
1 | 4 |
duplicate_json_field |
1 | 3 |
Classifying diagnostics (quoted verbatim from the run records):
- stage_outcome_blocked — a stage ends with
outcome=blocked. The split by blocked stage is implementer 18, ui_reviewer 4, review 1. Run 105:stage review returned outcome "blocked"; runs 32/45 have an empty rundiagnosticand animplementerfinal stepoutcome='blocked'; the rest carrystage implementer returned outcome "blocked"orstage ui_reviewer returned outcome "blocked". Run 146 shows a deliberate stop: "No screenshots/frames exist for this run (artifacts/146/1 contains only the APK; artifacts/146/3 is empty), and the task's device rules are mutually exclusive". - final_result_validation — malformed/schema-rejected model result. Run 8:
model stage implementer: FINAL RESULT VALIDATION: invalid model stage result: invalid character 'I' looking for beginning of value at byte 1; unsafe to repair incomplete values or a non-object response; completed edits and tool side effects are retained and not undone; a full retry repeats work. Run 221:model stage reviewer: FINAL RESULT VALIDATION: invalid model stage result: invalid character '}' after array element at byte 2929; isolated format repair failed: retained OpenCode session ses_eb8c5ab8affeAeLm9sFdG8uimz: context deadline exceeded; …. Two sub-shapes were observed:unsafe to repair incomplete values or a non-object response(runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) andisolated format repair failed: retained OpenCode session ses_…: context deadline exceeded(runs 55,81,174,179,183,221 — 6 runs). Runs 85/142 add a third:unknown result field; run 170's diagnostic isinvalid character '›' after object key:value pair at byte 780. - harness_transport_execution_error — e.g. run 73:
model stage ui_reviewer: HARNESS TRANSPORT: harness transport failure: OpenCode agent reported an execution error. - harness_transport_timeout — run 33:
model stage implementer: HARNESS TRANSPORT: context deadline exceeded\ncontext deadline exceeded. - cancelled_by_operator —
context canceled, statuscanceled. - review_loop_exhausted — run 31:
Pipelines stopped reviewer after 3 visits on outcome rejected. - stage_outcome_error — run 190:
stage implementer returned outcome "error"; runs 30/137/139 have a final stepoutcome='error'. - interrupted_by_deploy — run 13/24:
Server stopped; run was not resumed, statusinterrupted. - invalid_agent_messages — run 2/5:
model stage planner: invalid agent messages. - harness_transport_readback — run 161:
harness transport failure: final assistant readback had no completed result. - agent_quiescence_unconfirmed — run 43:
agent quiescence not confirmed; inspect retained session before further work. - session_transport_command_failed — run 4:
model stage implementer: OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait command failed: CLI exit code 1. - duplicate_json_field — run 3:
invalid model stage result: duplicate JSON field "performance_review".
server.log contains none of these diagnostic strings (bounded search:
FINAL RESULT VALIDATION 0, HARNESS TRANSPORT 0, invalid agent messages 0,
stopped reviewer after 0, quiescence 0); it is a request/response log
(command received/response sent, plus summarise: harness=opencode …).
Diagnostics live in the run records, not the server log.
2. Stage cost
The 223 runs produced 828 steps; 826 have both started_at and
finished_at and were used. The 2 without a finished_at belong to the 2
still-running runs and were excluded, not fabricated: run 223 seq2 reviewer
and run 222 seq4 implementer. By run: 221 runs have a non-empty finished_at
and all 223 runs contribute at least one complete step. This is the population
at the 2026-10-17 capture, when run 220 had already finished — an earlier working
note that called the population "222 finished" was stale and is corrected here.
Total wall-hours by stage key (sum of finished_at - started_at), with the share
spent on repeat visits (control.visit > 1, i.e. review/retry loops) split out:
| stage key | total (h) | single-pass (h) | repeat (h) | steps |
|---|---|---|---|---|
| implementer | 41.36 | 33.73 | 7.62 | 270 |
| reviewer | 20.72 | 15.76 | 4.96 | 220 |
| ui_reviewer | 15.18 | 13.93 | 1.25 | 135 |
| planner | 3.55 | 3.55 | 0.00 | 79 |
| committer | 2.00 | 2.00 | 0.00 | 99 |
| worker/research/review/probe/report/opencode/pi | 0.16 | 0.16 | 0.00 | 23 |
The dominant stage keys are implementer (41.36 h), reviewer (20.72 h) and ui_reviewer (15.18 h) — together 77.26 of 82.97 h. Repeat visits of the same stage cost 13.83 h (16.7 %), almost all in implementer (7.62 h) and reviewer (4.96 h).
Per (pipeline, stage key) distribution (count / median / p90 / max in seconds / total hours), for the highest-volume keys across the three highest-volume pipelines:
| pipeline | stage | n | median | p90 | max | total (h) |
|---|---|---|---|---|---|---|
| Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 |
| Pipelines maintenance - plan implement review | implementer | 90 | 370 | 847 | 3601 | 12.07 |
| Pipelines maintenance - plan implement review | reviewer | 79 | 245 | 503 | 12613 | 10.54 |
| Focused tweak - implement review commit | ui_reviewer | 81 | 293 | 1030 | 2158 | 10.05 |
| Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 |
| Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 |
| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 |
| Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 |
| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 |
| Pipelines quick - plan implement review | reviewer | 13 | 324 | 896 | 1800 | 1.69 |
| Focused tweak - implement review commit | committer | 63 | 31 | 180 | 644 | 1.13 |
Three pipelines are shown (Focused tweak, Pipelines maintenance, Pipelines quick); the full per-pipeline table was produced by the aggregation command.
Notable long tails: one reviewer step in Pipelines maintenance ran 12613 s
(3.5 h); one Pipelines-maintenance implementer step hit 3601 s (≈ the 1 h
stage timeout), while the Focused-tweak implementer max is 1801 s and its p90
1414 s. The ui_reviewer p90 is high (1030 s in Focused tweak).
3. Reliability trend, split at 910a7aa
Boundary: 2026-10-16T21:27:00Z, the deploy of commit 910a7aa. pipelines ping --json confirms the running build.commit is 910a7aa. deploy-server.log
corroborates: 22:27:45 quiescing the lanes … / 22:27:45 building from main (910a7aa Merge lane-app2: …) / 22:27:57 server healthy; deployed 910a7aa
(its clock is local UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by
created_at: the last "before" run is 193 (2026-10-16T21:21:33Z), the first
"after" is 194 (2026-10-16T21:28:00Z).
| side | n | running | failed | failed/(failed+succeeded) | non-success/terminal |
|---|---|---|---|---|---|
| before | 193 | 0 | 68 | 37.4 % | 40.9 % |
| after | 30 | 2 | 5 | 19.2 % | 25.0 % |
Failure-class mix, before → after (counts):
| class | before | after |
|---|---|---|
| stage_outcome_blocked | 21 | 2 |
| final_result_validation | 18 | 1 |
| harness_transport_execution_error | 8 | 2 |
| harness_transport_timeout | 6 | 0 |
| cancelled_by_operator | 9 | 2 |
| review_loop_exhausted | 5 | 0 |
| stage_outcome_error | 4 | 0 |
| interrupted_by_deploy | 2 | 0 |
| invalid_agent_messages | 2 | 0 |
| harness_transport_readback | 1 | 0 |
| agent_quiescence_unconfirmed | 1 | 0 |
| session_transport_command_failed | 1 | 0 |
| duplicate_json_field | 1 | 0 |
Read literally the change correlates with fewer malformed-result failures:
final_result_validation + duplicate_json_field + invalid_agent_messages falls
from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is confounded
and under-powered: the after side is only 30 runs (2 still running), the
pipeline mix differs (after is mostly Focused tweak 15, Pipelines maintenance 7,
Merge-resolution 3 — several one-off template/docker/research pipelines appear
only before), and several before-side classes (transport timeout, review-loop,
deploy interruption) simply never recurred in the short after window rather than
being fixed by 910a7aa. The data is consistent with the format/repair change
helping; it does not establish causation. A larger after sample is needed.
4. Throughput limits (measured)
Busy-workspace waits. The lane logs record workspace busy; waiting to launch <file> at a fixed 30 s cadence: 897 such lines. Grouping them into
episodes (same lane+file, ending at the launched <file> line) gives 27
episodes, 23 resolved and 4 still open at log end. For the 23 resolved
episodes the wait is launch_time - first_busy:
- n=23, median 992 s (16.5 min), p90 2293 s (38.2 min), max
9070 s (2.5 h), total 9.07 h. This independent reimplementation of the
rule includes the laneGo
malformed-result-brief.txtepisode (2170 s); an earlier working note that dropped it reported n=22 / 8.47 h and is corrected here.
Longest episodes: laneA s02-tui-omni-search.txt waited 13:12:33 → 15:43:43
(9070 s) before 15:43:43 launched s02-tui-omni-search.txt as run 137; laneGo
s02-tui-omni-search.txt waited 15:47:22 → 16:48:28 (3666 s).
Server-side rejection of a second run in a busy workspace. server.log
records 910 error="run <id> is already active in workspace <path>; wait for it to finish or start this run in a different workspace …" responses, from
29 distinct blocking runs, concentrated on 2026/10/16 (905) vs 2026/10/15
(5). By workspace: lane-go 387, /Users/alan/repos/pipelines 307, lane-app2
166, lane-c 49, ws-c 1. So the dominant limit is one-run-per-workspace, and
the lanes retry it hundreds of times.
Server-restart holds. holding: another lane is restarting the server
appears 800 times; grouping holds per lane while consecutive hold lines are
≤60 s apart (the 30 s poll cadence plus jitter; a wider gap starts a new
episode) gives 29 episodes totalling 6.42 h, median 480 s, max 3572 s. (A
strict consecutive-line regroup, which splits an episode on any interleaved
line, gives 32 episodes; the 29 figure is the gap-based grouping.) building server and cli from main 13 times; server redeployed and healthy 10 times.
Merge/main-lane waits. merge deferred: main lane busy appears 11 times;
merge requested for … 24 times; MERGE CONFLICT 15 times; SYNC CONFLICT: lane could not take main's latest; continuing on the older base 45 times.
Merge/resolver activity. merger.log shows the queue draining through
pipeline resolvers. 12 resolver runs have measurable durations, total
5153 s (1.43 h), median 421 s, max 1062 s: run 176 20 s and 177 20 s
("completed the merge" immediately) versus run 180 661 s, 182 602 s,
183 1062 s (handed to a pipeline). Four events record could not launch a resolver; aborting the merge for <lane>; three resolvers did NOT land their
lane (179 lane-app2, 183 lane-go, 221 lane-c). The merge-queue marker in
the worktree currently holds lane-go and lane-app2.
Deploy waits. deploy-server.log shows one deploy waiting for quiescence:
13:39:07 quiescing the lanes so no new run starts with waiting: N run(s) active every ~20 s from 3 down to 1, until 14:18:18 building from main (dce624b …) — a 2351 s (39.2 min) wait. The 910a7aa deploy at 22:27:45
waited 12 s (0 active runs).
Device/emulator churn. FAILED once: <file> - recovering the emulator and retrying appears 50 times and FAILED twice: <file> - recording and continuing
35 times (e.g. 02:53:47 [laneA] FAILED once: q03-pane-opaque-A.txt - recovering the emulator and retrying). No device lock event string was found in the lane
logs or server.log.
Real concurrency. Overlapping step intervals show a maximum of 7
simultaneous executing steps, at 2026-10-16T14:47:35Z, spanning the 6 runs
[134, 136, 137, 138, 139, 140] (one run had two steps overlapping at that
instant). Step intervals are counted as closed (started_at ≤ t ≤ finished_at), so run 134's reviewer (ending 14:47:35Z) and its implementer
(starting 14:47:35Z) both count at the touching boundary; with half-open
intervals (started_at ≤ t < finished_at) the maximum is 6 steps at
14:47:22Z, the same six runs. Using run wall-spans (created_at..finished_at) the maximum is 6
concurrent runs at 2026-10-16T14:47:22Z, the same six runs. 157 runs had at
least one step interval overlapping another run. So despite many queued runs,
the observed executing concurrency never exceeded 7 steps / 6 runs — consistent
with the lane count and the 897 workspace-busy lines, and not with a
memory/harness-capacity limit.
5. Ranked improvements
Ranked by measured impact. Items marked hypothesis have no direct measurement behind them.
-
Remove the one-run-per-workspace contention (measured 9.07 h waiting + 910 server rejections + 6.42 h restart holds). Problem: 23 resolved bus waits total 9.07 h (median 992 s, max 9070 s); the server rejected a second run in a busy workspace 910 times across 29 blocking runs; restart holds cost 6.42 h. Smallest change: give each lane its own worktree/workspace so a lane is never blocked by a run already active in it (or lower the fixed 30 s busy-poll). Expected effect: cut the multi-minute launch latency and the wasted poll lines. Risk: more concurrent worktrees consume more disk and parallel model capacity. Locus: orchestration. Evidence: the busy/rejection counts and the per-workspace breakdown above.
-
Finish or budget the malformed-result repair path. Problem:
final_result_validationis 19 of 86 non-success runs; in several (e.g. 55,81,174,179,183,221) the repair itself failed withisolated format repair failed: retained OpenCode session ses_…: context deadline exceeded, while others were judgedunsafe to repair. Smallest change: give the isolated format-repair session its own longer deadline (or a streaming repair), so a repairable-but-slow result is recovered instead of failing the run. Expected effect: recover part of the timeout-repair runs; the "unsafe to repair" set remains a separate problem. Risk: longer tail latency on already-slow runs. Locus: server code (internal/stage). Evidence: the quoted diagnostics and the before/after drop (10.9 %→3.3 %) for this cluster. -
Cut repeat-visit cost in the review loop. Problem: repeat visits of the same stage cost 13.83 h of 82.97 h (16.7 %), concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs (31,70,96,97,127) failed outright with
Pipelines stopped reviewer after 3 visits on outcome rejected. Smallest change: stop looping on advisory-only rejections (no must-fix finding), or reduce the reviewer max-visits for advisory findings. Expected effect: remove up to ~14 h of repeat work and the 5 exhausted-loop failures, if rejections are often non-blocking. Risk: a real defect could pass. Locus: pipeline definition (reviewercontrol). Evidence: the repeat/single-pass split in §2 and thereview_loop_exhaustedids. -
Make the
implementerblocked outcome actionable (measured 18/23 of the blocked class). Problem:stage_outcome_blockedis the single largest class (23/86, 26.7 %), and it is dominated by implementer (18) — notui_reviewer, which is only 4 (andreview1). Smallest change: have the implementer stage distinguish "environment/device unavailable" (retryable) from "acceptance genuinely unmeetable" (terminal), so the retryable subset is retried instead of failing the run. Expected effect: a portion of the 18 implementer-blocked runs recovered. Risk: retry storms. Locus: pipeline definition / server code. Evidence: the 23 ids and the measured blocked-stage split; the retryable/terminal split is hypothesis (not measured). -
Hand the
ui_reviewercaptured artifacts (measured 4/23 blocked). Problem:ui_reviewerblocked in 4 runs (146,167,207,210) and its p90 is 1030 s in Focused tweak; run 146's preview reportsartifacts/146/3 is empty. Smallest change: whatever captures screenshots for a UI run must hand them to theui_reviewerstage, and the stage should not block when no device is attached. Expected effect: fewer blocked UI runs. Risk: silently skipping visual verification. Locus: pipeline definition + orchestration (artifact handoff). Evidence: run 146 preview and the 4 blocked ids; partly hypothesis — the empty-artifact cause is quoted for one run only, not measured across all 4. -
Harden the harness transport path. Problem: 18 non-success runs are transport failures — 10
execution error, 6context deadline exceeded, 1final assistant readback had no completed result, 1OpenCode post … command failed. Smallest change: add bounded retry/backoff around the OpenCode…/waitcall for the generic "agent reported an execution error" case. Expected effect: recover the 10 execution-error runs if they are transient. Risk: retrying a genuinely broken session. Locus: server code (harness adapter). Evidence: the quoted diagnostics and ids. Whether the errors are transient is hypothesis. -
Stabilise merge resolution. Problem: merge-resolution pipelines contributed blocked implementers (176, 177, 178),
final_result_validation(179, 183, 221), and resolver runs cost 1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver'simplementerblocks on a conflict workspace (e.g. missing conflict markers or write-path scope). Locus: merge-resolution pipeline definition. Evidence:merger.loglines and the run diagnostics; root cause hypothesis.
6. What could not be verified
- server.log content. It is a request/response log and contains none of the stage diagnostics (0 matches for every diagnostic phrase above). It could not be used to re-derive failures; the run records were used instead. To settle whether transport errors correlate with request latency, the server would need to log per-request duration (not currently present).
- 2 non-terminal runs excluded. Runs 222 and 223 were
runningwhen the history and steps were captured; their incomplete steps (223 seq2reviewer, 222 seq4implementer) have an emptyfinished_atand were excluded from all distributions. Settle by re-readingpipelines inspectafter they finish. (An earlier capture also excluded run 220 seq3ui_reviewerwhen it was still running; it had finished by the later capture and is included above.) - 4 unresolved busy-wait episodes. Four
workspace busysequences were still open at log end (no matchinglaunchedline), so their final wait is unknown. (An earlier working note reported 5 unresolved episodes; the re-counted population gives 4.) Settle by re-parsing the lane logs once the lanes go idle. - Blocked-class cause distribution. Only run 146 has a quoted cause; the
other blocked runs were classified by
outcome=blockedplus stage name. A per-runresult_previewpass would quantify the retryable vs terminal split. - After-side sample size. The
910a7aacomparison rests on 30 runs (2 still running). A larger after-window sample is required before attributing the failure-rate drop to the change rather than to pipeline mix. - Device-lock waits. No
device lockevent string exists in the lane logs orserver.log; emulator trouble appears only asrecovering the emulatorretries. A device-lock measurement would need the orchestration to log an explicit acquire/wait/release. - No system-changing measurements were attempted. No run was started, retried, cancelled or reconfigured; the server, ntfy service, lanes and devices were not touched, and the phone/emulator was not used. Measurements that would require such changes (e.g. injecting a busy workspace to time a wait, or forcing a transport error to test retry) were deliberately not performed.
Verification and limits
Commands and observed results (all run in this session):
pipelines ping --json→ build/commit910a7aa…,dirty false.pipelines runs --json(2 calls, at different capture times) → 223 runs; the two run-status counts reconcile withpipelines stats --json(total 223, succeeded 134/135, failed 73, canceled 11, interrupted 2, active 3/2).pipelines inspect <id> --json→ 223 calls (one per run id), used for all step intervals and the taxonomy fallback.server.logbounded searches: 910is already active in workspacerejections; 0 diagnostic strings.- Lane/orchestration logs (streaming
grep/awk): 897workspace busylines, 800 restart holds, 11merge deferred, 12 resolver durations, 50/35 emulator retries. - Go verification, run with the repository's normal invocation from the module
root:
go build ./...→ exit 0, no output.go test ./...→ exit 0; 20 packagesok, 3cmd/*andinternal/wstesthave no test files. No Go source was changed; these confirm the worktree was left buildable.
Limits: the after-side sample is small; several classes did not recur after the
boundary for reasons the data cannot separate from the format change; the
blocked-class cause distribution and the transience of transport errors are
hypotheses, not measurements. Raw captures are under
/Users/alan/.local/share/pipelines/data/artifacts/222/2/ and
/Users/alan/.local/share/pipelines/data/artifacts/222/4/ (outside the repo).
7. Transport and repair recovery (source-level, 2026-10-17)
Read-only pass over the §1 diagnostics after the harness-transport and
format-repair change. No run was started, retried or cancelled; no server, ntfy,
lane or device was restarted or redeployed; no provider inference was performed.
Each shape is mapped to the code path that produces it. The per-run
pre-admission vs post-admission and read-only vs mutating split was not
re-inspected with pipelines inspect, so every per-run row is labelled not
verified and the counts are not summed into a "recovered" total.
| failure shape (§1 class, quoted diagnostic) | run ids | code path | can recover | remains unrecoverable |
|---|---|---|---|---|
execution error at readback (OpenCode agent reported an execution error) |
73,74,75,77,80,149,150,152,195,202 | post-admission completedAgentResult / opencode.ExecutionError |
a stage that is explicit read-only may retry within retries(not verified per run) |
a mutating/legacy stage never replays admitted work; a non-transient provider error stays terminal |
context deadline exceeded |
33,36,38,59,101,138 | attempt-boundary deadline, or a transport call that expired | a deadline during an in-flight request now keeps the transport cause and, when pre-admission, retries for any stage; a read-only stage retries a post-admission one | a genuine stage overrun on a mutating/legacy stage stays terminal; whether each run was a hung call or a real overrun is not verified |
readback failure (final assistant readback had no completed result) |
161 | post-admission completedAgentResult |
explicit read-only stage may retry | mutating/legacy stage never replays |
native V1 /wait command failed (CLI exit code 1) |
4 | post-admission openCodeError on /wait |
explicit read-only stage may retry | mutating/legacy stage never replays |
repair timeout (isolated format repair failed … context deadline exceeded) |
55,81,174,179,183,221 | repairDeadline (payload bytes + stage timeout, floored, capped, clamped to remaining − margin) |
a slow-but-repairable payload now gets a longer bounded budget | run 55's payload is the ambiguous unescaped-quote/colon class (byte offset 7837) → unrecoverable; any payload past the 32 KiB bound stays unrecoverable |
Mapping notes: the execution-error and readback shapes are read back after the
prompt was admitted, so they are post-admission transport failures and the
no-tool-replay rule keeps them terminal on a mutating stage. The
session_transport_command_failed shape (run 4) is the native V1 /wait call
issued after the prompt, not a session-create reach failure. The repair-timeout
shapes are FINAL RESULT VALIDATION (repair-stage), not transport. The new code
also reports an unknown provider execution-error name as a bounded sanitized
identifier plus any HTTP status, so future occurrences of runs 73–202 are more
precisely diagnostic even when they are not retried.