Combine the two reviewed versions of docs/pipeline-performance-review.md. - Local (main) review: the exact 201939-byte runs.json capture, the three final_result_validation sub-shapes, and the 5-to-4 unresolved busy-wait correction. - Incoming (lane-t2) review: both captures (artifacts/222/2 and artifacts/222/4), both stat snapshots (134 succeeded / 3 active and 135 succeeded / 2 active), plus the server-side rejection, deploy-wait, device-lock and refined improvement-ranking analysis. Checked: no conflict markers remain anywhere in the tree and the two sides are composed without duplicated headings or dead text. This is a documentation-only change to a dated analysis document; the packaged templates, the README and topic docs, and the packaged authoring skill references are not made stale by it, so nothing further needs updating.
25 KiB
Pipeline performance and reliability review
Read-only analysis of the live Pipelines run history, step records and orchestration logs, on 2026-10-17. The only change is this document.
Sources and method
All numbers come from commands run in this session against the running server
(pipelines ping --json reports build.version v0.0.0-20261016212730-910a7aa798af, commit 910a7aa798af3eecff72fcde28b9eafde470286c, dirty false).
pipelines runs --jsonwas captured twice: once into/Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json(223 entries, 201939 bytes) and once into/Users/alan/.local/share/pipelines/data/artifacts/222/4/runs.json(223 entries). Both captures were reused for the taxonomy, trend and concurrency.pipelines stats --jsonwas captured twice:/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.jsonreportedtotal 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3, and/Users/alan/.local/share/pipelines/data/artifacts/222/4/stats.jsonreportedtotal 223, succeeded 135, failed 73, canceled 11, interrupted 2, active 2. Both inventories reconcile with the run totals (134+73+11+2+3 = 223and135+73+11+2+2 = 223).pipelines inspect <id> --jsonwas called exactly once per run id (223 calls, saved underartifacts/222/4/inspect/); step intervals were aggregated in one pass from those files.- Orchestration logs:
/tmp/lane{A,B,C,D,E,Go,T2,Chat}.log,/tmp/merger.log,/tmp/deploy-server.log, and thelane-{hg,hs}.logfiles in/private/var/folders/jp/q0qvjyg54fl_c5vhhk1ylz2m0000gn/T/opencode/, plus~/.local/share/pipelines/logs/server.log(16,282,789 bytes, 143,788 lines).
Quantile convention: nearest-rank, i.e. the p-quantile is element
ceil(p*n) (1-based) of the sorted list; the median is the p=0.5 element.
Durations are finished_at - started_at parsed with RFC3339/RFC3339Nano rules
(Z normalised, empty string = absent), as in internal/pipeline/search.go.
1. Failure taxonomy over the whole history
The inventory holds 223 runs (running 2, succeeded 135, failed 73,
canceled 11, interrupted 2). 86 runs have a terminal non-success status
(failed 73, canceled 11, interrupted 2). Every one is classified below from
the run diagnostic, or — where that is empty (runs 30, 32, 45) — from the
failing step's outcome in its pipelines inspect record. The class counts sum
to 86; there is no unknown bucket.
| class | count | run ids |
|---|---|---|
stage_outcome_blocked |
23 | 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210 |
final_result_validation |
19 | 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221 |
harness_transport_execution_error |
10 | 73,74,75,77,80,149,150,152,195,202 |
harness_transport_timeout |
6 | 33,36,38,59,101,138 |
cancelled_by_operator |
11 | 28,39,57,63,78,108,113,114,191,211,219 |
review_loop_exhausted |
5 | 31,70,96,97,127 |
stage_outcome_error |
4 | 30,137,139,190 |
interrupted_by_deploy |
2 | 13,24 |
invalid_agent_messages |
2 | 2,5 |
harness_transport_readback |
1 | 161 |
agent_quiescence_unconfirmed |
1 | 43 |
session_transport_command_failed |
1 | 4 |
duplicate_json_field |
1 | 3 |
Classifying diagnostics (quoted verbatim from the run records):
- stage_outcome_blocked — a stage ends with
outcome=blocked. The split by blocked stage is implementer 18, ui_reviewer 4, review 1. Run 105:stage review returned outcome "blocked"; runs 32/45 have an empty rundiagnosticand animplementerfinal stepoutcome='blocked'; the rest carrystage implementer returned outcome "blocked"orstage ui_reviewer returned outcome "blocked". Run 146 shows a deliberate stop: "No screenshots/frames exist for this run (artifacts/146/1 contains only the APK; artifacts/146/3 is empty), and the task's device rules are mutually exclusive". - final_result_validation — malformed/schema-rejected model result. Run 8:
model stage implementer: FINAL RESULT VALIDATION: invalid model stage result: invalid character 'I' looking for beginning of value at byte 1; unsafe to repair incomplete values or a non-object response; completed edits and tool side effects are retained and not undone; a full retry repeats work. Run 221:model stage reviewer: FINAL RESULT VALIDATION: invalid model stage result: invalid character '}' after array element at byte 2929; isolated format repair failed: retained OpenCode session ses_eb8c5ab8affeAeLm9sFdG8uimz: context deadline exceeded; …. Two sub-shapes were observed:unsafe to repair incomplete values or a non-object response(runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) andisolated format repair failed: retained OpenCode session ses_…: context deadline exceeded(runs 55,81,174,179,183,221 — 6 runs). Runs 85/142 add a third:unknown result field; run 170's diagnostic isinvalid character '›' after object key:value pair at byte 780. - harness_transport_execution_error — e.g. run 73:
model stage ui_reviewer: HARNESS TRANSPORT: harness transport failure: OpenCode agent reported an execution error. - harness_transport_timeout — run 33:
model stage implementer: HARNESS TRANSPORT: context deadline exceeded\ncontext deadline exceeded. - cancelled_by_operator —
context canceled, statuscanceled. - review_loop_exhausted — run 31:
Pipelines stopped reviewer after 3 visits on outcome rejected. - stage_outcome_error — run 190:
stage implementer returned outcome "error"; runs 30/137/139 have a final stepoutcome='error'. - interrupted_by_deploy — run 13/24:
Server stopped; run was not resumed, statusinterrupted. - invalid_agent_messages — run 2/5:
model stage planner: invalid agent messages. - harness_transport_readback — run 161:
harness transport failure: final assistant readback had no completed result. - agent_quiescence_unconfirmed — run 43:
agent quiescence not confirmed; inspect retained session before further work. - session_transport_command_failed — run 4:
model stage implementer: OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait command failed: CLI exit code 1. - duplicate_json_field — run 3:
invalid model stage result: duplicate JSON field "performance_review".
server.log contains none of these diagnostic strings (bounded search:
FINAL RESULT VALIDATION 0, HARNESS TRANSPORT 0, invalid agent messages 0,
stopped reviewer after 0, quiescence 0); it is a request/response log
(command received/response sent, plus summarise: harness=opencode …).
Diagnostics live in the run records, not the server log.
2. Stage cost
The 223 runs produced 828 steps; 826 have both started_at and
finished_at and were used. The 2 without a finished_at belong to the 2
still-running runs and were excluded, not fabricated: run 223 seq2 reviewer
and run 222 seq4 implementer. By run: 221 runs have a non-empty finished_at
and all 223 runs contribute at least one complete step. This is the population
at the 2026-10-17 capture, when run 220 had already finished — an earlier working
note that called the population "222 finished" was stale and is corrected here.
Total wall-hours by stage key (sum of finished_at - started_at), with the share
spent on repeat visits (control.visit > 1, i.e. review/retry loops) split out:
| stage key | total (h) | single-pass (h) | repeat (h) | steps |
|---|---|---|---|---|
| implementer | 41.36 | 33.73 | 7.62 | 270 |
| reviewer | 20.72 | 15.76 | 4.96 | 220 |
| ui_reviewer | 15.18 | 13.93 | 1.25 | 135 |
| planner | 3.55 | 3.55 | 0.00 | 79 |
| committer | 2.00 | 2.00 | 0.00 | 99 |
| worker/research/review/probe/report/opencode/pi | 0.16 | 0.16 | 0.00 | 23 |
The dominant stage keys are implementer (41.36 h), reviewer (20.72 h) and ui_reviewer (15.18 h) — together 77.26 of 82.97 h. Repeat visits of the same stage cost 13.83 h (16.7 %), almost all in implementer (7.62 h) and reviewer (4.96 h).
Per (pipeline, stage key) distribution (count / median / p90 / max in seconds / total hours), for the highest-volume keys across the three highest-volume pipelines:
| pipeline | stage | n | median | p90 | max | total (h) |
|---|---|---|---|---|---|---|
| Focused tweak - implement review commit | implementer | 137 | 498 | 1414 | 1801 | 23.78 |
| Pipelines maintenance - plan implement review | implementer | 90 | 370 | 847 | 3601 | 12.07 |
| Pipelines maintenance - plan implement review | reviewer | 79 | 245 | 503 | 12613 | 10.54 |
| Focused tweak - implement review commit | ui_reviewer | 81 | 293 | 1030 | 2158 | 10.05 |
| Focused tweak - implement review commit | reviewer | 111 | 202 | 449 | 1088 | 7.75 |
| Pipelines quick - plan implement review | implementer | 14 | 683 | 1266 | 1716 | 2.99 |
| Pipelines maintenance - plan implement review | planner | 62 | 132 | 274 | 541 | 2.75 |
| Pipelines quick - plan implement review | ui_reviewer | 10 | 496 | 1572 | 2701 | 2.62 |
| Pipelines maintenance - plan implement review | ui_reviewer | 40 | 23 | 898 | 1362 | 2.50 |
| Pipelines quick - plan implement review | reviewer | 13 | 324 | 896 | 1800 | 1.69 |
| Focused tweak - implement review commit | committer | 63 | 31 | 180 | 644 | 1.13 |
Three pipelines are shown (Focused tweak, Pipelines maintenance, Pipelines quick); the full per-pipeline table was produced by the aggregation command.
Notable long tails: one reviewer step in Pipelines maintenance ran 12613 s
(3.5 h); one Pipelines-maintenance implementer step hit 3601 s (≈ the 1 h
stage timeout), while the Focused-tweak implementer max is 1801 s and its p90
1414 s. The ui_reviewer p90 is high (1030 s in Focused tweak).
3. Reliability trend, split at 910a7aa
Boundary: 2026-10-16T21:27:00Z, the deploy of commit 910a7aa. pipelines ping --json confirms the running build.commit is 910a7aa. deploy-server.log
corroborates: 22:27:45 quiescing the lanes … / 22:27:45 building from main (910a7aa Merge lane-app2: …) / 22:27:57 server healthy; deployed 910a7aa
(its clock is local UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by
created_at: the last "before" run is 193 (2026-10-16T21:21:33Z), the first
"after" is 194 (2026-10-16T21:28:00Z).
| side | n | running | failed | failed/(failed+succeeded) | non-success/terminal |
|---|---|---|---|---|---|
| before | 193 | 0 | 68 | 37.4 % | 40.9 % |
| after | 30 | 2 | 5 | 19.2 % | 25.0 % |
Failure-class mix, before → after (counts):
| class | before | after |
|---|---|---|
| stage_outcome_blocked | 21 | 2 |
| final_result_validation | 18 | 1 |
| harness_transport_execution_error | 8 | 2 |
| harness_transport_timeout | 6 | 0 |
| cancelled_by_operator | 9 | 2 |
| review_loop_exhausted | 5 | 0 |
| stage_outcome_error | 4 | 0 |
| interrupted_by_deploy | 2 | 0 |
| invalid_agent_messages | 2 | 0 |
| harness_transport_readback | 1 | 0 |
| agent_quiescence_unconfirmed | 1 | 0 |
| session_transport_command_failed | 1 | 0 |
| duplicate_json_field | 1 | 0 |
Read literally the change correlates with fewer malformed-result failures:
final_result_validation + duplicate_json_field + invalid_agent_messages falls
from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is confounded
and under-powered: the after side is only 30 runs (2 still running), the
pipeline mix differs (after is mostly Focused tweak 15, Pipelines maintenance 7,
Merge-resolution 3 — several one-off template/docker/research pipelines appear
only before), and several before-side classes (transport timeout, review-loop,
deploy interruption) simply never recurred in the short after window rather than
being fixed by 910a7aa. The data is consistent with the format/repair change
helping; it does not establish causation. A larger after sample is needed.
4. Throughput limits (measured)
Busy-workspace waits. The lane logs record workspace busy; waiting to launch <file> at a fixed 30 s cadence: 897 such lines. Grouping them into
episodes (same lane+file, ending at the launched <file> line) gives 27
episodes, 23 resolved and 4 still open at log end. For the 23 resolved
episodes the wait is launch_time - first_busy:
- n=23, median 992 s (16.5 min), p90 2293 s (38.2 min), max
9070 s (2.5 h), total 9.07 h. This independent reimplementation of the
rule includes the laneGo
malformed-result-brief.txtepisode (2170 s); an earlier working note that dropped it reported n=22 / 8.47 h and is corrected here.
Longest episodes: laneA s02-tui-omni-search.txt waited 13:12:33 → 15:43:43
(9070 s) before 15:43:43 launched s02-tui-omni-search.txt as run 137; laneGo
s02-tui-omni-search.txt waited 15:47:22 → 16:48:28 (3666 s).
Server-side rejection of a second run in a busy workspace. server.log
records 910 error="run <id> is already active in workspace <path>; wait for it to finish or start this run in a different workspace …" responses, from
29 distinct blocking runs, concentrated on 2026/10/16 (905) vs 2026/10/15
(5). By workspace: lane-go 387, /Users/alan/repos/pipelines 307, lane-app2
166, lane-c 49, ws-c 1. So the dominant limit is one-run-per-workspace, and
the lanes retry it hundreds of times.
Server-restart holds. holding: another lane is restarting the server
appears 800 times; grouping holds per lane while consecutive hold lines are
≤60 s apart (the 30 s poll cadence plus jitter; a wider gap starts a new
episode) gives 29 episodes totalling 6.42 h, median 480 s, max 3572 s. (A
strict consecutive-line regroup, which splits an episode on any interleaved
line, gives 32 episodes; the 29 figure is the gap-based grouping.) building server and cli from main 13 times; server redeployed and healthy 10 times.
Merge/main-lane waits. merge deferred: main lane busy appears 11 times;
merge requested for … 24 times; MERGE CONFLICT 15 times; SYNC CONFLICT: lane could not take main's latest; continuing on the older base 45 times.
Merge/resolver activity. merger.log shows the queue draining through
pipeline resolvers. 12 resolver runs have measurable durations, total
5153 s (1.43 h), median 421 s, max 1062 s: run 176 20 s and 177 20 s
("completed the merge" immediately) versus run 180 661 s, 182 602 s,
183 1062 s (handed to a pipeline). Four events record could not launch a resolver; aborting the merge for <lane>; three resolvers did NOT land their
lane (179 lane-app2, 183 lane-go, 221 lane-c). The merge-queue marker in
the worktree currently holds lane-go and lane-app2.
Deploy waits. deploy-server.log shows one deploy waiting for quiescence:
13:39:07 quiescing the lanes so no new run starts with waiting: N run(s) active every ~20 s from 3 down to 1, until 14:18:18 building from main (dce624b …) — a 2351 s (39.2 min) wait. The 910a7aa deploy at 22:27:45
waited 12 s (0 active runs).
Device/emulator churn. FAILED once: <file> - recovering the emulator and retrying appears 50 times and FAILED twice: <file> - recording and continuing
35 times (e.g. 02:53:47 [laneA] FAILED once: q03-pane-opaque-A.txt - recovering the emulator and retrying). No device lock event string was found in the lane
logs or server.log.
Real concurrency. Overlapping step intervals show a maximum of 7
simultaneous executing steps, at 2026-10-16T14:47:35Z, spanning the 6 runs
[134, 136, 137, 138, 139, 140] (one run had two steps overlapping at that
instant). Step intervals are counted as closed (started_at ≤ t ≤ finished_at), so run 134's reviewer (ending 14:47:35Z) and its implementer
(starting 14:47:35Z) both count at the touching boundary; with half-open
intervals (started_at ≤ t < finished_at) the maximum is 6 steps at
14:47:22Z, the same six runs. Using run wall-spans (created_at..finished_at) the maximum is 6
concurrent runs at 2026-10-16T14:47:22Z, the same six runs. 157 runs had at
least one step interval overlapping another run. So despite many queued runs,
the observed executing concurrency never exceeded 7 steps / 6 runs — consistent
with the lane count and the 897 workspace-busy lines, and not with a
memory/harness-capacity limit.
5. Ranked improvements
Ranked by measured impact. Items marked hypothesis have no direct measurement behind them.
-
Remove the one-run-per-workspace contention (measured 9.07 h waiting + 910 server rejections + 6.42 h restart holds). Problem: 23 resolved bus waits total 9.07 h (median 992 s, max 9070 s); the server rejected a second run in a busy workspace 910 times across 29 blocking runs; restart holds cost 6.42 h. Smallest change: give each lane its own worktree/workspace so a lane is never blocked by a run already active in it (or lower the fixed 30 s busy-poll). Expected effect: cut the multi-minute launch latency and the wasted poll lines. Risk: more concurrent worktrees consume more disk and parallel model capacity. Locus: orchestration. Evidence: the busy/rejection counts and the per-workspace breakdown above.
-
Finish or budget the malformed-result repair path. Problem:
final_result_validationis 19 of 86 non-success runs; in several (e.g. 55,81,174,179,183,221) the repair itself failed withisolated format repair failed: retained OpenCode session ses_…: context deadline exceeded, while others were judgedunsafe to repair. Smallest change: give the isolated format-repair session its own longer deadline (or a streaming repair), so a repairable-but-slow result is recovered instead of failing the run. Expected effect: recover part of the timeout-repair runs; the "unsafe to repair" set remains a separate problem. Risk: longer tail latency on already-slow runs. Locus: server code (internal/stage). Evidence: the quoted diagnostics and the before/after drop (10.9 %→3.3 %) for this cluster. -
Cut repeat-visit cost in the review loop. Problem: repeat visits of the same stage cost 13.83 h of 82.97 h (16.7 %), concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs (31,70,96,97,127) failed outright with
Pipelines stopped reviewer after 3 visits on outcome rejected. Smallest change: stop looping on advisory-only rejections (no must-fix finding), or reduce the reviewer max-visits for advisory findings. Expected effect: remove up to ~14 h of repeat work and the 5 exhausted-loop failures, if rejections are often non-blocking. Risk: a real defect could pass. Locus: pipeline definition (reviewercontrol). Evidence: the repeat/single-pass split in §2 and thereview_loop_exhaustedids. -
Make the
implementerblocked outcome actionable (measured 18/23 of the blocked class). Problem:stage_outcome_blockedis the single largest class (23/86, 26.7 %), and it is dominated by implementer (18) — notui_reviewer, which is only 4 (andreview1). Smallest change: have the implementer stage distinguish "environment/device unavailable" (retryable) from "acceptance genuinely unmeetable" (terminal), so the retryable subset is retried instead of failing the run. Expected effect: a portion of the 18 implementer-blocked runs recovered. Risk: retry storms. Locus: pipeline definition / server code. Evidence: the 23 ids and the measured blocked-stage split; the retryable/terminal split is hypothesis (not measured). -
Hand the
ui_reviewercaptured artifacts (measured 4/23 blocked). Problem:ui_reviewerblocked in 4 runs (146,167,207,210) and its p90 is 1030 s in Focused tweak; run 146's preview reportsartifacts/146/3 is empty. Smallest change: whatever captures screenshots for a UI run must hand them to theui_reviewerstage, and the stage should not block when no device is attached. Expected effect: fewer blocked UI runs. Risk: silently skipping visual verification. Locus: pipeline definition + orchestration (artifact handoff). Evidence: run 146 preview and the 4 blocked ids; partly hypothesis — the empty-artifact cause is quoted for one run only, not measured across all 4. -
Harden the harness transport path. Problem: 18 non-success runs are transport failures — 10
execution error, 6context deadline exceeded, 1final assistant readback had no completed result, 1OpenCode post … command failed. Smallest change: add bounded retry/backoff around the OpenCode…/waitcall for the generic "agent reported an execution error" case. Expected effect: recover the 10 execution-error runs if they are transient. Risk: retrying a genuinely broken session. Locus: server code (harness adapter). Evidence: the quoted diagnostics and ids. Whether the errors are transient is hypothesis. -
Stabilise merge resolution. Problem: merge-resolution pipelines contributed blocked implementers (176, 177, 178),
final_result_validation(179, 183, 221), and resolver runs cost 1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver'simplementerblocks on a conflict workspace (e.g. missing conflict markers or write-path scope). Locus: merge-resolution pipeline definition. Evidence:merger.loglines and the run diagnostics; root cause hypothesis.
6. What could not be verified
- server.log content. It is a request/response log and contains none of the stage diagnostics (0 matches for every diagnostic phrase above). It could not be used to re-derive failures; the run records were used instead. To settle whether transport errors correlate with request latency, the server would need to log per-request duration (not currently present).
- 2 non-terminal runs excluded. Runs 222 and 223 were
runningwhen the history and steps were captured; their incomplete steps (223 seq2reviewer, 222 seq4implementer) have an emptyfinished_atand were excluded from all distributions. Settle by re-readingpipelines inspectafter they finish. (An earlier capture also excluded run 220 seq3ui_reviewerwhen it was still running; it had finished by the later capture and is included above.) - 4 unresolved busy-wait episodes. Four
workspace busysequences were still open at log end (no matchinglaunchedline), so their final wait is unknown. (An earlier working note reported 5 unresolved episodes; the re-counted population gives 4.) Settle by re-parsing the lane logs once the lanes go idle. - Blocked-class cause distribution. Only run 146 has a quoted cause; the
other blocked runs were classified by
outcome=blockedplus stage name. A per-runresult_previewpass would quantify the retryable vs terminal split. - After-side sample size. The
910a7aacomparison rests on 30 runs (2 still running). A larger after-window sample is required before attributing the failure-rate drop to the change rather than to pipeline mix. - Device-lock waits. No
device lockevent string exists in the lane logs orserver.log; emulator trouble appears only asrecovering the emulatorretries. A device-lock measurement would need the orchestration to log an explicit acquire/wait/release. - No system-changing measurements were attempted. No run was started, retried, cancelled or reconfigured; the server, ntfy service, lanes and devices were not touched, and the phone/emulator was not used. Measurements that would require such changes (e.g. injecting a busy workspace to time a wait, or forcing a transport error to test retry) were deliberately not performed.
Verification and limits
Commands and observed results (all run in this session):
pipelines ping --json→ build/commit910a7aa…,dirty false.pipelines runs --json(2 calls, at different capture times) → 223 runs; the two run-status counts reconcile withpipelines stats --json(total 223, succeeded 134/135, failed 73, canceled 11, interrupted 2, active 3/2).pipelines inspect <id> --json→ 223 calls (one per run id), used for all step intervals and the taxonomy fallback.server.logbounded searches: 910is already active in workspacerejections; 0 diagnostic strings.- Lane/orchestration logs (streaming
grep/awk): 897workspace busylines, 800 restart holds, 11merge deferred, 12 resolver durations, 50/35 emulator retries. - Go verification, run with the repository's normal invocation from the module
root:
go build ./...→ exit 0, no output.go test ./...→ exit 0; 20 packagesok, 3cmd/*andinternal/wstesthave no test files. No Go source was changed; these confirm the worktree was left buildable.
Limits: the after-side sample is small; several classes did not recur after the
boundary for reasons the data cannot separate from the format change; the
blocked-class cause distribution and the transience of transport errors are
hypotheses, not measurements. Raw captures are under
/Users/alan/.local/share/pipelines/data/artifacts/222/2/ and
/Users/alan/.local/share/pipelines/data/artifacts/222/4/ (outside the repo).