Files
pipelines/docs/pipeline-performance-review.md
T

20 KiB

Pipeline performance and reliability review

Read-only analysis of the live Pipelines run history, step records and orchestration logs, on 2026-10-17. The only change is this document.

Sources and method

All numbers come from commands run in this session against the running server (pipelines ping reports build.version v0.0.0-20261016212730-910a7aa798af, commit 910a7aa798af3eecff72fcde28b9eafde470286c, dirty false).

  • pipelines runs --json was captured once into /Users/alan/.local/share/pipelines/data/artifacts/222/2/runs.json (223 entries, 201939 bytes) and reused for the taxonomy and trend.
  • pipelines stats --json was captured once (/Users/alan/.local/share/pipelines/data/artifacts/222/2/stats.json): total 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3. 134+73+11+2+3 = 223, so the run inventory reconciles with the totals.
  • pipelines inspect <id> --json was called exactly once per run id (223 calls, saved under artifacts/222/2/inspect/); step intervals were aggregated in one pass from those files.
  • Orchestration logs: /tmp/lane{A,B,C,D,E,Go,T2}.log, /tmp/merger.log, /tmp/deploy-server.log, and .../T/opencode/lane-{hg,hs}.log, plus ~/.local/share/pipelines/logs/server.log (16.2 MB, 143080 lines).

Quantile convention: nearest-rank, i.e. the p-quantile is element ceil(p*n) (1-based) of the sorted list; the median is the p=0.5 element. Durations are finished_at - started_at parsed with RFC3339/RFC3339Nano rules (Z normalised), as in internal/pipeline/search.go.

1. Failure taxonomy over the whole history

86 runs have a terminal non-success status: failed 73, canceled 11, interrupted 2. Every one is classified below from the run diagnostic, or — where that is empty — from the failing step's outcome in its inspect record. The class counts sum to 86; there is no unknown bucket.

class count run ids
stage_outcome_blocked 23 32,45,105,115,141,145,146,151,154,156,158,159,160,167,173,175,176,177,178,188,189,207,210
final_result_validation 19 8,12,29,55,81,85,94,118,120,126,142,148,157,164,170,174,179,183,221
cancelled_by_operator 11 28,39,57,63,78,108,113,114,191,211,219
harness_transport_execution_error 10 73,74,75,77,80,149,150,152,195,202
harness_transport_timeout 6 33,36,38,59,101,138
review_loop_exhausted 5 31,70,96,97,127
stage_outcome_error 4 30,137,139,190
interrupted_by_deploy 2 13,24
invalid_agent_messages 2 2,5
harness_transport_readback 1 161
agent_quiescence_unconfirmed 1 43
session_transport_command_failed 1 4
duplicate_json_field 1 3

Classifying diagnostics (quoted verbatim from the run records):

  • stage_outcome_blocked — a stage returns blocked, e.g. run 32/45 (implementer final step outcome='blocked', run diagnostic empty) and run 105: stage review returned outcome "blocked"; run 115/141/145/151/154/ 156/158/159/160/167/173/175/176/177/178/188/189/207/210: stage implementer returned outcome "blocked" or stage ui_reviewer returned outcome "blocked". The blocked stages are dominated by implementer and ui_reviewer, and their result_preview shows the stage stopping deliberately — e.g. run 146: "No screenshots/frames exist for this run (artifacts/146/1 contains only the APK; artifacts/146/3 is empty), and the task's device rules are mutually ex[c]lusive".
  • final_result_validation — malformed/schema-rejected model result. Example, run 8: model stage implementer: FINAL RESULT VALIDATION: invalid model stage result: invalid character 'I' looking for beginning of value at byte 1; unsafe to repair incomplete values or a non-object response; completed edits and tool side effects are retained and not undone; a full retry repeats work. Two sub-shapes: unsafe to repair incomplete values or a non-object response (runs 8,12,29,94,118,120,126,148,157,164,170 — 11 runs) and isolated format repair failed: retained OpenCode session ses_…: context deadline exceeded (runs 55,81,174,179,183,221 — 6 runs). Run 85/142 add a third: unknown result field.
  • harness_transport_execution_error — run 73/74/75/77/80/149/150/152/195/202: model stage <stage>: HARNESS TRANSPORT: harness transport failure: OpenCode agent reported an execution error.
  • harness_transport_timeout — run 33: model stage implementer: HARNESS TRANSPORT: context deadline exceeded \n context deadline exceeded (also 36,38,59,101,138).
  • review_loop_exhausted — run 31: Pipelines stopped reviewer after 3 visits on outcome rejected (also 70,96,97,127).
  • cancelled_by_operator — context canceled with status canceled.
  • interrupted_by_deploy — run 13/24: Server stopped; run was not resumed with status interrupted.
  • invalid_agent_messages — run 2/5: model stage planner: invalid agent messages.
  • duplicate_json_field — run 3: invalid model stage result: duplicate JSON field "performance_review".
  • harness_transport_readback — run 161: harness transport failure: final assistant readback had no completed result.
  • agent_quiescence_unconfirmed — run 43: agent quiescence not confirmed; inspect retained session before further work.
  • session_transport_command_failed — run 4: model stage implementer: OpenCode post /api/experimental/session/ses_ed7a88fbcffeWXS5D5gd22DIm0/wait command failed: CLI exit code 1.

server.log contains none of these diagnostic strings (bounded search: FINAL RESULT VALIDATION 0, HARNESS TRANSPORT 0, invalid agent messages 0, stopped reviewer after 0); it is a request/response log (command received/response sent, plus summarise: harness=opencode …). Diagnostics live in the run records, not the server log.

2. Stage cost

821 steps across the 222 finished runs (3 steps of the 3 live runs have an empty finished_at and were excluded, not fabricated: runs 223 seq1 implementer, 222 seq2 implementer, 220 seq3 ui_reviewer).

Total wall-hours by stage key (sum of finished_at - started_at), with the share spent on repeat visits (control.visit > 1, i.e. review/retry loops) split out:

stage key total (h) single-pass (h) repeat (h) steps
implementer 41.13 33.51 7.62 268
reviewer 20.65 15.69 4.96 219
ui_reviewer 15.09 13.84 1.25 134
planner 3.55 3.55 0.00 79
committer 1.95 1.95 0.00 98
(worker/research/review/probe/report/opencode/pi) 0.15 0.15 0.00 23

The dominant stage keys are implementer (41.1 h), reviewer (20.7 h) and ui_reviewer (15.1 h) — together 76.9 of 82.53 h. Repeat visits of the same stage cost 13.83 h (16.8 %), almost all in implementer (7.62 h) and reviewer (4.96 h).

Per (pipeline, stage key) distribution (count / median / p90 / max seconds / total hours). Only the two high-volume pipelines are shown; the full table was produced by the aggregation command.

pipeline stage n median p90 max total (h)
Focused tweak - implement review commit implementer 137 498 1414 1801 23.78
Focused tweak - implement review commit reviewer 111 202 449 1088 7.75
Focused tweak - implement review commit ui_reviewer 80 290 1030 2158 9.96
Focused tweak - implement review commit committer 62 29 178 644 1.08
Pipelines maintenance - plan implement review implementer 89 375 884 3601 11.98
Pipelines maintenance - plan implement review reviewer 78 242 503 12613 10.47
Pipelines maintenance - plan implement review ui_reviewer 40 23 898 1362 2.50
Pipelines maintenance - plan implement review planner 62 132 274 541 2.75
Pipelines quick - plan implement review implementer 14 683 1266 1716 2.99
Pipelines quick - plan implement review ui_reviewer 10 496 1572 2701 2.62

Notable long tails: one reviewer step in Pipelines maintenance ran 12613 s (3.5 h); one Pipelines-maintenance implementer step hit 3601 s (≈ the 1 h stage timeout), while the Focused-tweak implementer max is 1801 s and its p90 1414 s. The ui_reviewer p90 is high (1030 s in Focused tweak).

3. Reliability trend, split at 910a7aa

Boundary: 2026-10-16T21:27:00Z, the deploy of commit 910a7aa. pipelines ping confirms the running build.commit is 910a7aa. deploy-server.log corroborates: 22:27:45 quiescing the lanes … / 22:27:45 building from main (910a7aa Merge lane-app2: …) / 22:27:57 server healthy; deployed 910a7aa (its clock is local UTC+1, so 22:27:57 ≈ 21:27:57Z). Runs are partitioned by created_at: the last "before" run is 193 (2026-10-16T21:21:33Z), the first "after" is 194 (2026-10-16T21:28:00Z).

side n running failed failed/(failed+succeeded) non-success/terminal
before 193 0 68 37.4 % 40.9 %
after 30 3 5 20.0 % 25.9 %

Failure-class mix, before → after (counts):

class before after
stage_outcome_blocked 21 2
final_result_validation 18 1
harness_transport_execution_error 8 2
harness_transport_timeout 6 0
cancelled_by_operator 9 2
review_loop_exhausted 5 0
stage_outcome_error 4 0
interrupted_by_deploy 2 0
invalid_agent_messages 2 0
harness_transport_readback 1 0
agent_quiescence_unconfirmed 1 0
session_transport_command_failed 1 0
duplicate_json_field 1 0

Read literally the change correlates with fewer malformed-result failures: final_result_validation + duplicate_json_field + invalid_agent_messages falls from 21/193 (10.9 % of runs) to 1/30 (3.3 %). But the comparison is confounded and under-powered: the after side is only 30 runs (3 still running), the pipeline mix differs (after is mostly Focused tweak 15, Pipelines maintenance 7, Merge-resolution 3 — the one-off template/docker/research pipelines appear only before), and several before-side classes (transport timeout, review-loop, deploy interruption) simply never recurred in the short after window rather than being fixed by 910a7aa. The data is consistent with the format/repair change helping; it does not establish causation. A larger after sample is needed.

4. Throughput limits (measured)

Busy-workspace waits. The lane logs record workspace busy; waiting to launch <file> at a fixed 30 s cadence. 897 such lines were counted across the lane logs. Grouping them into wait episodes (same lane+file, gaps ≤ 120 s) gives 28 episodes, 23 of which end with a launched <file> line and 5 still open at log end. For the 23 resolved episodes the wait is launch_time - first_busy:

  • n=23, median 992 s (16.5 min), p90 2293 s (38.2 min), max 3666 s (61.1 min), total 7.34 h.

Longest episodes: laneGo s02-tui-omni-search.txt waited 15:47:22→16:48:28 (3666 s, 122 busy lines) before 16:48:28 launched s02-tui-omni-search.txt as run 149; laneA s02-tui-omni-search.txt waited 2825 s; laneGo search-server-tui-brief.txt waited 2293 s.

Server-restart holds. holding: another lane is restarting the server appears 769 times; building server and cli from main 13 times; server redeployed and healthy 10 times. These are lanes idling while a deploy or rebuild owns the server.

Merge/main-lane waits. merge deferred: main lane busy appears 11 times (e.g. /tmp/laneB.log:243 04:14:33 [laneB] merge deferred: main lane busy). merge requested for … appears 23 times; MERGE CONFLICT recorded 15 times; SYNC CONFLICT: lane could not take main's latest; continuing on the older base 44 times.

Merge/resolver activity. merger.log shows the queue draining through pipeline resolvers. 12 resolver runs have measurable durations, total 5153 s (1.43 h): run 176 20 s, 177 20 s (these "completed the merge" immediately), versus run 180 661 s, 182 602 s, 183 1062 s (handed to a pipeline). Three did not land: 179 did NOT land lane-app2, 183 did NOT land lane-go, 221 did NOT land lane-c. The merge-queue marker in the worktree currently holds lane-go, lane-app2.

Device/emulator churn. FAILED once: <file> - recovering the emulator and retrying appears 50 times and FAILED twice: <file> - recording and continuing 35 times (e.g. 02:53:47 [laneA] FAILED once: q03-pane-opaque-A.txt - recovering the emulator and retrying).

Real concurrency. Overlapping step intervals show a maximum of 6 simultaneous executing steps, at 2026-10-16T14:47:22Z, in runs [134, 136, 137, 138, 139, 140]. 156 of the 222 finished runs had at least one step overlapping another run. Using run wall-spans (created_at..finished_at) the maximum overlap is also 6 concurrent runs at the same instant, with 154 runs overlapping. So despite many queued runs, the observed executing concurrency never exceeded 6 — consistent with the lane count and the 897 workspace-busy lines, and not with a memory/harness-capacity limit.

5. Ranked improvements

Ranked by measured impact. Items marked hypothesis have no direct measurement behind them.

  1. Reduce busy-workspace waiting (measured 7.34 h + 769 restart holds). Problem: 23 resolved waits, median 992 s, total 7.34 h, plus 769 server-restart holds. Smallest change: lower the lane's fixed 30 s busy-poll to a few seconds and/or keep a lane resident while a workspace is held, rather than relaunching the poll each cycle. Expected effect: cut the median multi-minute launch latency. Risk: more frequent polling and contention. Locus: orchestration. Evidence: lane log cadence and the 897 workspace busy lines.

  2. Finish or budget the malformed-result repair path. Problem: final_result_validation is 19 of 86 non-success runs; in 6 of them (55,81,174,179,183,221) the repair itself failed with isolated format repair failed: retained OpenCode session ses_…: context deadline exceeded, and in 11 (8,12,29,94,118,120,126,148,157,164,170) the result was judged unsafe to repair. Smallest change: give the isolated format-repair session its own longer deadline (or a streaming repair), so a repairable-but-slow result is recovered instead of failing the run. Expected effect: recover part of the 6 timeout-repair runs; the 11 "unsafe to repair" remain a separate problem. Risk: longer tail latency on already-slow runs. Locus: server code (internal/stage). Evidence: the six quoted diagnostics and the before/after drop (10.9 %→3.3 %) for this cluster.

  3. Cut repeat-visit cost in the review loop. Problem: repeat visits of the same stage cost 13.83 h of 82.53 h (16.8 %), concentrated in implementer (7.62 h) and reviewer (4.96 h); 5 runs (31,70,96,97,127) failed outright with Pipelines stopped reviewer after 3 visits on outcome rejected. Smallest change: identify which rejections were advisory-only (no must-fix finding) and stop looping on those, or reduce the reviewer max-visits for advisory findings. Expected effect: remove ~14 h of repeat work and the 5 exhausted-loop failures, if rejections are often non-blocking. Risk: a real defect could pass. Locus: pipeline definition (reviewer control). Evidence: repeat/single-pass split in §2 and the review_loop_exhausted ids.

  4. Make the ui_reviewer receive (or not require) captured artifacts. Problem: ui_reviewer blocked is a large part of the 23-run stage_outcome_blocked class (run 146: artifacts/146/3 is empty), and ui_reviewer p90 is 1030 s. Smallest change: whatever captures screenshots for a UI run must hand them to the ui_reviewer stage, and the stage should not block when no device is attached. Expected effect: fewer blocked UI runs. Risk: silently skipping visual verification. Locus: pipeline definition + orchestration (artifact handoff). Evidence: run 146 preview and the blocked ids; partly hypothesis — the artifact-empty condition is quoted for one run only, not measured across all 23.

  5. Triage the implementer/ui_reviewer "blocked" outcomes. Problem: stage_outcome_blocked is the single largest class (23/86, 26.7 %). Smallest change: have the stage distinguish "environment/device unavailable" (retryable) from "acceptance genuinely unmeetable" (terminal), so the retryable subset is retried instead of failing the run. Expected effect: a portion of 23 runs recovered. Risk: retry storms. Locus: pipeline definition / server code. Evidence: the 23 ids and their result_preview; the retryable/terminal split is hypothesis (not measured).

  6. Harden the harness transport path. Problem: 17 non-success runs are transport failures — 10 execution error, 6 context deadline exceeded, 1 final assistant readback had no completed result. Smallest change: add bounded retry/backoff around the OpenCode …/wait call for the generic "agent reported an execution error" case. Expected effect: recover the 10 execution-error runs if they are transient. Risk: retrying a genuinely broken session. Locus: server code (harness adapter). Evidence: the quoted diagnostics and ids. Whether the errors are transient is hypothesis.

  7. Stabilise merge resolution. Problem: merge-resolution pipelines contributed blocked implementers (176, 177, 178), final_result_validation (179, 183, 221), and resolver runs cost 1.43 h with 3 "did NOT land". Smallest change: investigate why the resolver's implementer blocks on a conflict workspace (e.g. missing conflict markers or write-path scope). Locus: merge-resolution pipeline definition. Evidence: merger.log lines and the run diagnostics; root cause hypothesis.

6. What could not be verified

  • server.log content. It is a request/response log and contains none of the stage diagnostics (0 matches for all diagnostic phrases). It could not be used to re-derive failures; the run records were used instead. To settle whether transport errors correlate with request latency, the server would need to log per-request duration (not currently present).
  • 3 non-terminal runs excluded. Runs 220, 222, 223 were running when the history and steps were captured; their incomplete steps (223 seq1, 222 seq2, 220 seq3) have empty finished_at and were excluded from all distributions. Settle by re-reading pipelines inspect after they finish.
  • 5 unresolved busy-wait episodes. Five workspace busy sequences were still open at log end (no matching launched line), so their final wait is unknown. Settle by re-parsing the lane logs once the lanes go idle.
  • Blocked-class cause distribution. Only one blocked run (146) has a quoted cause; the other 22 were classified by outcome=blocked plus stage name. A per-run result_preview pass would quantify the retryable vs terminal split.
  • After-side sample size. The 910a7aa comparison rests on 30 runs (3 running). A larger after-window sample is required before attributing the failure-rate drop to the change rather than to pipeline mix.
  • No system-changing measurements were attempted. No run was started, retried, cancelled or reconfigured; the server, ntfy service, lanes and devices were not touched, and the phone/emulator was not used. Measurements that would require such changes (e.g. injecting a busy workspace to time a wait, or forcing a transport error to test retry) were deliberately not performed.

Verification and limits

Commands and observed results:

  • pipelines ping --json → build/commit 910a7aa….
  • pipelines runs --json (1 call) → 223 runs; the run-status counts reconcile with pipelines stats --json (total 223, succeeded 134, failed 73, canceled 11, interrupted 2, active 3).
  • pipelines inspect <id> --json → 223 calls (one per run id), used for all step intervals and the taxonomy fallback.
  • Go verification, run with the repository's normal invocation from the module root on go1.27.1/darwin-arm64:
    • go build ./... → exit 0, no output.
    • go test ./... → all packages ok (cached), exit 0; the three cmd/* packages and internal/wstest have no test files. No Go source was changed; these commands confirm the worktree was left buildable.

Limits: the after-side sample is small; several classes did not recur after the boundary for reasons the data cannot separate from the format change; the blocked-class causes and the transience of transport errors are hypotheses, not measurements. Raw captures are under /Users/alan/.local/share/pipelines/data/artifacts/222/2/ (outside the repo).