Files
pipelines/docs/reference-workflow-results.md
T
alan c42a19f7d3 Improve workflow UI, retain stage diffs, and add portable pipeline transfers
Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
2026-10-10 20:55:42 +01:00

4.3 KiB

Reference validation — October 10, 2026

Execution: existing Docker installation, Linux aarch64, OpenCode v2.0.20, model opencode-go/deepseek-v4.1-flash. The test binaries were compiled from the current checkout and run against a separate SQLite database and isolated fixtures. No definitions were published to the normal server database and no phone alerts were sent. Notification checks used an isolated local HTTP endpoint.

Reference workflow Live result Duration
Incident extraction → report Passed: exact incident evidence/counts, handoff, session readback and local success receipt 6.63s
Edit → independently verify Passed: exact file bytes, untouched fixtures, harness-run cmp, completed read/edit/shell evidence, retained stage diff 14.60s
Missing required input Passed: deliberate blocked outcome terminates as failed; missing file not invented; read attempt and local failure receipt retained 4.21s
Accounting and one-token budget Initially failed the acceptance test; passed after the accounting fix below 2.43s on focused rerun

Defect found and fixed

The first accounting run succeeded despite a one-token limit. Its retained V2 session had no messages and zero usage counters, even though /generate returned a valid result. Polling that session treated placeholder zeros as known accounting and did not enforce the intended limit.

Budgeted V2 text calls now use tracked /prompt → /wait → message readback, with all tools denied. Their messages are retained in the stage audit, usage is read from the tracked session, and a completed token-budgeted response with zero reported tokens fails closed. Native V1 handling and unbudgeted text generation keep their existing paths.

The focused live rerun retained 952 reported tokens and $0.00016440, failed with stage budget exceeded, and stopped its owned session. This is reported usage, not an independently reconciled provider bill. A one-token limit can be overshot by a single in-flight request; this run demonstrates exhaustion handling, not an exact hard cap or proof of immediate mid-stream cancellation.

The three previously passing workflows were not charged again. Their copied artifacts were rechecked with current independent assertions; the fixed accounting artifacts were rechecked separately. There has not been a fresh full-suite live run of all four cases after the fix.

Evidence and reproduction

Retained container directories for this trial:

  • /data/reference-validation/pipelines-reference-284076920: first complete run.
  • /data/reference-validation/pipelines-reference-2247403140: fixed accounting rerun.

Each has isolated SQLite history, pinned definitions, run/session JSON, fixture files and report.json. Host copies are in the session's approved temporary directory under reference-first/ and reference-accounting-fixed/.

Commands that performed live inference:

docker exec -e PIPELINES_REFERENCE_LIVE=1 \
  -e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
  -e PIPELINES_REFERENCE_DIR=/data/reference-validation \
  pipelines-pipelines-1 /data/reference-validation/reference-20261010.test \
  -test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m

docker exec -e PIPELINES_REFERENCE_LIVE=1 \
  -e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
  -e PIPELINES_REFERENCE_DIR=/data/reference-validation \
  pipelines-pipelines-1 /data/reference-validation/reference-20261010-fixed.test \
  -test.run '^TestReferenceWorkflowsLive/accounting-and-budget$' -test.v -test.timeout 5m

TestReferenceArtifactsReview rechecks copied run JSON/fixtures without inference; PIPELINES_REFERENCE_REVIEW_DIR selects the copy and -run selects its cases. See reference workflows for a fresh run and its opt-in, cost, retention and safety boundaries.

Remaining unverified

  • Other models/providers and native V1 schema/tool compatibility.
  • Real ntfy authentication/storage and actual phone reception.
  • Actual provider outages, timeouts, long-stream interruption and retry behavior.
  • General workflow correctness beyond these bounded fixtures.

The fix is in the source/test binaries, not deployed to the installed Docker server. Rebuild the installation when ready to use it for ordinary runs.