Files
pipelines/docs/reference-workflow-results.md
T
alan 05ae530442 Make Pipelines Docker-only with OpenCode and Pi
Install current verified harness releases inside Docker, preserve persistent accounts and settings, and provide separate interactive and pipeline model defaults with terminal onboarding and fuzzy pickers.

Add Docker-aware authoring transport, Pi stage execution, bounded workflow controls, reliable final-result readback and usage settlement, and finite short-lived CLI pipe draining. Keep dated installation evidence and packaged authoring guidance consistent.

Verified with a real Docker installation, both harness stages and interactive authors, the built-in plan/implement/review starter, independent host race/vet checks, installer tests, and packaged skill tests. Generated release receipts stay ignored in clean checkouts.
2026-10-11 23:00:25 +01:00

4.7 KiB

Reference validation — October 10, 2026

Execution: existing Docker installation, Linux aarch64, OpenCode v2.0.20, model opencode-go/deepseek-v4.1-flash. The test binaries were compiled from the current checkout and run against a separate SQLite database and isolated fixtures. No definitions were published to the normal server database and no phone alerts were sent. Notification checks used an isolated local HTTP endpoint.

Reference workflow Live result Duration
Incident extraction → report Passed: exact incident evidence/counts, handoff, session readback and local success receipt 6.63s
Edit → independently verify Passed: exact file bytes, untouched fixtures, harness-run cmp, completed read/edit/shell evidence, retained stage diff 14.60s
Missing required input Passed: deliberate blocked outcome terminates as failed; missing file not invented; read attempt and local failure receipt retained 4.21s
Accounting and one-token budget Initially failed the acceptance test; passed after the accounting fix below 2.43s on focused rerun

Defect found and fixed

The first accounting run succeeded despite a one-token limit. Its retained V2 session had no messages and zero usage counters, even though /generate returned a valid result. Polling that session treated placeholder zeros as known accounting and did not enforce the intended limit.

Budgeted V2 text calls now use tracked /prompt → bounded active-session polling → message readback, with all tools denied. Their messages are retained in the stage audit, usage is read from the tracked session, and a completed token-budgeted response with zero reported tokens fails closed. Native V1 handling and unbudgeted text generation keep their existing paths.

A later reliability fix replaced the V2 .../experimental/session/{id}/wait request, which timed out at the CLI's own ~six-minute limit (HTTP 499) on long stages, with roughly one-second GET /api/session/active polls under the stage context/deadline. That change has fake-CLI regression coverage in internal/stage; it has not yet been exercised by a fresh live run.

The focused live rerun retained 952 reported tokens and $0.00016440, failed with stage budget exceeded, and stopped its owned session. This is reported usage, not an independently reconciled provider bill. A one-token limit can be overshot by a single in-flight request; this run demonstrates exhaustion handling, not an exact hard cap or proof of immediate mid-stream cancellation.

The three previously passing workflows were not charged again. Their copied artifacts were rechecked with current independent assertions; the fixed accounting artifacts were rechecked separately. There has not been a fresh full-suite live run of all four cases after the fix.

Evidence and reproduction

Retained container directories for this trial:

  • /data/reference-validation/pipelines-reference-284076920: first complete run.
  • /data/reference-validation/pipelines-reference-2247403140: fixed accounting rerun.

Each has isolated SQLite history, pinned definitions, run/session JSON, fixture files and report.json. Host copies are in the session's approved temporary directory under reference-first/ and reference-accounting-fixed/.

Commands that performed live inference:

docker exec -e PIPELINES_REFERENCE_LIVE=1 \
  -e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
  -e PIPELINES_REFERENCE_DIR=/data/reference-validation \
  pipelines-pipelines-1 /data/reference-validation/reference-20261010.test \
  -test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m

docker exec -e PIPELINES_REFERENCE_LIVE=1 \
  -e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
  -e PIPELINES_REFERENCE_DIR=/data/reference-validation \
  pipelines-pipelines-1 /data/reference-validation/reference-20261010-fixed.test \
  -test.run '^TestReferenceWorkflowsLive/accounting-and-budget$' -test.v -test.timeout 5m

TestReferenceArtifactsReview rechecks copied run JSON/fixtures without inference; PIPELINES_REFERENCE_REVIEW_DIR selects the copy and -run selects its cases. See reference workflows for a fresh run and its opt-in, cost, retention and safety boundaries.

Remaining unverified

  • Other models/providers and native V1 schema/tool compatibility.
  • Real ntfy authentication/storage and actual phone reception.
  • Actual provider outages, timeouts, long-stream interruption and retry behavior.
  • General workflow correctness beyond these bounded fixtures.

The fix is in the source/test binaries, not deployed to the installed Docker server. Rebuild the installation when ready to use it for ordinary runs.