Files
pipelines/docs/reference-workflow-results.md
alan 05ae530442 Make Pipelines Docker-only with OpenCode and Pi
Install current verified harness releases inside Docker, preserve persistent accounts and settings, and provide separate interactive and pipeline model defaults with terminal onboarding and fuzzy pickers.

Add Docker-aware authoring transport, Pi stage execution, bounded workflow controls, reliable final-result readback and usage settlement, and finite short-lived CLI pipe draining. Keep dated installation evidence and packaged authoring guidance consistent.

Verified with a real Docker installation, both harness stages and interactive authors, the built-in plan/implement/review starter, independent host race/vet checks, installer tests, and packaged skill tests. Generated release receipts stay ignored in clean checkouts.
2026-10-11 23:00:25 +01:00

87 lines
4.7 KiB
Markdown

# Reference validation — October 10, 2026
Execution: existing Docker installation, Linux `aarch64`, OpenCode **v2.0.20**,
model **opencode-go/deepseek-v4.1-flash**. The test binaries were compiled from the
current checkout and run against a separate SQLite database and isolated fixtures.
No definitions were published to the normal server database and no phone alerts
were sent. Notification checks used an isolated local HTTP endpoint.
| Reference workflow | Live result | Duration |
|---|---|---|
| Incident extraction → report | Passed: exact incident evidence/counts, handoff, session readback and local success receipt | 6.63s |
| Edit → independently verify | Passed: exact file bytes, untouched fixtures, harness-run `cmp`, completed read/edit/shell evidence, retained stage diff | 14.60s |
| Missing required input | Passed: deliberate blocked outcome terminates as failed; missing file not invented; read attempt and local failure receipt retained | 4.21s |
| Accounting and one-token budget | Initially failed the acceptance test; passed after the accounting fix below | 2.43s on focused rerun |
## Defect found and fixed
The first accounting run succeeded despite a one-token limit. Its retained V2
session had **no messages and zero usage counters**, even though `/generate`
returned a valid result. Polling that session treated placeholder zeros as known
accounting and did not enforce the intended limit.
Budgeted V2 text calls now use tracked `/prompt` → bounded active-session polling
→ message readback, with **all tools denied**. Their messages are retained in the
stage audit, usage is read from the tracked session, and a completed
token-budgeted response with zero reported tokens fails closed. Native V1 handling
and unbudgeted text generation keep their existing paths.
A later reliability fix replaced the V2 `.../experimental/session/{id}/wait`
request, which timed out at the CLI's own ~six-minute limit (HTTP 499) on long
stages, with roughly one-second `GET /api/session/active` polls under the stage
context/deadline. That change has fake-CLI regression coverage in `internal/stage`;
it has not yet been exercised by a fresh live run.
The focused live rerun retained **952 reported tokens and $0.00016440**, failed
with `stage budget exceeded`, and stopped its owned session. This is reported
usage, not an independently reconciled provider bill. A one-token limit can be
overshot by a single in-flight request; this run demonstrates exhaustion handling,
not an exact hard cap or proof of immediate mid-stream cancellation.
The three previously passing workflows were not charged again. Their copied
artifacts were rechecked with current independent assertions; the fixed accounting
artifacts were rechecked separately. There has **not** been a fresh full-suite live
run of all four cases after the fix.
## Evidence and reproduction
Retained container directories for this trial:
- `/data/reference-validation/pipelines-reference-284076920`: first complete run.
- `/data/reference-validation/pipelines-reference-2247403140`: fixed accounting rerun.
Each has isolated SQLite history, pinned definitions, run/session JSON, fixture
files and `report.json`. Host copies are in the session's approved temporary
directory under `reference-first/` and `reference-accounting-fixed/`.
Commands that performed live inference:
```sh
docker exec -e PIPELINES_REFERENCE_LIVE=1 \
-e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
-e PIPELINES_REFERENCE_DIR=/data/reference-validation \
pipelines-pipelines-1 /data/reference-validation/reference-20261010.test \
-test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m
docker exec -e PIPELINES_REFERENCE_LIVE=1 \
-e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
-e PIPELINES_REFERENCE_DIR=/data/reference-validation \
pipelines-pipelines-1 /data/reference-validation/reference-20261010-fixed.test \
-test.run '^TestReferenceWorkflowsLive/accounting-and-budget$' -test.v -test.timeout 5m
```
`TestReferenceArtifactsReview` rechecks copied run JSON/fixtures without inference;
`PIPELINES_REFERENCE_REVIEW_DIR` selects the copy and `-run` selects its cases.
See [reference workflows](reference-workflows.md) for a fresh run and its opt-in,
cost, retention and safety boundaries.
## Remaining unverified
- Other models/providers and native V1 schema/tool compatibility.
- Real ntfy authentication/storage and actual phone reception.
- Actual provider outages, timeouts, long-stream interruption and retry behavior.
- General workflow correctness beyond these bounded fixtures.
The fix is in the source/test binaries, **not deployed to the installed Docker
server**. Rebuild the installation when ready to use it for ordinary runs.