c42a19f7d3
Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
81 lines
4.3 KiB
Markdown
81 lines
4.3 KiB
Markdown
# Reference validation — October 10, 2026
|
|
|
|
Execution: existing Docker installation, Linux `aarch64`, OpenCode **v2.0.20**,
|
|
model **opencode-go/deepseek-v4.1-flash**. The test binaries were compiled from the
|
|
current checkout and run against a separate SQLite database and isolated fixtures.
|
|
No definitions were published to the normal server database and no phone alerts
|
|
were sent. Notification checks used an isolated local HTTP endpoint.
|
|
|
|
| Reference workflow | Live result | Duration |
|
|
|---|---|---|
|
|
| Incident extraction → report | Passed: exact incident evidence/counts, handoff, session readback and local success receipt | 6.63s |
|
|
| Edit → independently verify | Passed: exact file bytes, untouched fixtures, harness-run `cmp`, completed read/edit/shell evidence, retained stage diff | 14.60s |
|
|
| Missing required input | Passed: deliberate blocked outcome terminates as failed; missing file not invented; read attempt and local failure receipt retained | 4.21s |
|
|
| Accounting and one-token budget | Initially failed the acceptance test; passed after the accounting fix below | 2.43s on focused rerun |
|
|
|
|
## Defect found and fixed
|
|
|
|
The first accounting run succeeded despite a one-token limit. Its retained V2
|
|
session had **no messages and zero usage counters**, even though `/generate`
|
|
returned a valid result. Polling that session treated placeholder zeros as known
|
|
accounting and did not enforce the intended limit.
|
|
|
|
Budgeted V2 text calls now use tracked `/prompt` → `/wait` → message readback,
|
|
with **all tools denied**. Their messages are retained in the stage audit, usage
|
|
is read from the tracked session, and a completed token-budgeted response with
|
|
zero reported tokens fails closed. Native V1 handling and unbudgeted text
|
|
generation keep their existing paths.
|
|
|
|
The focused live rerun retained **952 reported tokens and $0.00016440**, failed
|
|
with `stage budget exceeded`, and stopped its owned session. This is reported
|
|
usage, not an independently reconciled provider bill. A one-token limit can be
|
|
overshot by a single in-flight request; this run demonstrates exhaustion handling,
|
|
not an exact hard cap or proof of immediate mid-stream cancellation.
|
|
|
|
The three previously passing workflows were not charged again. Their copied
|
|
artifacts were rechecked with current independent assertions; the fixed accounting
|
|
artifacts were rechecked separately. There has **not** been a fresh full-suite live
|
|
run of all four cases after the fix.
|
|
|
|
## Evidence and reproduction
|
|
|
|
Retained container directories for this trial:
|
|
|
|
- `/data/reference-validation/pipelines-reference-284076920`: first complete run.
|
|
- `/data/reference-validation/pipelines-reference-2247403140`: fixed accounting rerun.
|
|
|
|
Each has isolated SQLite history, pinned definitions, run/session JSON, fixture
|
|
files and `report.json`. Host copies are in the session's approved temporary
|
|
directory under `reference-first/` and `reference-accounting-fixed/`.
|
|
|
|
Commands that performed live inference:
|
|
|
|
```sh
|
|
docker exec -e PIPELINES_REFERENCE_LIVE=1 \
|
|
-e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
|
|
-e PIPELINES_REFERENCE_DIR=/data/reference-validation \
|
|
pipelines-pipelines-1 /data/reference-validation/reference-20261010.test \
|
|
-test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m
|
|
|
|
docker exec -e PIPELINES_REFERENCE_LIVE=1 \
|
|
-e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
|
|
-e PIPELINES_REFERENCE_DIR=/data/reference-validation \
|
|
pipelines-pipelines-1 /data/reference-validation/reference-20261010-fixed.test \
|
|
-test.run '^TestReferenceWorkflowsLive/accounting-and-budget$' -test.v -test.timeout 5m
|
|
```
|
|
|
|
`TestReferenceArtifactsReview` rechecks copied run JSON/fixtures without inference;
|
|
`PIPELINES_REFERENCE_REVIEW_DIR` selects the copy and `-run` selects its cases.
|
|
See [reference workflows](reference-workflows.md) for a fresh run and its opt-in,
|
|
cost, retention and safety boundaries.
|
|
|
|
## Remaining unverified
|
|
|
|
- Other models/providers and native V1 schema/tool compatibility.
|
|
- Real ntfy authentication/storage and actual phone reception.
|
|
- Actual provider outages, timeouts, long-stream interruption and retry behavior.
|
|
- General workflow correctness beyond these bounded fixtures.
|
|
|
|
The fix is in the source/test binaries, **not deployed to the installed Docker
|
|
server**. Rebuild the installation when ready to use it for ordinary runs.
|