Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
5.9 KiB
Verification, accounting and real-provider evidence
Structural validation is not inference, tool execution, correct task completion, phone delivery or correct billing. Distinguish observed artifacts/command results from model-written reports. State the command/check actually run and say not tested for anything not executed. Never present mocked results as live-provider evidence or a different backend/model as already validated.
Authoring useful verification
Preserve explicit output handoffs and evidence fields. Implementer/reviewer stages can use appropriate permitted commands; do not accept a preceding stage's success summary as independent proof. Actual file bytes, command exit status, retained tool parts and server-captured diffs are stronger evidence than prose. A read failure can prove a deliberately missing input, but is not a successful read. Tool names mentioned in text do not prove tools executed.
Do not add technology-specific checks unless the task/repository requires them.
Keep runtime verification separate from authoring: ask before running project code,
making chargeable model calls, editing fixtures, or sending test notifications.
Do not weaken an acceptance check just to get a passing run.
provider-check must run on the intended execution computer/container with its
credentials; a client-side check is not proof of a remote server's provider health.
Budgets and backend caveats
Defaults remain two retries and one hour per stage attempt; a run-level wall-clock deadline is not added. Budgets are optional, positive and independent USD/token reported-usage limits shared across attempts, format repairs and repeat visits. Accounting failures stop budget-limited stages. In-flight calls can overshoot; these are not provider-side hard caps or a guaranteed bill ceiling. Reported USD may validly be zero; do not fabricate prices/accounting from output length.
A live V2 trial found /generate returned text while its retained session had no
messages and zero counters. Current source uses tracked /prompt → /wait →
message readback for budgeted V2 text-only calls, with all tools denied.
Completed token-budgeted responses reporting zero tokens fail closed. No new
definition flags or agent tools are required for this fix. Unbudgeted tool-free
V2 generation can still have empty backend message history; pipeline inputs,
results and command audit remain retained. Native V1 uses its existing path.
This describes current source behavior, not a guarantee that a user's installed server includes the fix. An updated skill/client is not deployment evidence; never rebuild/restart services or log into providers without specific authorization.
Dated live reference validation: 2026-10-10
Environment: Docker Linux aarch64, OpenCode v2.0.20, opencode-go/deepseek-v4.1-flash. Isolated databases/fixtures and local HTTP notification endpoints; normal published pipelines/repositories were not changed.
- Incident extraction → report: exact IDs/severities/counts, unchanged handoff, retained session readback and one local success receipt passed (6.63s).
- File edit → verification: exact final release-file bytes, unchanged fixtures,
independently executed
cmp, completed read/edit/shell evidence, retained edit diff and no verifier edits passed (14.60s). - Missing required input: blocked route ended failed; missing file was not invented; retained read attempt and one local failure receipt passed (4.21s).
- Accounting/one-token budget initially failed its acceptance check because session zeros let the run succeed. After the tracked-prompt fix, a focused live rerun passed: 952 reported tokens, $0.00016440, budget-exceeded failure, stopped owned session (2.43s). This is not an independently reconciled provider bill.
The earlier three workflows were rechecked from retained artifacts with current assertions, without more inference. There was no fresh full-suite live run of all four workflows after the fix. Do not generalize these results to all five starter templates, other models/providers, or compatible native schema/tool usage. Real ntfy authentication/storage, phone reception, provider outages/timeouts, long-stream interruption and promptness of cleanup were not tested by this trial. The trial ran rebuilt test binaries; it did not deploy the fix to the ordinary server.
Reference suite when explicitly requested
The source checkout's TestReferenceWorkflowsLive uses six model stage attempts
plus possible compatibility/format-repair calls. Retries are disabled and stages
have two-minute timeouts. It uses real existing credentials and retains private
runs.db, pinned definitions, fixture directories, run/session JSON and report.
The accounting case has a one-token/$0.25 budget; other cases have no spending
limit. Test notifications use a local endpoint, not the configured phone topic.
Live inference requires explicit PIPELINES_REFERENCE_LIVE=1, a chosen
PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL, and an existing private
PIPELINES_REFERENCE_DIR. Get consent for chargeable inference and confirm which
execution computer/account to use. Ordinary tests skip live inference. Do not
search the repository or run the suite during normal draft authoring.
If the user explicitly asks to run the suite and provides a source checkout:
PIPELINES_REFERENCE_LIVE=1 \
PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL \
PIPELINES_REFERENCE_DIR=/private/approved/artifacts \
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m
Docker credentials require running the correctly compiled test binary in that
container; do not copy host credentials or publish tests to the normal database.
TestReferenceDefinitions checks definitions without inference.
TestReferenceArtifactsReview rechecks copied run/fixture evidence without
inference using PIPELINES_REFERENCE_REVIEW_DIR; select only cases present in
that artifact directory. Missing cases skip and must not be reported as passed.