Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
5.6 KiB
Real-provider reference workflows
These are small acceptance workflows, not production templates or performance benchmarks. They use real OpenCode inference and independent checks; a model's claim of success is not enough. They never publish to the normal Pipelines database. The initial live validation results record what passed, the accounting defect discovered, its focused rerun and remaining limits.
| Workflow | What is exercised | Independent acceptance evidence |
|---|---|---|
| Incident handoff | Extract two incidents, then produce a report in a new stage | Exact IDs/severities, counts, unchanged handoff, retained sessions, one local success alert |
| Edit and verify | Read/edit a release record, then run a read-only comparison | Exact final file bytes, unchanged sentinel/reference files, harness-run cmp, retained read/edit/shell evidence, stage-local diff and no verifier edits |
| Missing input | Inspect a deliberately missing required file and take its blocked route | Failed run, blocked outcome, missing file remains missing, retained read attempt, one local failure alert |
| Accounting and budget | Generate a tiny answer with a one-token stage budget | Reported USD/tokens retained, budget-exceeded failure, one attempt and stopped session |
The suite permits six model stage attempts. Backend compatibility probes or format repairs may add calls. Retries are disabled; each stage has a two-minute timeout and the suite has a twelve-minute context deadline. Provider charges may apply. The accounting workflow also sets a $0.25 reported-usage budget. This is not a hard bill ceiling: an in-flight request can overshoot. Other workflows have no spending budget so missing accounting does not mask basic inference/tool compatibility.
Notification delivery uses an isolated local HTTP endpoint, not your configured
topic or phone. This tests publisher requests and retained receipts, not real
ntfy authentication/storage or phone reception. The failure case exercises a
declared model blocked route, not a real provider outage. The tiny budget case
checks exhaustion of reported usage; it is not proof that interruption promptly
stops a long-running provider response or external shell process.
Run with native credentials
Choose a model you are authorized to charge. Use a private, existing artifacts
directory. Do not use normal go test ./... to request live inference; opt-in is
explicit:
mkdir -p "$HOME/Library/Caches/pipelines/reference-validation"
PIPELINES_REFERENCE_LIVE=1 \
PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
PIPELINES_REFERENCE_DIR="$HOME/Library/Caches/pipelines/reference-validation" \
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m
PIPELINES_OPENCODE can select an executable. Without the live opt-in, the live
test skips; TestReferenceDefinitions validates all four definitions without
inference. These examples use the configured model selected for the initial
Docker trial; other providers/models need their own acceptance run.
Run current source inside the existing Docker installation
The test binary contains the current server source. Copying it into a private
test directory avoids replacing/restarting the installed server or touching its
database. Choose GOARCH=arm64 for container uname -m equal to aarch64, or
GOARCH=amd64 for x86_64.
docker exec pipelines-pipelines-1 uname -m
GOOS=linux GOARCH=arm64 go test -c -o reference.test ./internal/server
docker exec pipelines-pipelines-1 mkdir -p /data/reference-validation
docker exec pipelines-pipelines-1 chmod 700 /data/reference-validation
docker cp reference.test pipelines-pipelines-1:/data/reference-validation/reference.test
docker exec \
-e PIPELINES_REFERENCE_LIVE=1 \
-e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
-e PIPELINES_REFERENCE_DIR=/data/reference-validation \
pipelines-pipelines-1 /data/reference-validation/reference.test \
-test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m
This uses the container's existing provider credentials. It does not copy host
credentials, log into a provider, rebuild the installation image or publish any
definitions to /data/pipelines.db.
Retained evidence
The suite prints its uniquely created pipelines-reference-* directory. It retains:
runs.db: isolated definitions, immutable runs/visits, diffs and receipts.- One pinned Markdown definition and run/session JSON per workflow.
- Per-workflow fixture directories, including final edited files.
report.json: model, verification verdicts, run IDs, durations and diagnostics.
Keep the test console log alongside these artifacts. A failed assertion does not delete evidence; OpenCode sessions are also retained. A verified result applies to that model/backend/environment and suite revision, not every provider or task. Suite artifacts contain fixture prompts/results and should stay private. They are not a reconciled provider bill. No result is labeled verified until its independent assertions and session readback checks pass.
Unbudgeted tool-free V2 generation may leave an empty OpenCode message history: the pipeline's definition, input/output and command audit are still retained. Budgeted V2 calls use tracked, deny-all prompting so usage and messages are retained.
To recheck copied artifacts without inference, set
PIPELINES_REFERENCE_REVIEW_DIR and run TestReferenceArtifactsReview, selecting
the relevant case names with -run when combining artifacts from separate trials.
Missing cases explicitly skip; they are never counted as verified.