c42a19f7d3
Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
98 lines
5.6 KiB
Markdown
98 lines
5.6 KiB
Markdown
# Real-provider reference workflows
|
|
|
|
These are small acceptance workflows, not production templates or performance
|
|
benchmarks. They use real OpenCode inference and independent checks; a model's
|
|
claim of success is not enough. They never publish to the normal Pipelines database.
|
|
The [initial live validation results](reference-workflow-results.md) record what
|
|
passed, the accounting defect discovered, its focused rerun and remaining limits.
|
|
|
|
| Workflow | What is exercised | Independent acceptance evidence |
|
|
|---|---|---|
|
|
| Incident handoff | Extract two incidents, then produce a report in a new stage | Exact IDs/severities, counts, unchanged handoff, retained sessions, one local success alert |
|
|
| Edit and verify | Read/edit a release record, then run a read-only comparison | Exact final file bytes, unchanged sentinel/reference files, harness-run `cmp`, retained read/edit/shell evidence, stage-local diff and no verifier edits |
|
|
| Missing input | Inspect a deliberately missing required file and take its blocked route | Failed run, blocked outcome, missing file remains missing, retained read attempt, one local failure alert |
|
|
| Accounting and budget | Generate a tiny answer with a one-token stage budget | Reported USD/tokens retained, budget-exceeded failure, one attempt and stopped session |
|
|
|
|
The suite permits six model stage attempts. Backend compatibility probes or format
|
|
repairs may add calls. Retries are disabled; each stage has a two-minute timeout
|
|
and the suite has a twelve-minute context deadline. **Provider charges may apply.**
|
|
The accounting workflow also sets a $0.25 reported-usage budget. This is not a hard
|
|
bill ceiling: an in-flight request can overshoot. Other workflows have no spending
|
|
budget so missing accounting does not mask basic inference/tool compatibility.
|
|
|
|
Notification delivery uses an isolated local HTTP endpoint, not your configured
|
|
topic or phone. This tests publisher requests and retained receipts, **not real
|
|
ntfy authentication/storage or phone reception**. The failure case exercises a
|
|
declared model `blocked` route, not a real provider outage. The tiny budget case
|
|
checks exhaustion of reported usage; it is not proof that interruption promptly
|
|
stops a long-running provider response or external shell process.
|
|
|
|
## Run with native credentials
|
|
|
|
Choose a model you are authorized to charge. Use a private, existing artifacts
|
|
directory. Do not use normal `go test ./...` to request live inference; opt-in is
|
|
explicit:
|
|
|
|
```sh
|
|
mkdir -p "$HOME/Library/Caches/pipelines/reference-validation"
|
|
PIPELINES_REFERENCE_LIVE=1 \
|
|
PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
|
|
PIPELINES_REFERENCE_DIR="$HOME/Library/Caches/pipelines/reference-validation" \
|
|
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m
|
|
```
|
|
|
|
`PIPELINES_OPENCODE` can select an executable. Without the live opt-in, the live
|
|
test skips; `TestReferenceDefinitions` validates all four definitions without
|
|
inference. These examples use the configured model selected for the initial
|
|
Docker trial; other providers/models need their own acceptance run.
|
|
|
|
## Run current source inside the existing Docker installation
|
|
|
|
The test binary contains the current server source. Copying it into a private
|
|
test directory avoids replacing/restarting the installed server or touching its
|
|
database. Choose `GOARCH=arm64` for container `uname -m` equal to `aarch64`, or
|
|
`GOARCH=amd64` for `x86_64`.
|
|
|
|
```sh
|
|
docker exec pipelines-pipelines-1 uname -m
|
|
GOOS=linux GOARCH=arm64 go test -c -o reference.test ./internal/server
|
|
docker exec pipelines-pipelines-1 mkdir -p /data/reference-validation
|
|
docker exec pipelines-pipelines-1 chmod 700 /data/reference-validation
|
|
docker cp reference.test pipelines-pipelines-1:/data/reference-validation/reference.test
|
|
docker exec \
|
|
-e PIPELINES_REFERENCE_LIVE=1 \
|
|
-e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
|
|
-e PIPELINES_REFERENCE_DIR=/data/reference-validation \
|
|
pipelines-pipelines-1 /data/reference-validation/reference.test \
|
|
-test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m
|
|
```
|
|
|
|
This uses the container's existing provider credentials. It does not copy host
|
|
credentials, log into a provider, rebuild the installation image or publish any
|
|
definitions to `/data/pipelines.db`.
|
|
|
|
## Retained evidence
|
|
|
|
The suite prints its uniquely created `pipelines-reference-*` directory. It retains:
|
|
|
|
- `runs.db`: isolated definitions, immutable runs/visits, diffs and receipts.
|
|
- One pinned Markdown definition and run/session JSON per workflow.
|
|
- Per-workflow fixture directories, including final edited files.
|
|
- `report.json`: model, verification verdicts, run IDs, durations and diagnostics.
|
|
|
|
Keep the test console log alongside these artifacts. A failed assertion does not
|
|
delete evidence; OpenCode sessions are also retained. A verified result applies
|
|
to that model/backend/environment and suite revision, not every provider or task.
|
|
Suite artifacts contain fixture prompts/results and should stay private. They
|
|
are not a reconciled provider bill. No result is labeled verified until its
|
|
independent assertions and session readback checks pass.
|
|
|
|
Unbudgeted tool-free V2 generation may leave an empty OpenCode message history:
|
|
the pipeline's definition, input/output and command audit are still retained.
|
|
Budgeted V2 calls use tracked, deny-all prompting so usage and messages are retained.
|
|
|
|
To recheck copied artifacts without inference, set
|
|
`PIPELINES_REFERENCE_REVIEW_DIR` and run `TestReferenceArtifactsReview`, selecting
|
|
the relevant case names with `-run` when combining artifacts from separate trials.
|
|
Missing cases explicitly skip; they are never counted as verified.
|