Files
pipelines/docs/reference-workflows.md
alan c42a19f7d3 Improve workflow UI, retain stage diffs, and add portable pipeline transfers
Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
2026-10-10 20:55:42 +01:00

98 lines
5.6 KiB
Markdown

# Real-provider reference workflows
These are small acceptance workflows, not production templates or performance
benchmarks. They use real OpenCode inference and independent checks; a model's
claim of success is not enough. They never publish to the normal Pipelines database.
The [initial live validation results](reference-workflow-results.md) record what
passed, the accounting defect discovered, its focused rerun and remaining limits.
| Workflow | What is exercised | Independent acceptance evidence |
|---|---|---|
| Incident handoff | Extract two incidents, then produce a report in a new stage | Exact IDs/severities, counts, unchanged handoff, retained sessions, one local success alert |
| Edit and verify | Read/edit a release record, then run a read-only comparison | Exact final file bytes, unchanged sentinel/reference files, harness-run `cmp`, retained read/edit/shell evidence, stage-local diff and no verifier edits |
| Missing input | Inspect a deliberately missing required file and take its blocked route | Failed run, blocked outcome, missing file remains missing, retained read attempt, one local failure alert |
| Accounting and budget | Generate a tiny answer with a one-token stage budget | Reported USD/tokens retained, budget-exceeded failure, one attempt and stopped session |
The suite permits six model stage attempts. Backend compatibility probes or format
repairs may add calls. Retries are disabled; each stage has a two-minute timeout
and the suite has a twelve-minute context deadline. **Provider charges may apply.**
The accounting workflow also sets a $0.25 reported-usage budget. This is not a hard
bill ceiling: an in-flight request can overshoot. Other workflows have no spending
budget so missing accounting does not mask basic inference/tool compatibility.
Notification delivery uses an isolated local HTTP endpoint, not your configured
topic or phone. This tests publisher requests and retained receipts, **not real
ntfy authentication/storage or phone reception**. The failure case exercises a
declared model `blocked` route, not a real provider outage. The tiny budget case
checks exhaustion of reported usage; it is not proof that interruption promptly
stops a long-running provider response or external shell process.
## Run with native credentials
Choose a model you are authorized to charge. Use a private, existing artifacts
directory. Do not use normal `go test ./...` to request live inference; opt-in is
explicit:
```sh
mkdir -p "$HOME/Library/Caches/pipelines/reference-validation"
PIPELINES_REFERENCE_LIVE=1 \
PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
PIPELINES_REFERENCE_DIR="$HOME/Library/Caches/pipelines/reference-validation" \
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m
```
`PIPELINES_OPENCODE` can select an executable. Without the live opt-in, the live
test skips; `TestReferenceDefinitions` validates all four definitions without
inference. These examples use the configured model selected for the initial
Docker trial; other providers/models need their own acceptance run.
## Run current source inside the existing Docker installation
The test binary contains the current server source. Copying it into a private
test directory avoids replacing/restarting the installed server or touching its
database. Choose `GOARCH=arm64` for container `uname -m` equal to `aarch64`, or
`GOARCH=amd64` for `x86_64`.
```sh
docker exec pipelines-pipelines-1 uname -m
GOOS=linux GOARCH=arm64 go test -c -o reference.test ./internal/server
docker exec pipelines-pipelines-1 mkdir -p /data/reference-validation
docker exec pipelines-pipelines-1 chmod 700 /data/reference-validation
docker cp reference.test pipelines-pipelines-1:/data/reference-validation/reference.test
docker exec \
-e PIPELINES_REFERENCE_LIVE=1 \
-e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
-e PIPELINES_REFERENCE_DIR=/data/reference-validation \
pipelines-pipelines-1 /data/reference-validation/reference.test \
-test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m
```
This uses the container's existing provider credentials. It does not copy host
credentials, log into a provider, rebuild the installation image or publish any
definitions to `/data/pipelines.db`.
## Retained evidence
The suite prints its uniquely created `pipelines-reference-*` directory. It retains:
- `runs.db`: isolated definitions, immutable runs/visits, diffs and receipts.
- One pinned Markdown definition and run/session JSON per workflow.
- Per-workflow fixture directories, including final edited files.
- `report.json`: model, verification verdicts, run IDs, durations and diagnostics.
Keep the test console log alongside these artifacts. A failed assertion does not
delete evidence; OpenCode sessions are also retained. A verified result applies
to that model/backend/environment and suite revision, not every provider or task.
Suite artifacts contain fixture prompts/results and should stay private. They
are not a reconciled provider bill. No result is labeled verified until its
independent assertions and session readback checks pass.
Unbudgeted tool-free V2 generation may leave an empty OpenCode message history:
the pipeline's definition, input/output and command audit are still retained.
Budgeted V2 calls use tracked, deny-all prompting so usage and messages are retained.
To recheck copied artifacts without inference, set
`PIPELINES_REFERENCE_REVIEW_DIR` and run `TestReferenceArtifactsReview`, selecting
the relevant case names with `-run` when combining artifacts from separate trials.
Missing cases explicitly skip; they are never counted as verified.