Files
pipelines/docs/reference-workflows.md
alan c42a19f7d3 Improve workflow UI, retain stage diffs, and add portable pipeline transfers
Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
2026-10-10 20:55:42 +01:00

5.6 KiB

Real-provider reference workflows

These are small acceptance workflows, not production templates or performance benchmarks. They use real OpenCode inference and independent checks; a model's claim of success is not enough. They never publish to the normal Pipelines database. The initial live validation results record what passed, the accounting defect discovered, its focused rerun and remaining limits.

Workflow What is exercised Independent acceptance evidence
Incident handoff Extract two incidents, then produce a report in a new stage Exact IDs/severities, counts, unchanged handoff, retained sessions, one local success alert
Edit and verify Read/edit a release record, then run a read-only comparison Exact final file bytes, unchanged sentinel/reference files, harness-run cmp, retained read/edit/shell evidence, stage-local diff and no verifier edits
Missing input Inspect a deliberately missing required file and take its blocked route Failed run, blocked outcome, missing file remains missing, retained read attempt, one local failure alert
Accounting and budget Generate a tiny answer with a one-token stage budget Reported USD/tokens retained, budget-exceeded failure, one attempt and stopped session

The suite permits six model stage attempts. Backend compatibility probes or format repairs may add calls. Retries are disabled; each stage has a two-minute timeout and the suite has a twelve-minute context deadline. Provider charges may apply. The accounting workflow also sets a $0.25 reported-usage budget. This is not a hard bill ceiling: an in-flight request can overshoot. Other workflows have no spending budget so missing accounting does not mask basic inference/tool compatibility.

Notification delivery uses an isolated local HTTP endpoint, not your configured topic or phone. This tests publisher requests and retained receipts, not real ntfy authentication/storage or phone reception. The failure case exercises a declared model blocked route, not a real provider outage. The tiny budget case checks exhaustion of reported usage; it is not proof that interruption promptly stops a long-running provider response or external shell process.

Run with native credentials

Choose a model you are authorized to charge. Use a private, existing artifacts directory. Do not use normal go test ./... to request live inference; opt-in is explicit:

mkdir -p "$HOME/Library/Caches/pipelines/reference-validation"
PIPELINES_REFERENCE_LIVE=1 \
PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
PIPELINES_REFERENCE_DIR="$HOME/Library/Caches/pipelines/reference-validation" \
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m

PIPELINES_OPENCODE can select an executable. Without the live opt-in, the live test skips; TestReferenceDefinitions validates all four definitions without inference. These examples use the configured model selected for the initial Docker trial; other providers/models need their own acceptance run.

Run current source inside the existing Docker installation

The test binary contains the current server source. Copying it into a private test directory avoids replacing/restarting the installed server or touching its database. Choose GOARCH=arm64 for container uname -m equal to aarch64, or GOARCH=amd64 for x86_64.

docker exec pipelines-pipelines-1 uname -m
GOOS=linux GOARCH=arm64 go test -c -o reference.test ./internal/server
docker exec pipelines-pipelines-1 mkdir -p /data/reference-validation
docker exec pipelines-pipelines-1 chmod 700 /data/reference-validation
docker cp reference.test pipelines-pipelines-1:/data/reference-validation/reference.test
docker exec \
  -e PIPELINES_REFERENCE_LIVE=1 \
  -e PIPELINES_REFERENCE_MODEL=opencode-go/deepseek-v4.1-flash \
  -e PIPELINES_REFERENCE_DIR=/data/reference-validation \
  pipelines-pipelines-1 /data/reference-validation/reference.test \
  -test.run '^TestReferenceWorkflowsLive$' -test.v -test.timeout 15m

This uses the container's existing provider credentials. It does not copy host credentials, log into a provider, rebuild the installation image or publish any definitions to /data/pipelines.db.

Retained evidence

The suite prints its uniquely created pipelines-reference-* directory. It retains:

  • runs.db: isolated definitions, immutable runs/visits, diffs and receipts.
  • One pinned Markdown definition and run/session JSON per workflow.
  • Per-workflow fixture directories, including final edited files.
  • report.json: model, verification verdicts, run IDs, durations and diagnostics.

Keep the test console log alongside these artifacts. A failed assertion does not delete evidence; OpenCode sessions are also retained. A verified result applies to that model/backend/environment and suite revision, not every provider or task. Suite artifacts contain fixture prompts/results and should stay private. They are not a reconciled provider bill. No result is labeled verified until its independent assertions and session readback checks pass.

Unbudgeted tool-free V2 generation may leave an empty OpenCode message history: the pipeline's definition, input/output and command audit are still retained. Budgeted V2 calls use tracked, deny-all prompting so usage and messages are retained.

To recheck copied artifacts without inference, set PIPELINES_REFERENCE_REVIEW_DIR and run TestReferenceArtifactsReview, selecting the relevant case names with -run when combining artifacts from separate trials. Missing cases explicitly skip; they are never counted as verified.