Files
alan 5090f1cf4e Pipelines skill: outcome-first output, aligned review and loop guidance, operator material moved to docs
Rewrite format.md around turn files, the output.md order (# outcome, edge word, # handoff, detail), checks, configs, human stages and the land stage delivery gate (sh -c in a fresh detached worktree, clean checkouts incl. untracked, 64 KB output cap, opencode only). Align review guidance across practices, verification, orchestration and the feedback-loop example. Fix loop-limit semantics in control.md against runner.go, budget_tokens sizing, read for large inputs, blocked-else-error and one stage naming set. Move dated verification logs to docs/verification-history.md and brokered chat, TUI keys and harness internals to docs/operator-reference.md; label operator-only references in SKILL.md.
2026-11-04 17:27:01 +00:00

3.3 KiB

Verification and evidence

Structural validation is not inference, tool execution, correct task completion, phone delivery or correct billing. Distinguish observed artifacts and command results from model-written reports. State the command or check actually run and say not tested for anything not executed. Never present mocked results as live-provider evidence or a different backend/model as already validated. Dated verification logs live in docs/verification-history.md in the source checkout.

Where verification happens in a pipeline

  • The delivery=true stage's checks block is the run's deterministic suite; the server runs it (references/format.md, Checks).
  • The implementer runs focused checks for what it touched and reports the exact commands and results.
  • The reviewer reads the changed hunks (the server-captured changed-file list, then git diff --stat, then per-file diffs), judges the reported checks and names missing coverage. It runs no suites. Its verdict concerns the current change only.
  • Actual file bytes, command exit status, retained tool parts and server-captured diffs are stronger evidence than prose. Tool names mentioned in text do not prove tools executed. Do not accept a preceding stage's summary as proof of runtime behavior.

Do not add technology-specific checks unless the task or repository requires them, and do not weaken an acceptance check just to get a passing run. Ask before unrelated chargeable calls or notifications. provider-check must run on the intended execution computer/container with its credentials; a client-side check is not proof of a remote server's provider health.

Usage evidence

pipelines inspect RUN_ID --json exposes each model step's tokens_total, tokens_in/tokens_out (only when split_known), cost_usd (only when cost_known) and the model it ran with, plus the run's totalled usage; pipelines stats --json adds history-wide totals. Token totals include cache reads. An unreported cost or split is shown as unavailable, never zero; runs recorded before usage tracking omit usage. Budgets are reported-usage limits, not provider-side hard caps or a guaranteed bill ceiling; in-flight calls can overshoot.

Orchestration verification habits

A run's status and a stage's summary are claims, not evidence. When the agent manages runs rather than making the change, its job includes checking what each run claims.

  • Read the retained evidence yourself: inspect RUN_ID --json for the exact stage outputs, outcomes, diagnostics, file changes and artifacts.
  • Run the check. Where an item claims a build or tests, run the exact command and read its output. Never report a check you did not execute.
  • Read the images. Open the screenshots a reviewer cites and confirm each is a real image (magic bytes or content type), not a text file named .png.
  • Compare the built artefact with what was delivered (md5 or equivalent), so a stale or substituted file cannot pass (references/delivery.md).
  • State "not tested" explicitly. A succeeded run means its stages reported success, not that the operator's acceptance criteria were met.
  • Report failures as failures. Never re-run until something passes and then call that verified; record the failure and its category.