Files
pipelines/internal/author/skill/references/verification.md
T
alan 5090f1cf4e Pipelines skill: outcome-first output, aligned review and loop guidance, operator material moved to docs
Rewrite format.md around turn files, the output.md order (# outcome, edge word, # handoff, detail), checks, configs, human stages and the land stage delivery gate (sh -c in a fresh detached worktree, clean checkouts incl. untracked, 64 KB output cap, opencode only). Align review guidance across practices, verification, orchestration and the feedback-loop example. Fix loop-limit semantics in control.md against runner.go, budget_tokens sizing, read for large inputs, blocked-else-error and one stage naming set. Move dated verification logs to docs/verification-history.md and brokered chat, TUI keys and harness internals to docs/operator-reference.md; label operator-only references in SKILL.md.
2026-11-04 17:27:01 +00:00

60 lines
3.3 KiB
Markdown

# Verification and evidence
Structural validation is not inference, tool execution, correct task completion,
phone delivery or correct billing. Distinguish observed artifacts and command
results from model-written reports. State the command or check actually run and
say **not tested** for anything not executed. Never present mocked results as
live-provider evidence or a different backend/model as already validated. Dated
verification logs live in `docs/verification-history.md` in the source checkout.
## Where verification happens in a pipeline
- The `delivery=true` stage's `checks` block is the run's deterministic suite;
the server runs it (`references/format.md`, Checks).
- The implementer runs focused checks for what it touched and reports the exact
commands and results.
- The reviewer reads the changed hunks (the server-captured changed-file list,
then `git diff --stat`, then per-file diffs), judges the reported checks and
names missing coverage. It runs no suites. Its verdict concerns the current
change only.
- Actual file bytes, command exit status, retained tool parts and server-captured
diffs are stronger evidence than prose. Tool names mentioned in text do not prove
tools executed. Do not accept a preceding stage's summary as proof of runtime
behavior.
Do not add technology-specific checks unless the task or repository requires
them, and do not weaken an acceptance check just to get a passing run. Ask before
unrelated chargeable calls or notifications. `provider-check` must run on the
intended execution computer/container with its credentials; a client-side check
is not proof of a remote server's provider health.
## Usage evidence
`pipelines inspect RUN_ID --json` exposes each model step's `tokens_total`,
`tokens_in`/`tokens_out` (only when `split_known`), `cost_usd` (only when
`cost_known`) and the model it ran with, plus the run's totalled `usage`;
`pipelines stats --json` adds history-wide totals. Token totals include cache
reads. An unreported cost or split is shown as unavailable, never zero; runs
recorded before usage tracking omit `usage`. Budgets are reported-usage limits,
**not provider-side hard caps or a guaranteed bill ceiling**; in-flight calls can
overshoot.
## Orchestration verification habits
A run's status and a stage's summary are **claims**, not evidence. When the agent
manages runs rather than making the change, its job includes checking what each
run claims.
- **Read the retained evidence yourself**: `inspect RUN_ID --json` for the exact
stage outputs, outcomes, diagnostics, file changes and artifacts.
- **Run the check.** Where an item claims a build or tests, run the exact command
and read its output. Never report a check you did not execute.
- **Read the images.** Open the screenshots a reviewer cites and confirm each is a
real image (magic bytes or content type), not a text file named `.png`.
- **Compare the built artefact with what was delivered** (md5 or equivalent), so a
stale or substituted file cannot pass (`references/delivery.md`).
- **State "not tested" explicitly.** A `succeeded` run means its stages reported
success, not that the operator's acceptance criteria were met.
- **Report failures as failures.** Never re-run until something passes and then
call that verified; record the failure and its category.