Rewrite format.md around turn files, the output.md order (# outcome, edge word, # handoff, detail), checks, configs, human stages and the land stage delivery gate (sh -c in a fresh detached worktree, clean checkouts incl. untracked, 64 KB output cap, opencode only). Align review guidance across practices, verification, orchestration and the feedback-loop example. Fix loop-limit semantics in control.md against runner.go, budget_tokens sizing, read for large inputs, blocked-else-error and one stage naming set. Move dated verification logs to docs/verification-history.md and brokered chat, TUI keys and harness internals to docs/operator-reference.md; label operator-only references in SKILL.md.
3.3 KiB
Verification and evidence
Structural validation is not inference, tool execution, correct task completion,
phone delivery or correct billing. Distinguish observed artifacts and command
results from model-written reports. State the command or check actually run and
say not tested for anything not executed. Never present mocked results as
live-provider evidence or a different backend/model as already validated. Dated
verification logs live in docs/verification-history.md in the source checkout.
Where verification happens in a pipeline
- The
delivery=truestage'schecksblock is the run's deterministic suite; the server runs it (references/format.md, Checks). - The implementer runs focused checks for what it touched and reports the exact commands and results.
- The reviewer reads the changed hunks (the server-captured changed-file list,
then
git diff --stat, then per-file diffs), judges the reported checks and names missing coverage. It runs no suites. Its verdict concerns the current change only. - Actual file bytes, command exit status, retained tool parts and server-captured diffs are stronger evidence than prose. Tool names mentioned in text do not prove tools executed. Do not accept a preceding stage's summary as proof of runtime behavior.
Do not add technology-specific checks unless the task or repository requires
them, and do not weaken an acceptance check just to get a passing run. Ask before
unrelated chargeable calls or notifications. provider-check must run on the
intended execution computer/container with its credentials; a client-side check
is not proof of a remote server's provider health.
Usage evidence
pipelines inspect RUN_ID --json exposes each model step's tokens_total,
tokens_in/tokens_out (only when split_known), cost_usd (only when
cost_known) and the model it ran with, plus the run's totalled usage;
pipelines stats --json adds history-wide totals. Token totals include cache
reads. An unreported cost or split is shown as unavailable, never zero; runs
recorded before usage tracking omit usage. Budgets are reported-usage limits,
not provider-side hard caps or a guaranteed bill ceiling; in-flight calls can
overshoot.
Orchestration verification habits
A run's status and a stage's summary are claims, not evidence. When the agent manages runs rather than making the change, its job includes checking what each run claims.
- Read the retained evidence yourself:
inspect RUN_ID --jsonfor the exact stage outputs, outcomes, diagnostics, file changes and artifacts. - Run the check. Where an item claims a build or tests, run the exact command and read its output. Never report a check you did not execute.
- Read the images. Open the screenshots a reviewer cites and confirm each is a
real image (magic bytes or content type), not a text file named
.png. - Compare the built artefact with what was delivered (md5 or equivalent), so a
stale or substituted file cannot pass (
references/delivery.md). - State "not tested" explicitly. A
succeededrun means its stages reported success, not that the operator's acceptance criteria were met. - Report failures as failures. Never re-run until something passes and then call that verified; record the failure and its category.