5090f1cf4e
Rewrite format.md around turn files, the output.md order (# outcome, edge word, # handoff, detail), checks, configs, human stages and the land stage delivery gate (sh -c in a fresh detached worktree, clean checkouts incl. untracked, 64 KB output cap, opencode only). Align review guidance across practices, verification, orchestration and the feedback-loop example. Fix loop-limit semantics in control.md against runner.go, budget_tokens sizing, read for large inputs, blocked-else-error and one stage naming set. Move dated verification logs to docs/verification-history.md and brokered chat, TUI keys and harness internals to docs/operator-reference.md; label operator-only references in SKILL.md.
60 lines
3.3 KiB
Markdown
60 lines
3.3 KiB
Markdown
# Verification and evidence
|
|
|
|
Structural validation is not inference, tool execution, correct task completion,
|
|
phone delivery or correct billing. Distinguish observed artifacts and command
|
|
results from model-written reports. State the command or check actually run and
|
|
say **not tested** for anything not executed. Never present mocked results as
|
|
live-provider evidence or a different backend/model as already validated. Dated
|
|
verification logs live in `docs/verification-history.md` in the source checkout.
|
|
|
|
## Where verification happens in a pipeline
|
|
|
|
- The `delivery=true` stage's `checks` block is the run's deterministic suite;
|
|
the server runs it (`references/format.md`, Checks).
|
|
- The implementer runs focused checks for what it touched and reports the exact
|
|
commands and results.
|
|
- The reviewer reads the changed hunks (the server-captured changed-file list,
|
|
then `git diff --stat`, then per-file diffs), judges the reported checks and
|
|
names missing coverage. It runs no suites. Its verdict concerns the current
|
|
change only.
|
|
- Actual file bytes, command exit status, retained tool parts and server-captured
|
|
diffs are stronger evidence than prose. Tool names mentioned in text do not prove
|
|
tools executed. Do not accept a preceding stage's summary as proof of runtime
|
|
behavior.
|
|
|
|
Do not add technology-specific checks unless the task or repository requires
|
|
them, and do not weaken an acceptance check just to get a passing run. Ask before
|
|
unrelated chargeable calls or notifications. `provider-check` must run on the
|
|
intended execution computer/container with its credentials; a client-side check
|
|
is not proof of a remote server's provider health.
|
|
|
|
## Usage evidence
|
|
|
|
`pipelines inspect RUN_ID --json` exposes each model step's `tokens_total`,
|
|
`tokens_in`/`tokens_out` (only when `split_known`), `cost_usd` (only when
|
|
`cost_known`) and the model it ran with, plus the run's totalled `usage`;
|
|
`pipelines stats --json` adds history-wide totals. Token totals include cache
|
|
reads. An unreported cost or split is shown as unavailable, never zero; runs
|
|
recorded before usage tracking omit `usage`. Budgets are reported-usage limits,
|
|
**not provider-side hard caps or a guaranteed bill ceiling**; in-flight calls can
|
|
overshoot.
|
|
|
|
## Orchestration verification habits
|
|
|
|
A run's status and a stage's summary are **claims**, not evidence. When the agent
|
|
manages runs rather than making the change, its job includes checking what each
|
|
run claims.
|
|
|
|
- **Read the retained evidence yourself**: `inspect RUN_ID --json` for the exact
|
|
stage outputs, outcomes, diagnostics, file changes and artifacts.
|
|
- **Run the check.** Where an item claims a build or tests, run the exact command
|
|
and read its output. Never report a check you did not execute.
|
|
- **Read the images.** Open the screenshots a reviewer cites and confirm each is a
|
|
real image (magic bytes or content type), not a text file named `.png`.
|
|
- **Compare the built artefact with what was delivered** (md5 or equivalent), so a
|
|
stale or substituted file cannot pass (`references/delivery.md`).
|
|
- **State "not tested" explicitly.** A `succeeded` run means its stages reported
|
|
success, not that the operator's acceptance criteria were met.
|
|
- **Report failures as failures.** Never re-run until something passes and then
|
|
call that verified; record the failure and its category.
|