Record and expose per-run model usage (tokens, split, cost)

Carry the usage a model stage already measures for its budget
(stage.Budget.Snapshot()) onto the step as typed evidence and total it in
one pass, rather than a second harness read, a transcript parse or an
output-length estimate. A shared budget across stage revisits records only
the visit's increment, and a revisit that observes no new report records no
usage instead of a fabricated known zero.

- protocol: StageUsage/RunUsage/TotalUsage and UsageTotals.
- pipeline: step_usage table + migration, COALESCEd into the one step
  projection, FinishStep persistence, one json_extract history aggregate,
  retention removes the rows with their run.
- server: stageUsage maps the budget snapshot, stamping per-stage model and
  usage for every model stage (including failures).
- cli: inspect/stats human and JSON output.
- budget/pi: retain the reported in/out split, mark a total-only (Pi) report
  split-unknown, and keep an unreported cost unknown (never zero).

Evidence and docs updated in the packaged skill references and docs/cli.md;
the live database has no step_usage rows yet (all history predates this), so
per-run figures are reported as unavailable, not zero.
This commit is contained in:
2026-10-17 20:40:56 +01:00
parent 1ae56b7d14
commit 3f5c11b7bf
16 changed files with 919 additions and 14 deletions
+28 -4
View File
@@ -65,8 +65,14 @@ The usage summary is aggregated over all stored history, including retry attempt
total/active runs, outcomes, success rate, average completed duration, pipeline
usage and a zero-filled 14-day daily activity chart (UTC). Success rate is
succeeded / (succeeded + failed); active, canceled and interrupted runs are excluded.
The average includes all runs with a finish timestamp. No model cost/token counts
are inferred from missing data. `pipelines stats --json` exposes these aggregates.
The average includes all runs with a finish timestamp. It also totals the reported
model usage across history: `tokens_total`, an in/out split only when every
token-reporting stage reported one, and the summed cost, plus counts of stages
that did or did not report a cost. A stage whose provider reported no cost is
counted as cost-unknown and never contributes a fabricated or zero cost, and no
cost/token count is inferred from output length or transcript text. The totals are
the sum of the per-run totals `inspect` shows. `pipelines stats --json` exposes
these aggregates.
Use `:status` on any TUI page to open server and ntfy information. Scroll with
arrows or `j/k`, PgUp/PgDn; `r` refreshes and Esc closes without changing the page.
@@ -404,7 +410,17 @@ because that stage never runs a model; `--model` on a pipeline with no model sta
is accepted and inert. The effective model reaches the stage and its
repair/formatter path from the same resolved definition, and each step's evidence
records the provider/model it actually ran with (`inspect <id> --json` exposes
per-step `provider`/`model`, and the run's graph shows the effective pair). The
per-step `provider`/`model`, and the run's graph shows the effective pair). Each
model step also records the usage the harness reported, carried from the same
accounting the stage already used for its budget: `usage.tokens_total`, the
`tokens_in`/`tokens_out` split when the harness reported it (`split_known`),
`cost_usd` when reported, and the explicit `tokens_known`/`cost_known`/
`split_known` markers. The run detail totals those stages as `usage`, so a
mixed-model run shows each stage's model, tokens and cost and the run total. A
stage whose provider reported no cost shows its tokens with `cost_known: false`
(never a fabricated or zero cost), and a harness that reports only a total (Pi)
shows `tokens_total` with the split marked unavailable rather than presenting the
total as input tokens. The
CLI is the deliverable; a TUI picker for choosing overrides is a documented
follow-up, not half-built.
@@ -622,6 +638,13 @@ retry potentially delivered notifications and do not accept model budgets.
Pinned definitions contain the configured limits. Run transcripts retain limits,
attempt counts and reported budget usage, also shown in the TUI's Evidence pane.
The same reported usage is recorded on the run as typed per-step evidence and
totalled for the run, so `inspect <id> --json` exposes tokens in/out, the total,
the cost when reported and the model each stage ran with. A stage revisited within
a run shares one budget, so each step records only that visit's increment of it
and the run total is not the cumulative budget counted again. Missing or unknown cost
or split is reported as unavailable (never zero or inferred), and a historical or
asked-for run recorded before this evidence existed simply omits `usage`.
Exact stage outputs are authoritative; a transcript over the 32 KiB retained
bound is stored as a valid, credential-redacted preview envelope that preserves
available stage limits and failure/format diagnostics and names the retained
@@ -709,7 +732,8 @@ anything already renamed. A removal never half-completes.
`artifacts/.retention-staging-<timestamp>-<pid>/` directory (a same-filesystem
rename), deletes the rows for those runs in one immediate write transaction
(`step_notifications`, `step_file_changes`, `step_control`, `step_artifacts`,
`step_verdicts`, `run_session_owners`, `run_steps`, `run_retries`, then
`step_verdicts`, `step_session_exports`, `step_usage`, `run_session_owners`,
`run_steps`, `run_retries`, then
`pipeline_runs`, batched at 500 ids), commits, and removes the staging directory.
A symlink at `<root>/<id>` is removed as a link without touching its target; a
regular file at `<root>/<id>` is refused. A symlinked artifacts root is followed