Record and expose per-run model usage (tokens, split, cost)
Carry the usage a model stage already measures for its budget (stage.Budget.Snapshot()) onto the step as typed evidence and total it in one pass, rather than a second harness read, a transcript parse or an output-length estimate. A shared budget across stage revisits records only the visit's increment, and a revisit that observes no new report records no usage instead of a fabricated known zero. - protocol: StageUsage/RunUsage/TotalUsage and UsageTotals. - pipeline: step_usage table + migration, COALESCEd into the one step projection, FinishStep persistence, one json_extract history aggregate, retention removes the rows with their run. - server: stageUsage maps the budget snapshot, stamping per-stage model and usage for every model stage (including failures). - cli: inspect/stats human and JSON output. - budget/pi: retain the reported in/out split, mark a total-only (Pi) report split-unknown, and keep an unreported cost unknown (never zero). Evidence and docs updated in the packaged skill references and docs/cli.md; the live database has no step_usage rows yet (all history predates this), so per-run figures are reported as unavailable, not zero.
This commit is contained in:
+28
-4
@@ -65,8 +65,14 @@ The usage summary is aggregated over all stored history, including retry attempt
|
||||
total/active runs, outcomes, success rate, average completed duration, pipeline
|
||||
usage and a zero-filled 14-day daily activity chart (UTC). Success rate is
|
||||
succeeded / (succeeded + failed); active, canceled and interrupted runs are excluded.
|
||||
The average includes all runs with a finish timestamp. No model cost/token counts
|
||||
are inferred from missing data. `pipelines stats --json` exposes these aggregates.
|
||||
The average includes all runs with a finish timestamp. It also totals the reported
|
||||
model usage across history: `tokens_total`, an in/out split only when every
|
||||
token-reporting stage reported one, and the summed cost, plus counts of stages
|
||||
that did or did not report a cost. A stage whose provider reported no cost is
|
||||
counted as cost-unknown and never contributes a fabricated or zero cost, and no
|
||||
cost/token count is inferred from output length or transcript text. The totals are
|
||||
the sum of the per-run totals `inspect` shows. `pipelines stats --json` exposes
|
||||
these aggregates.
|
||||
|
||||
Use `:status` on any TUI page to open server and ntfy information. Scroll with
|
||||
arrows or `j/k`, PgUp/PgDn; `r` refreshes and Esc closes without changing the page.
|
||||
@@ -404,7 +410,17 @@ because that stage never runs a model; `--model` on a pipeline with no model sta
|
||||
is accepted and inert. The effective model reaches the stage and its
|
||||
repair/formatter path from the same resolved definition, and each step's evidence
|
||||
records the provider/model it actually ran with (`inspect <id> --json` exposes
|
||||
per-step `provider`/`model`, and the run's graph shows the effective pair). The
|
||||
per-step `provider`/`model`, and the run's graph shows the effective pair). Each
|
||||
model step also records the usage the harness reported, carried from the same
|
||||
accounting the stage already used for its budget: `usage.tokens_total`, the
|
||||
`tokens_in`/`tokens_out` split when the harness reported it (`split_known`),
|
||||
`cost_usd` when reported, and the explicit `tokens_known`/`cost_known`/
|
||||
`split_known` markers. The run detail totals those stages as `usage`, so a
|
||||
mixed-model run shows each stage's model, tokens and cost and the run total. A
|
||||
stage whose provider reported no cost shows its tokens with `cost_known: false`
|
||||
(never a fabricated or zero cost), and a harness that reports only a total (Pi)
|
||||
shows `tokens_total` with the split marked unavailable rather than presenting the
|
||||
total as input tokens. The
|
||||
CLI is the deliverable; a TUI picker for choosing overrides is a documented
|
||||
follow-up, not half-built.
|
||||
|
||||
@@ -622,6 +638,13 @@ retry potentially delivered notifications and do not accept model budgets.
|
||||
|
||||
Pinned definitions contain the configured limits. Run transcripts retain limits,
|
||||
attempt counts and reported budget usage, also shown in the TUI's Evidence pane.
|
||||
The same reported usage is recorded on the run as typed per-step evidence and
|
||||
totalled for the run, so `inspect <id> --json` exposes tokens in/out, the total,
|
||||
the cost when reported and the model each stage ran with. A stage revisited within
|
||||
a run shares one budget, so each step records only that visit's increment of it
|
||||
and the run total is not the cumulative budget counted again. Missing or unknown cost
|
||||
or split is reported as unavailable (never zero or inferred), and a historical or
|
||||
asked-for run recorded before this evidence existed simply omits `usage`.
|
||||
Exact stage outputs are authoritative; a transcript over the 32 KiB retained
|
||||
bound is stored as a valid, credential-redacted preview envelope that preserves
|
||||
available stage limits and failure/format diagnostics and names the retained
|
||||
@@ -709,7 +732,8 @@ anything already renamed. A removal never half-completes.
|
||||
`artifacts/.retention-staging-<timestamp>-<pid>/` directory (a same-filesystem
|
||||
rename), deletes the rows for those runs in one immediate write transaction
|
||||
(`step_notifications`, `step_file_changes`, `step_control`, `step_artifacts`,
|
||||
`step_verdicts`, `run_session_owners`, `run_steps`, `run_retries`, then
|
||||
`step_verdicts`, `step_session_exports`, `step_usage`, `run_session_owners`,
|
||||
`run_steps`, `run_retries`, then
|
||||
`pipeline_runs`, batched at 500 ids), commits, and removes the staging directory.
|
||||
A symlink at `<root>/<id>` is removed as a link without touching its target; a
|
||||
regular file at `<root>/<id>` is refused. A symlinked artifacts root is followed
|
||||
|
||||
Reference in New Issue
Block a user