retention: explicit run-history and artifact retention

Add `pipelines retention report` (strictly read-only) and
`pipelines retention apply` (explicit, approval-gated removal) with an age
and/or count policy, protected-run rules, atomic single-transaction removal
and honest database space reclaim. Nothing prunes automatically.
This commit is contained in:
2026-10-17 15:45:46 +01:00
parent b2e920a150
commit 14175eba70
15 changed files with 2127 additions and 12 deletions
+106
View File
@@ -269,6 +269,7 @@ Submitting activates included schedules, so review them before publishing.
| Connection and phone settings | `connection --json`, `ping --json` |
| Server/ntfy status popup | `status --json` |
| Usage statistics/charts | `stats --json` |
| Run-history retention | `retention report --keep N --json`, `retention apply --keep N --yes` |
| Legacy operational diagnostics | managed `logs`, `doctor` |
The TUI does not spawn CLI processes for each request. Both front ends share
@@ -607,6 +608,111 @@ for panning and `c` to center. Secondary progress/input/evidence/execution secti
are collapsed initially. Small inactive graph previews collapse instead of showing
severed boxes. See [UI/UX review](ui-ux-review.md) for screen-by-screen feedback.
## Run retention
Nothing prunes run history automatically. `pipelines retention report` and
`pipelines retention apply` are the only commands that remove run history or run
artifacts, they are local and operator-controlled, and they never run as a side
effect of any other command, server startup, `newRunner`, the scheduler,
`list_runs`, `stats`, `runs` or `inspect`. Pipeline definitions and their
immutable versions are never touched: retention is about run history and the
artifacts runs produced.
```sh
# Read-only: show exactly what a policy would remove.
pipelines retention report --db ~/.local/share/pipelines/data/pipelines.db --keep 200 --json
# Explicit removal of that same set (artifacts go with their run).
pipelines retention apply --db ~/.local/share/pipelines/data/pipelines.db --keep 200 --yes --json
```
Both commands accept `--older-than DURATION`, `--keep N`, `--db FILE`,
`--artifacts DIR` and `--json`; `apply` also accepts `--yes`. At least one policy
flag is required; an unknown subcommand, a negative count and a non-positive age
are refused. `--db` must name an existing database and is never created. With only
`--db`, the artifacts root defaults to `artifacts/` beside the database, matching
`cmd/pipelines-server/main.go`; `--artifacts` overrides it. The commands are
dispatched before server URL resolution, so they work without a running server
(for example `--url not-a-server`) and never require, restart or redeploy the
deployed server.
### Policy
- `--older-than` selects runs created **strictly before** `now - DURATION`. The
age is a rolling UTC duration, not a calendar-day boundary. It accepts Go
durations plus a count of days or weeks (`720h`, `30d`, `12w`).
- `--keep N` retains the newest `N` runs by run id and selects ranks `N+1` and
beyond. `--keep 0` selects every eligible run; a count larger than the history
selects none.
- Both together are the conservative **intersection**: a run is selected only
when it is both older than the age bound and outside the newest `N`.
### Protected runs
A policy-selected run is nonetheless never removed when it:
- is `queued` or `running`;
- is `waiting` on a person (a `waiting` run or a `waiting` step);
- has any step with `agent_active=1` (cleanup still active); or
- is referenced by a run that will remain through retry provenance
(`run_retries.original_run_id`), propagated transitively along a retry chain.
If both an original and its retry are selected, no retained link remains and
both may be removed.
Protected runs are reported with their reason and their artifacts are never
counted as reclaimable. Free-text run ids in prompts are not treated as
dependencies; retry provenance is the only structured cross-run dependency the
store records.
If a selected run becomes protected between the report and the delete — a new
retry reference, a waiting/active state or new agent activity — `apply` aborts
with the run ids and reasons, deletes no rows or artifacts, and un-stages
anything already renamed. A removal never half-completes.
### What apply removes and reclaims
`apply` stages only the would-remove run directories into one
`artifacts/.retention-staging-<timestamp>-<pid>/` directory (a same-filesystem
rename), deletes the rows for those runs in one immediate write transaction
(`step_notifications`, `step_file_changes`, `step_control`, `step_artifacts`,
`step_verdicts`, `run_session_owners`, `run_steps`, `run_retries`, then
`pipeline_runs`, batched at 500 ids), commits, and removes the staging directory.
A symlink at `<root>/<id>` is removed as a link without touching its target; a
regular file at `<root>/<id>` is refused. All regular files under a removed run
directory are counted, including files beyond the server's 200-file/8-MiB/64-MiB
inventory caps, which bound the recorded inventory, not disk ownership.
The database is then compacted with one `VACUUM` and one
`PRAGMA wal_checkpoint(TRUNCATE)`, retried at most three short times on
`SQLITE_BUSY`. `database_bytes_reclaimed` and `space_reclaimed_bytes` are
measured, not estimated. If `VACUUM` fails the command returns an error, sets
`vacuum_error` and claims no reclaimed database pages; a later `apply` with
nothing left to remove can retry the `VACUUM`. If staging cleanup fails the
report names the staging path and remaining bytes/files and does not count them
as reclaimed.
### Measured example (this machine)
Reported before, applied, reported after, against the live database while its
server kept running (server PID unchanged):
```text
report --keep 200: total 279 · would remove 79 · protected 0 · retained 200
artifacts 537 files · 499046205 bytes (476 MiB)
database 41426944 bytes · free 4096 · reclaimable 499050301
apply --keep 200: removed 79 runs · 537 files · 499046205 bytes
vacuumed · database 41426944 -> 26898432 bytes
database_bytes_reclaimed 14528512 · space_reclaimed_bytes 513574717
report --keep 200: total 200 · would remove 0 · retained 200
pipelines 18 · pipeline_versions 39 (unchanged)
```
Independent observation: `du -sk` of the artifacts root fell from 3213324 to
2724872 KiB, the database file from 41426944 to 26898432 bytes, and the logs
stayed at 18404 KiB. A retained run's artifact still served (HTTP 200) while a
removed run's URL returned 404. The measured numbers above come from that
before/apply/after output, not an estimate.
## Verification scope
CLI integration tests exercise draft/validate/submit/history/conflict handling and