Recover incomplete session exports through durable runtime jobs

This commit is contained in:
2026-10-20 16:28:14 +01:00
parent 6841cdd2a4
commit a049b552f6
14 changed files with 960 additions and 20 deletions
+36
View File
@@ -43,6 +43,42 @@ There are at most 20 manual repair attempts per visit. Repairs cannot guarantee
recovery if the upstream session or original directory is gone. The web pane
explains original failure and lists repair attempts/links separately.
## Durable automatic recovery
Incomplete native exports now enqueue a `session_export_jobs` row in the same
transaction as `FinishStep`. The snapshot records the exact original session ID,
backend, directory and harness reference. Runtime startup also reconciles missing
jobs from persisted ownership/steps, including interrupted initial capture; it
never guesses missing ownership or edits original evidence. Reconciliation is
bounded to 256 newly discovered jobs per sweep (startup, then every 30 seconds).
A runtime-owned worker claims jobs only for settled runs/steps, with no active
agent in that run or using that session, and with `cleanup_unconfirmed=0` on
current main. The runner's in-memory cleanup gate also defers execution without
spending a retry. Each automatic attempt uses the existing versioned, read-only
repair path and verifies ownership against the durable snapshot before readback.
No stage prompt, pipeline definition, stage execution or run outcome changes.
The first attempt is due after 5 seconds; further attempts wait 20 and 60 seconds.
There are at most 3 automatic claims, each with the existing 10-second readback
budget. A 45-second persisted lease recovers interrupted workers after restart;
attempt counts/backoff are not reset, including process kills. Saved complete
repair evidence can acknowledge a job after a crash without exporting again.
Expired final-attempt leases become exhausted, not an infinite retry loop.
Wrong-session readbacks never qualify as complete evidence. Failed/empty/partial
repairs and exhausted-job diagnostics remain visible in run inspection and the
web visit pane (`session_export_recovery`). Originals remain unchanged.
Worker shutdown follows runner context cancellation and current main's
`Store.Done()` (including cancellation of in-flight readback). The small startup
helper detects that lifecycle interface without modifying this branch's older
store/server-lock implementation. Its compatibility fallback exits on a closed
database; current main's lifecycle/cleanup implementations must be retained when
integrating. Upstream loss, missing original ownership/directory, inventory
limits or sessions exceeding export caps can still leave exhausted evidence;
manual versioned repair remains available. Browser captures for recovery status
are not captured under the no-live-runs audit constraint.
No production service, live run or historical data was modified for this audit.
Unit/fixture tests cover reports, failure gates, retry inheritance, idempotency,
migration, transient/permanent export readback, empty-export diagnostics and