Recover incomplete session exports through durable runtime jobs
This commit is contained in:
@@ -43,6 +43,42 @@ There are at most 20 manual repair attempts per visit. Repairs cannot guarantee
|
||||
recovery if the upstream session or original directory is gone. The web pane
|
||||
explains original failure and lists repair attempts/links separately.
|
||||
|
||||
## Durable automatic recovery
|
||||
|
||||
Incomplete native exports now enqueue a `session_export_jobs` row in the same
|
||||
transaction as `FinishStep`. The snapshot records the exact original session ID,
|
||||
backend, directory and harness reference. Runtime startup also reconciles missing
|
||||
jobs from persisted ownership/steps, including interrupted initial capture; it
|
||||
never guesses missing ownership or edits original evidence. Reconciliation is
|
||||
bounded to 256 newly discovered jobs per sweep (startup, then every 30 seconds).
|
||||
|
||||
A runtime-owned worker claims jobs only for settled runs/steps, with no active
|
||||
agent in that run or using that session, and with `cleanup_unconfirmed=0` on
|
||||
current main. The runner's in-memory cleanup gate also defers execution without
|
||||
spending a retry. Each automatic attempt uses the existing versioned, read-only
|
||||
repair path and verifies ownership against the durable snapshot before readback.
|
||||
No stage prompt, pipeline definition, stage execution or run outcome changes.
|
||||
|
||||
The first attempt is due after 5 seconds; further attempts wait 20 and 60 seconds.
|
||||
There are at most 3 automatic claims, each with the existing 10-second readback
|
||||
budget. A 45-second persisted lease recovers interrupted workers after restart;
|
||||
attempt counts/backoff are not reset, including process kills. Saved complete
|
||||
repair evidence can acknowledge a job after a crash without exporting again.
|
||||
Expired final-attempt leases become exhausted, not an infinite retry loop.
|
||||
Wrong-session readbacks never qualify as complete evidence. Failed/empty/partial
|
||||
repairs and exhausted-job diagnostics remain visible in run inspection and the
|
||||
web visit pane (`session_export_recovery`). Originals remain unchanged.
|
||||
|
||||
Worker shutdown follows runner context cancellation and current main's
|
||||
`Store.Done()` (including cancellation of in-flight readback). The small startup
|
||||
helper detects that lifecycle interface without modifying this branch's older
|
||||
store/server-lock implementation. Its compatibility fallback exits on a closed
|
||||
database; current main's lifecycle/cleanup implementations must be retained when
|
||||
integrating. Upstream loss, missing original ownership/directory, inventory
|
||||
limits or sessions exceeding export caps can still leave exhausted evidence;
|
||||
manual versioned repair remains available. Browser captures for recovery status
|
||||
are not captured under the no-live-runs audit constraint.
|
||||
|
||||
No production service, live run or historical data was modified for this audit.
|
||||
Unit/fixture tests cover reports, failure gates, retry inheritance, idempotency,
|
||||
migration, transient/permanent export readback, empty-export diagnostics and
|
||||
|
||||
Reference in New Issue
Block a user