diff --git a/docs/notes/session-export-handoff-gate.md b/docs/notes/session-export-handoff-gate.md index 6f09e72c..086d550a 100644 --- a/docs/notes/session-export-handoff-gate.md +++ b/docs/notes/session-export-handoff-gate.md @@ -73,7 +73,7 @@ retry and refresh of an already-open Session tab. Large (>256 KiB) API fixtures, wrong identity, partial/empty exports, restart, cancellation, workspace reservation, and exactly-once successor admission have focused regressions. A full ordinary artifact inventory no longer blocks versioned audit repair. -Repairs have a separate bounded allowance (twenty attempts, each at most 2 MiB), +Repairs have a separate bounded allowance (twenty attempts, each at most 8 MiB), so neither the ordinary inventory's file count nor its byte budget consumes the mandatory audit-recovery capacity. The ordinary scan limits remain unchanged. diff --git a/docs/notes/session-export-render-cap.md b/docs/notes/session-export-render-cap.md new file mode 100644 index 00000000..3ffd0b34 --- /dev/null +++ b/docs/notes/session-export-render-cap.md @@ -0,0 +1,44 @@ +# Session export: rendered-file cap aligned with the artifact bound + +## Problem + +Six computed Flutter runs (34, 37, 39, 42, 49, 50) reached Code visit 1 with a +successful, quiescent agent session but stopped at the export handoff gate. Each +recorded `session_export.status: partial`, `html_bytes: 2097152`, +`truncated: true`; the manual versioned repair of run 34 hit the same exact size. +The sessions had completed their computation, so a stage rerun was not the right +recovery. + +A read-only inspection of the recorded session rows showed the original native +data for those six sessions was between 1,083,391 and 2,111,470 bytes (the +complete HTTP export envelope for one of them was 2,126,449 bytes). All were +well inside the existing 8 MiB source read cap. The binding limit was the 2 MiB +rendered-file cap: html escaping and the readable projection keep native text +more than once, so an ordinary session can render larger than its source. +Re-exporting the six exact sessions read-only into scratch directories with the +new cap produced complete, untruncated files of 2,252,874 to 4,276,358 bytes. + +## Change + +`internal/stage/session_export.go` raises `SessionExportFileLimit` from 2 MiB to +8 MiB. That value is the same as the server's ordinary per-artifact file bound +(`maxArtifactFileBytes` in `internal/server/artifacts.go`), so a complete export +remains listed and served instead of being cut below the bound that the artifact +inventory already enforces. + +The source read remains capped by `SessionExportSourceLimit` (the single +`opencode.MaxSessionExport` value, 8 MiB). Normal export verification, versioned +export repair and the handoff gate all use `SessionExportFileLimit`, so they now +share one larger bound. The export still refuses to overwrite an existing file, +still marks any cut with a visible leading and trailing banner, and still reports +`truncated: true` and `status: partial`; nothing treats a partial export as +complete. + +## Checks + +Focused stage and server regressions cover a complete export above the old 2 MiB +cap but inside the new cap, a file above the new cap rejected as partial with the +verification diagnostic, repair accepting the larger complete file while the +original evidence stays unchanged, and the constant staying within the ordinary +artifact file bound. Focused stage/server suites and `go build ./...` plus +`go vet ./...` are run before landing; see the delivery report for exact results. diff --git a/docs/stage-artifacts.md b/docs/stage-artifacts.md index b29260d0..b2da6462 100644 --- a/docs/stage-artifacts.md +++ b/docs/stage-artifacts.md @@ -97,10 +97,13 @@ no special casing. `opencode session export `. Both reuse the same invocation helper and session-ID/messages validation as the transcript readback. The file is the harness's real session, never a reconstruction. -- Bounds: the source read is capped at 8 MiB and the rendered file at 2 MiB. When - either is cut, the file carries a visible truncation banner near the start and - the end and the step records `truncated: true`; a truncated export is never - described as whole. +- Bounds: the source read is capped at 8 MiB and the rendered file at 8 MiB, + matching the ordinary per-artifact file bound. The rendered file can exceed + its source because html escaping and the readable projection keep native text + more than once, so the previous 2 MiB render cap cut complete sessions that + were well inside the source bound. When either bound is exceeded, the file + carries a visible truncation banner near the start and the end and the step + records `truncated: true`; a truncated export is never described as whole. - Evidence: the step records `session_export` (`session_id`, `backend`, `source`, `html`, `html_bytes`, `messages`, `truncated`, `diagnostic`), so `pipelines inspect RUN_ID --json` shows what was written without reading the diff --git a/internal/author/skill/references/operations.md b/internal/author/skill/references/operations.md index 06a6013a..c67de23b 100644 --- a/internal/author/skill/references/operations.md +++ b/internal/author/skill/references/operations.md @@ -249,9 +249,10 @@ landed. presented. A run recorded before the HTML-only change still lists `session.md` among its stored artifacts (dated evidence kept on disk, not deleted); every listing — app, TUI, web spike — hides it rather than showing it beside the html. -- The source read is capped at 8 MiB and the rendered file at 2 MiB; a cut source or - a cut rendered file carries a visible truncation banner at both ends and - `session_export.truncated: true`, and a truncated export is never called whole. +- The source read is capped at 8 MiB and the rendered file at 8 MiB, matching the + ordinary per-artifact file bound; a cut source or a cut rendered file carries a + visible truncation banner at both ends and `session_export.truncated: true`, and + a truncated export is never called whole. - The step records `session_export` (`session_id`, `backend`, `source`, the html file name, `html_bytes`, `messages`, `truncated`, `diagnostic`) in `inspect RUN_ID --json`. A record written before the change still carried diff --git a/internal/server/session_export_gate_test.go b/internal/server/session_export_gate_test.go index a32a0943..3c9519c4 100644 --- a/internal/server/session_export_gate_test.go +++ b/internal/server/session_export_gate_test.go @@ -136,6 +136,56 @@ func TestSessionExportGateRetryOnlyAndExactlyOnce(t *testing.T) { } } +// TestSessionExportGateAcceptsRepairAboveOldRenderCap keeps the handoff gate's +// versioned-repair verification on the same larger bound as normal export. +func TestSessionExportGateAcceptsRepairAboveOldRenderCap(t *testing.T) { + h, s, id, first, second := gateRunner(t, func(context.Context, stage.SessionExportRequest) (stage.SessionExportResult, error) { + return stage.SessionExportResult{}, errors.New("export offline") + }, false) + h.runner.wg.Wait() + ctx := context.Background() + if err := s.ConfirmRecoveredSession(ctx, pipeline.SessionOwner{RunID: id, ID: "ses_exact"}); err != nil { + t.Fatal(err) + } + size := int64(2<<20) + 4096 + h.runner.exportSession = func(_ context.Context, req stage.SessionExportRequest) (stage.SessionExportResult, error) { + if err := os.MkdirAll(req.ArtifactsDir, 0700); err != nil { + return stage.SessionExportResult{}, err + } + f, err := os.Create(filepath.Join(req.ArtifactsDir, req.FileName)) + if err != nil { + return stage.SessionExportResult{}, err + } + if err := f.Truncate(size); err != nil { + f.Close() + return stage.SessionExportResult{}, err + } + if err := f.Close(); err != nil { + return stage.SessionExportResult{}, err + } + return stage.SessionExportResult{Source: "native_export", HTML: req.FileName, HTMLBytes: size, Messages: 2}, nil + } + if _, err := h.runner.repairSessionExport(ctx, id, 1); err != nil { + t.Fatal(err) + } + // The worker promotes a persisted complete repair before claiming a lease. + job, err := s.ClaimSessionExportJob(ctx, time.Now().Add(time.Second), time.Minute) + if err != nil { + t.Fatal(err) + } + if job != nil { + t.Fatalf("complete repair still claimed as a recovery job: %+v", job) + } + if err := h.runner.resumeExportGates(ctx); err != nil { + t.Fatal(err) + } + h.runner.wg.Wait() + d, _ := s.GetRun(ctx, id) + if first.Load() != 1 || second.Load() != 1 || d.Status != "succeeded" || len(d.Steps) != 2 { + t.Fatalf("large repair did not advance the gate: %+v (%d/%d)", d, first.Load(), second.Load()) + } +} + func TestSessionExportGateCleanupUnconfirmedDoesNotExportOrAdvance(t *testing.T) { var exports atomic.Int32 h, s, id, _, second := gateRunner(t, func(ctx context.Context, req stage.SessionExportRequest) (stage.SessionExportResult, error) { diff --git a/internal/server/session_export_repair_test.go b/internal/server/session_export_repair_test.go index 4c0d8252..c2aabf3a 100644 --- a/internal/server/session_export_repair_test.go +++ b/internal/server/session_export_repair_test.go @@ -111,6 +111,57 @@ func TestRuntimeSessionExportRepairPreservesOriginalAndUsesRecordedOwner(t *test } } +// TestRuntimeSessionExportRepairAcceptsAboveTheOldRenderCap proves a versioned +// repair uses the same larger bound as a normal export while leaving the +// original recorded evidence untouched. +func TestRuntimeSessionExportRepairAcceptsAboveTheOldRenderCap(t *testing.T) { + h, s, id := sessionExportRunner(t, "v2", func(stage.SessionExportRequest) (stage.SessionExportResult, error) { + return stage.SessionExportResult{}, errors.New("initial export unavailable") + }) + if err := s.ConfirmRecoveredSession(context.Background(), pipeline.SessionOwner{RunID: id, ID: "ses_run"}); err != nil { + t.Fatal(err) + } + original, err := s.GetRun(context.Background(), id) + if err != nil { + t.Fatal(err) + } + before, _ := json.Marshal(original.Steps[0].SessionExport) + size := int64(2<<20) + 4096 + h.runner.exportSession = func(_ context.Context, req stage.SessionExportRequest) (stage.SessionExportResult, error) { + if err := os.MkdirAll(req.ArtifactsDir, 0700); err != nil { + return stage.SessionExportResult{}, err + } + f, err := os.Create(filepath.Join(req.ArtifactsDir, req.FileName)) + if err != nil { + return stage.SessionExportResult{}, err + } + if err := f.Truncate(size); err != nil { + f.Close() + return stage.SessionExportResult{}, err + } + if err := f.Close(); err != nil { + return stage.SessionExportResult{}, err + } + return stage.SessionExportResult{Source: "native_export", HTML: req.FileName, HTMLBytes: size, Messages: 2}, nil + } + repaired, err := h.runner.repairSessionExport(context.Background(), id, 1) + if err != nil { + t.Fatal(err) + } + latest := repaired.Steps[0].SessionExportRepairs[0] + if latest.Status != "complete" || latest.HTMLBytes != size || latest.HTML == "" { + t.Fatalf("repair above the old cap was not complete: %+v", latest) + } + after, _ := json.Marshal(repaired.Steps[0].SessionExport) + if string(before) != string(after) { + t.Fatal("repair above the old cap overwrote original evidence") + } + info, err := os.Stat(filepath.Join(h.runner.artifactsRoot, "1", "stages", "01-"+repaired.Steps[0].Stage, latest.HTML)) + if err != nil || info.Size() != size { + t.Fatalf("repair file missing or resized: %v %+v", err, info) + } +} + func TestRuntimeRepairRejectsUnrecordedAndActiveVisits(t *testing.T) { h, _, id := sessionExportRunner(t, "v2", func(stage.SessionExportRequest) (stage.SessionExportResult, error) { return stage.SessionExportResult{}, errors.New("unavailable") diff --git a/internal/server/session_export_status_test.go b/internal/server/session_export_status_test.go index a1bae02a..10daae33 100644 --- a/internal/server/session_export_status_test.go +++ b/internal/server/session_export_status_test.go @@ -1,10 +1,13 @@ package server import ( - "git.a-wan.net/alan/pipelines/internal/protocol" "os" "path/filepath" + "strings" "testing" + + "git.a-wan.net/alan/pipelines/internal/protocol" + "git.a-wan.net/alan/pipelines/internal/stage" ) func TestSessionExportCompleteness(t *testing.T) { @@ -56,3 +59,61 @@ func TestRuntimeSessionExportFileVerificationDoesNotClaimMissingEvidence(t *test }) } } + +func sizedSessionFile(t *testing.T, size int64) string { + t.Helper() + dir := t.TempDir() + path := filepath.Join(dir, "session.html") + f, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + if err := f.Truncate(size); err != nil { + f.Close() + t.Fatal(err) + } + if err := f.Close(); err != nil { + t.Fatal(err) + } + return dir +} + +// TestRuntimeSessionExportVerificationAcceptsAboveTheOldRenderCap pins the +// ordinary-session case the previous 2 MiB rendered bound cut. The file is +// larger than that old cap and inside the artifact-aligned bound, so normal +// export verification must keep it and classify the evidence complete. +func TestRuntimeSessionExportVerificationAcceptsAboveTheOldRenderCap(t *testing.T) { + size := int64(2<<20) + 4096 + dir := sizedSessionFile(t, size) + evidence := &protocol.SessionExport{Source: "native_export", Messages: 2, HTML: "session.html", HTMLBytes: size} + verifySessionExportFile(evidence, dir) + classifySessionExport(evidence) + if evidence.Status != "complete" || evidence.HTML != "session.html" || evidence.HTMLBytes != size || evidence.Diagnostic != "" { + t.Fatalf("complete export above the old cap was rejected: %+v", evidence) + } +} + +// TestRuntimeSessionExportVerificationRejectsAboveTheNewCap confirms the larger +// bound is still a hard bound: an oversized file is not presented as evidence. +func TestRuntimeSessionExportVerificationRejectsAboveTheNewCap(t *testing.T) { + size := int64(stage.SessionExportFileLimit) + 1 + dir := sizedSessionFile(t, size) + evidence := &protocol.SessionExport{Source: "native_export", Messages: 2, HTML: "session.html", HTMLBytes: size} + verifySessionExportFile(evidence, dir) + classifySessionExport(evidence) + if evidence.Status != "partial" || evidence.HTML != "" || evidence.HTMLBytes != 0 || !strings.Contains(evidence.Diagnostic, "bounded verification") { + t.Fatalf("oversized export was not rejected as partial: %+v", evidence) + } +} + +// TestSessionExportFileLimitStaysInsideArtifactBounds keeps the render cap +// principled: larger than the cap that cut ordinary sessions, and no larger +// than the ordinary per-artifact file bound that serves the file. +func TestSessionExportFileLimitStaysInsideArtifactBounds(t *testing.T) { + if stage.SessionExportFileLimit <= 2<<20 { + t.Fatalf("render cap %d did not grow beyond the old 2 MiB cutoff", stage.SessionExportFileLimit) + } + if stage.SessionExportFileLimit > maxArtifactFileBytes { + t.Fatalf("render cap %d exceeds the artifact file bound %d", stage.SessionExportFileLimit, maxArtifactFileBytes) + } +} diff --git a/internal/stage/session_export.go b/internal/stage/session_export.go index 583602dd..2c752c5c 100644 --- a/internal/stage/session_export.go +++ b/internal/stage/session_export.go @@ -20,7 +20,7 @@ import ( // artifacts directory as one html file so the existing artifact listing and // serving paths find it with no special casing. The source read is capped at // SessionExportSourceLimit and the rendered file at SessionExportFileLimit, so -// the export adds at most ~2 MiB per step, inside the artifact scan limits. +// the export stays inside the server's per-artifact file bound. const ( // SessionHTMLFile is the one session export a stage writes. SessionHTMLFile = "session.html" @@ -32,8 +32,15 @@ const ( // SessionExportSourceLimit is the native session read cap; it is the one // opencode.MaxSessionExport value, not a second copy. SessionExportSourceLimit = opencode.MaxSessionExport - SessionExportFileLimit = 2 << 20 - SessionExportTimeout = 10 * time.Second + // SessionExportFileLimit is the rendered file cap. Ordinary native sessions + // commonly render a little larger than their raw source because html escaping + // and the readable projection duplicate native text; the previous 2 MiB cap + // cut complete sessions that were well inside the source bound. This cap + // matches the server's per-artifact file bound (maxArtifactFileBytes in + // internal/server/artifacts.go), so a complete export remains listed and + // served while a genuinely oversized one is still visibly truncated. + SessionExportFileLimit = 8 << 20 + SessionExportTimeout = 10 * time.Second ) // SessionExportRequest names the stage session to export and the evidence the diff --git a/internal/stage/session_export_test.go b/internal/stage/session_export_test.go index 5db417fd..8cf2efe2 100644 --- a/internal/stage/session_export_test.go +++ b/internal/stage/session_export_test.go @@ -222,6 +222,38 @@ func TestExportSessionLargeHTTPExportWritesBoundedRealHTML(t *testing.T) { } } +// TestExportSessionCompletesAboveTheOldRenderedCap covers the ordinary case the +// previous 2 MiB render cap cut: a complete native session inside the 8 MiB +// source bound whose escaped html is larger than 2 MiB. +func TestExportSessionCompletesAboveTheOldRenderedCap(t *testing.T) { + const oldRenderCap = 2 << 20 + dir := t.TempDir() + line := "complete ordinary session line " + text := strings.Repeat(line, oldRenderCap/len(line)+1000) + body := `{"info":{"id":"ses_ok"},"messages":[{"info":{"id":"m1","role":"user"},"text":"` + text + `"}]}` + sessionCLI(t, "ses_ok", body) + result, err := ExportSession(context.Background(), sessionRequest(dir)) + if err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(filepath.Join(dir, result.HTML)) + if err != nil { + t.Fatal(err) + } + if result.Truncated || result.Messages != 1 || result.HTMLBytes != int64(len(data)) { + t.Fatalf("complete export above the old cap was not recorded: %+v", result) + } + if len(data) <= oldRenderCap || len(data) > SessionExportFileLimit { + t.Fatalf("rendered size %d is not above the old cap and inside the new one", len(data)) + } + if strings.Contains(string(data), "TRUNCATED") { + t.Fatal("a complete export carried a truncation banner") + } + if !strings.Contains(string(data), "complete ordinary session line") { + t.Fatal("complete export lost its session text") + } +} + func TestExportSessionSourceTruncationIsLabelledNotWhole(t *testing.T) { session := opencode.ExportedSession{ ID: "ses_ok",