diff --git a/docs/cli.md b/docs/cli.md index 5dae6a7e..a8521693 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -396,9 +396,12 @@ Usage is polled during execution and read again afterward; an in-flight request may exceed a limit before usage arrives. These are not provider-side hard caps. Automatic retries cover retryable transport failures and per-attempt timeouts. -They do not replay admitted tool work, invalid results, or budget failures. -Cleanup must be confirmed before another attempt. Existing explicit `retries=0` -settings remain unchanged. The old 30-minute run deadline no longer truncates +An explicit read-only stage (a tools block listing neither `edit` nor `shell`) +may also retry a post-admission deadline or transport failure and re-emit a +malformed final result up to `retries`; a stage whose tools include `edit`/`shell` +or that has no tools block never replays admitted tool work or invalid results. +Budget failures always stop. Cleanup must be confirmed before another attempt. +Existing explicit `retries=0` settings remain unchanged. The old 30-minute run deadline no longer truncates stage timeouts; the 30-step and five-visits-per-stage limits still apply. Native ntfy stages also accept `timeout` (default `1h`), but never automatically retry potentially delivered notifications and do not accept model budgets. diff --git a/docs/model-results.md b/docs/model-results.md index 78424aee..8d122a2e 100644 --- a/docs/model-results.md +++ b/docs/model-results.md @@ -47,7 +47,10 @@ Format recovery has a separate bounded budget, even with `retries=0`: If recovery fails, report the original precise error and repair failure; never rerun project edits, commands, or the entire stage to repair final formatting. Configured execution retries remain for retryable text-generation transport -errors; audited tool-enabled execution is never automatically replayed. +errors; an explicit read-only stage may additionally re-emit a malformed final +result and retry a post-admission deadline or transport failure up to `retries`. +A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy +read/glob/grep/edit/shell bundle), never has admitted tool work replayed. Repair records include the original/candidate response, parsing diagnostic, method and acceptance flag in the bounded transcript preview. `pipelines inspect --json` and the TUI evidence section expose these records. @@ -86,7 +89,9 @@ failure, and vice versa. Neither label includes a provider response body, prompt credential or arbitrary model content. Duplicate nested fields are rejected by strict parsing rather than merged or last-wins, and only a lossless insertion of missing commas/colons or trailing closers is repaired, so an unknown evidence or -verdict value can never become `success` and admitted tool work is never replayed. +verdict value can never become `success`, and admitted tool work on a mutating or +legacy-bundle stage is never replayed (only an explicit read-only stage may +re-emit a malformed result within `retries`). Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned exactly 262144 bytes of incomplete JSON (`invalid agent messages`); a reviewer diff --git a/internal/author/skill/references/format.md b/internal/author/skill/references/format.md index 40e9c2a0..fa1ccb08 100644 --- a/internal/author/skill/references/format.md +++ b/internal/author/skill/references/format.md @@ -56,9 +56,11 @@ authentication and inference require `pipelines provider-check` on the execution computer/container. Optional `retries` defaults to 2 (three total attempts), and accepts integers 0–100; explicit 0 disables execution retries. Optional `timeout` defaults to `1h` per attempt and accepts positive durations such as -`30m` or `90s`. Automatic retries only cover retryable transport failures and -timeouts; admitted tool work is never replayed automatically. Optional -`budget_usd=1.50` and `budget_tokens=50000` independently cap reported usage. +`30m` or `90s`. Automatic retries cover retryable transport failures and +timeouts; an explicit read-only stage may additionally retry a post-admission +deadline or transport failure, while a stage with `edit`/`shell` (or no tools +block) never replays admitted work. Optional `budget_usd=1.50` and +`budget_tokens=50000` independently cap reported usage. They accumulate across attempts, format repair sessions, and repeated visits to this stage in the same run. Omit either budget to leave that resource unlimited. Both budgets must be positive; tokens include input, output, reasoning and cache @@ -104,7 +106,10 @@ permission policy. Keep ordinary workflows language-neutral and unrestricted within their run directory unless the user asks for a narrower scope. Malformed final JSON has bounded, syntax-only recovery even with `retries=0`; -it never replays stage tools or changes verdict/evidence. Compatible native stable +it never changes verdict/evidence. On an explicit read-only stage a rejected +malformed result may be re-emitted up to `retries` with its `format_repairs` +evidence retained; a stage with `edit`/`shell` or no tools block never replays +stage tools or re-emits the result. Compatible native stable backends request schema output with schema retries disabled; a retained deny-all, no-inference compatibility probe identifies known format incompatibility. Incompatible native backends and V2 require strict local envelope validation. Do not promise diff --git a/internal/author/skill/references/operations.md b/internal/author/skill/references/operations.md index 04cdcf0b..bb716717 100644 --- a/internal/author/skill/references/operations.md +++ b/internal/author/skill/references/operations.md @@ -196,5 +196,10 @@ VALIDATION` names the exact duplicate-field/syntax detail and byte offset for a strictly invalid final result; `HARNESS TRANSPORT` covers request, active-session poll and final-readback failures. A validation failure never re-prompts the admitted tool-using stage: completed edits and tool side effects are retained and -not undone, so a full retry repeats work. Neither category exposes provider/model -bodies or credentials. +not undone, so a full retry repeats work. An explicit read-only stage is the one +exception: a malformed result there mutates nothing, so it may be re-emitted up +to `retries` with its `format_repairs` evidence retained, and a post-admission +deadline or transport failure may be retried. A stage whose tools include +`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell +bundle), is never replayed. Neither category exposes provider/model bodies or +credentials. diff --git a/internal/author/skill/references/verification.md b/internal/author/skill/references/verification.md index c7028970..b0cbd242 100644 --- a/internal/author/skill/references/verification.md +++ b/internal/author/skill/references/verification.md @@ -94,6 +94,20 @@ field is sent. A review-only fresh execution is a future recovery idea, **not implemented**. Onboarding UI visual approval remains unresolved and must not be reported complete. +## Failure-handling source changes: 2026-10-15 + +Source-level, mocked `internal/stage` and `internal/server` regression coverage +(not live-provider evidence): an explicit read-only tools stage now actually +retries a post-admission stage deadline or harness transport failure, creating a +fresh bounded session per attempt, while a stage with `edit`/`shell` or no tools +block still never replays admitted work; a malformed final result is re-emitted +up to `retries` only on such a read-only stage with its `format_repairs` evidence +retained. A timed-out agent stage settles knowable V2 usage and retains one +bounded, redacted latest-assistant readback plus a `retained_session` pointer in +its stored transcript. A declared outcome routed to `#failure` with no step +diagnostic now yields a run diagnostic like `stage returned outcome +"blocked"`; an existing step diagnostic is still used verbatim. + ## Dated source-level coverage: 2026-10-11 Source-level/mocked regression coverage (not live-provider evidence): diff --git a/internal/server/run_diagnostic_test.go b/internal/server/run_diagnostic_test.go new file mode 100644 index 00000000..ec29bfa0 --- /dev/null +++ b/internal/server/run_diagnostic_test.go @@ -0,0 +1,66 @@ +package server + +import ( + "context" + "errors" + "path/filepath" + "strings" + "testing" + + "git.a-wan.net/alan/pipelines/internal/pipeline" + "git.a-wan.net/alan/pipelines/internal/protocol" + "git.a-wan.net/alan/pipelines/internal/stage" +) + +// TestFailureRouteRunDiagnostic proves A9: a declared outcome routed to #failure +// with an empty step diagnostic still yields a meaningful run diagnostic, while +// an existing step diagnostic is preferred verbatim. +func TestFailureRouteRunDiagnostic(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + store, err := pipeline.Open(filepath.Join(t.TempDir(), "runs.db")) + if err != nil { + t.Fatal(err) + } + defer store.Close() + source := strings.Replace(pipeline.Template([]string{"worker"}, true), "success -> #success", "success -> #success\nblocked -> #failure", 1) + if _, err := store.Submit(ctx, "diagnostic", source); err != nil { + t.Fatal(err) + } + items, _ := store.List(ctx) + r, err := newRunner(ctx, store) + if err != nil { + t.Fatal(err) + } + r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) { + return stage.Result{Outcome: "blocked", Output: stage.Values{}}, nil + } + run, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "blocked", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}}) + if err != nil { + t.Fatal(err) + } + r.wg.Wait() + detail, err := store.GetRun(ctx, run.ID) + if err != nil { + t.Fatal(err) + } + if detail.Status != "failed" || detail.Outcome != "error" || !strings.Contains(detail.Diagnostic, `stage worker returned outcome "blocked"`) { + t.Fatalf("empty run diagnostic not replaced: %+v", detail) + } + + r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) { + return stage.Result{}, errors.New("fixture execution failed") + } + run, err = r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "execution", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}}) + if err != nil { + t.Fatal(err) + } + r.wg.Wait() + detail, err = store.GetRun(ctx, run.ID) + if err != nil { + t.Fatal(err) + } + if detail.Status != "failed" || detail.Diagnostic != "fixture execution failed" { + t.Fatalf("existing step diagnostic not preferred: %+v", detail) + } +} diff --git a/internal/server/runner.go b/internal/server/runner.go index 14024761..9977dd36 100644 --- a/internal/server/runner.go +++ b/internal/server/runner.go @@ -473,6 +473,11 @@ func (r *runner) run(ctx context.Context, id int64, d *protocol.PipelineDetail, step.Outcome = "error" step.Output = stage.Values{"error": err.Error(), "stage": key, "input": input} } + // A failed or timed-out step carries the session pointer even when the + // bounded transcript would otherwise be only its stage_limits envelope. + if err != nil && stageSession != "" { + step.Transcript = stage.RetainSession(step.Transcript, stageSession) + } step.Target = def.Edges[step.Outcome] step.Control = &protocol.ControlEvidence{Visit: visits[key], MaxVisits: maxVisits, Memory: reads} if err == nil && selected.Limits != nil && visits[key] >= maxVisits { @@ -526,7 +531,11 @@ func (r *runner) run(ctx context.Context, id int64, d *protocol.PipelineDetail, return } if step.Target == protocol.FailureEnd { - finish("failed", "error", step.Diagnostic) + diagnostic := step.Diagnostic + if diagnostic == "" { + diagnostic = fmt.Sprintf("stage %s returned outcome %q", step.Stage, step.Outcome) + } + finish("failed", "error", diagnostic) return } if step.Target == "" { diff --git a/internal/stage/agent.go b/internal/stage/agent.go index bb05085d..18f35dad 100644 --- a/internal/stage/agent.go +++ b/internal/stage/agent.go @@ -93,6 +93,7 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 15*time.Second) defer cancel() if err != nil { + c.captureTimedOutEvidence(ctx, err, config, watch) if stopErr := c.Interrupt(cleanup, c.SessionID); stopErr != nil { err = fmt.Errorf("agent interrupt failed; retained session %s needs inspection: %w", c.SessionID, errors.Join(err, stopErr, ErrStageCleanup)) return @@ -185,6 +186,27 @@ func (c OpenCodeClient) settleV2Usage(ctx context.Context, budget *Budget, id st _ = budget.readUsage(id, data, false) } +// captureTimedOutEvidence retains one bounded, redacted best-effort readback of +// the latest assistant message after a post-admission deadline or transport +// failure, and settles knowable V2 usage for an unbudgeted stage. A failed +// readback is ignored and never changes the returned error. +func (c *AgentClient) captureTimedOutEvidence(ctx context.Context, err error, config ModelConfig, watch *budgetWatch) { + if !c.WorkStarted || c.SessionID == "" || !(errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport)) { + return + } + readCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second) + data, readErr := c.call(readCtx, "get", "/api/session/"+c.SessionID+finalAssistantReadback, nil) + cancel() + if readErr == nil && len(data) > 0 { + c.Transcript = RedactEvidence(string(data)) + } + // A budgeted stage already settles through watch.finish; only an unbudgeted + // V2 stage needs this bounded aggregate read. + if watch == nil && c.Backend == "v2" { + c.settleV2Usage(ctx, config.Budget, c.SessionID) + } +} + // finalAssistantReadback is the single bounded request used to select a V2 // final result: only the latest assistant message. Fetching the full assistant // history also pulls large prior tool-read contents and can exceed the response diff --git a/internal/stage/diagnostics.go b/internal/stage/diagnostics.go index 1fbfdb79..2d662edc 100644 --- a/internal/stage/diagnostics.go +++ b/internal/stage/diagnostics.go @@ -234,6 +234,30 @@ func BoundTranscript(transcript, sessionID string) string { return string(encoded) } +// RetainSession adds a retained_session pointer to an already-JSON-object +// transcript so a failed or timed-out step carries the session ID even below the +// 32 KiB BoundTranscript threshold. Non-object and empty transcripts are +// returned unchanged and the value is JSON-encoded, so the result stays valid. +func RetainSession(transcript, sessionID string) string { + if sessionID == "" { + return transcript + } + var data map[string]json.RawMessage + if json.Unmarshal([]byte(transcript), &data) != nil || data == nil { + return transcript + } + encoded, err := json.Marshal(sessionID) + if err != nil { + return transcript + } + data["retained_session"] = encoded + merged, err := json.Marshal(data) + if err != nil { + return transcript + } + return string(merged) +} + func (l *OperationLog) record(e OperationEvidence, started time.Time) { if l == nil { return diff --git a/internal/stage/model.go b/internal/stage/model.go index 525d125c..dc52a161 100644 --- a/internal/stage/model.go +++ b/internal/stage/model.go @@ -59,6 +59,26 @@ func wrapModelStageFailure(key string, err error) error { } } +// replayablePostAdmission reports whether a post-admission error can be retried +// from the start without risking a repeated tool mutation: a stage context +// deadline or a harness transport failure. A structured-output validation +// failure is classified separately as ErrInvalidModelResult. +func replayablePostAdmission(err error) bool { + return errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport) +} + +// toleratedModelResultRetry reports whether a malformed final result may be +// re-emitted. It requires an explicit read-only stage, remaining attempts, +// live work, available budget and a plain validation error; fail-closed +// accounting and cleanup errors still stop immediately. +func toleratedModelResultRetry(ctx context.Context, def Definition, config ModelConfig, attempt int, resultErr error) bool { + return errors.Is(resultErr, ErrInvalidModelResult) && + !errors.Is(resultErr, ErrBudgetExceeded) && + !errors.Is(resultErr, ErrBudgetAccounting) && + !errors.Is(resultErr, ErrStageCleanup) && + safeToReplay(def) && attempt < config.Retries && ctx.Err() == nil && config.Budget != nil && config.Budget.Check() == nil +} + var providerID = regexp.MustCompile(`^[A-Za-z0-9_.-]+$`) var modelID = regexp.MustCompile(`^[A-Za-z0-9_./:-]+$`) @@ -208,8 +228,9 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu return Result{}, fmt.Errorf("%w: generator cannot enforce usage limits", ErrBudgetAccounting) } } - // Replaying admitted tool work can repeat mutations. Only pre-admission - // transport errors may automatically retry an audited agent. + // Replaying admitted tool work can repeat mutations. Only a stage whose + // explicit tools cannot mutate the run directory (safeToReplay) may + // automatically retry work that started after admission. if agent, ok := client.(*AgentClient); ok { if tools, explicit, _ := ToolNames(def); explicit { agent.Tools = tools @@ -232,6 +253,7 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu if generateErr == nil && attemptCtx.Err() != nil { generateErr = attemptCtx.Err() } + replayPostAdmission := false if generateErr == nil { result, resultErr := decodeModelResult(stageResultText(text), def.Edges) if resultErr != nil { @@ -241,27 +263,41 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu resultErr = errors.Join(resultErr, attemptCtx.Err()) } cancel() - return result, wrapModelStageFailure(def.Key, resultErr) - } - err = generateErr - if attemptCtx.Err() != nil && !errors.Is(err, ErrBudgetExceeded) && !errors.Is(err, ErrBudgetAccounting) { - err = errors.Join(err, attemptCtx.Err()) - } - cancel() - if ctx.Err() != nil { - return Result{}, ctx.Err() - } - if errors.Is(err, ErrBudgetExceeded) || errors.Is(err, ErrBudgetAccounting) || errors.Is(err, ErrStageCleanup) { - break - } - if agent, ok := client.(*AgentClient); ok && agent.WorkStarted { - break - } - if pi, ok := client.(*PiClient); ok && pi.WorkStarted { - break - } - if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded)) { - break + if resultErr == nil { + return result, nil + } + // A malformed final result mutates nothing on an explicit read-only + // stage, so it may be re-emitted within the retry budget. A mutating + // or legacy-bundle stage retains its admitted work and returns now. + if toleratedModelResultRetry(ctx, def, spec.Config, attempt, resultErr) { + err = resultErr + } else { + return result, wrapModelStageFailure(def.Key, resultErr) + } + } else { + err = generateErr + if attemptCtx.Err() != nil && !errors.Is(err, ErrBudgetExceeded) && !errors.Is(err, ErrBudgetAccounting) { + err = errors.Join(err, attemptCtx.Err()) + } + cancel() + if ctx.Err() != nil { + return Result{}, ctx.Err() + } + if errors.Is(err, ErrBudgetExceeded) || errors.Is(err, ErrBudgetAccounting) || errors.Is(err, ErrStageCleanup) { + break + } + replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err) + // Never blindly replay admitted tool work: only a read-only stage may + // retry a post-admission deadline or transport failure. + if agent, ok := client.(*AgentClient); ok && agent.WorkStarted && !replayPostAdmission { + break + } + if pi, ok := client.(*PiClient); ok && pi.WorkStarted && !replayPostAdmission { + break + } + if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission) { + break + } } delay := time.Duration(1< RetainSession -> +// BoundTranscript) yields a valid, bounded JSON object that keeps both the +// session pointer and the stage limits. +func TestTimedOutTranscriptEnvelopeRetainsSessionAndLimits(t *testing.T) { + evidence := `{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"partial\":true}"}]}]}` + composed := BoundTranscript(RetainSession(BudgetTranscript(evidence, NewBudget(0, 0), time.Hour, 2), "ses_timeout"), "ses_timeout") + var data map[string]json.RawMessage + if err := json.Unmarshal([]byte(composed), &data); err != nil { + t.Fatal(err) + } + for _, key := range []string{"retained_session", "stage_limits"} { + if _, ok := data[key]; !ok { + t.Fatalf("composed envelope missing %s: %s", key, composed) + } + } + if len(composed) > transcriptBound { + t.Fatalf("composed envelope exceeds bound: %d", len(composed)) + } + if strings.Contains(composed, "Bearer") { + t.Fatal("composed envelope leaked a credential") + } +} + +func TestRetainSessionAddsPointerToJSONObjectOnly(t *testing.T) { + if got := RetainSession("", "ses_1"); got != "" { + t.Fatalf("empty transcript was wrapped: %q", got) + } + if got := RetainSession("plain text", "ses_1"); got != "plain text" { + t.Fatalf("non-object transcript was wrapped: %q", got) + } + if got := RetainSession(`{"a":1}`, ""); got != `{"a":1}` { + t.Fatalf("empty session id changed the transcript: %q", got) + } + got := RetainSession(`{"stage_limits":{"attempts":1}}`, "ses_1") + var data map[string]any + if err := json.Unmarshal([]byte(got), &data); err != nil { + t.Fatal(err) + } + if data["retained_session"] != "ses_1" || data["stage_limits"] == nil { + t.Fatalf("retained session/limits lost: %v", data) + } +} diff --git a/internal/stage/tools.go b/internal/stage/tools.go index 2d0f2294..b1a63f36 100644 --- a/internal/stage/tools.go +++ b/internal/stage/tools.go @@ -35,6 +35,24 @@ func ToolNames(def Definition) ([]string, bool, error) { return tools, true, nil } +// safeToReplay is the single definition of which post-admission work may be +// automatically retried. Only an explicit tools block that selects neither +// "edit" nor "shell" is side-effect free. An absent block is the legacy +// read/glob/grep/edit/shell bundle and is treated as mutating, matching the +// runner's useAgent/permissionRules semantics. +func safeToReplay(def Definition) bool { + tools, explicit, err := ToolNames(def) + if err != nil || !explicit { + return false + } + for _, tool := range tools { + if tool == "edit" || tool == "shell" { + return false + } + } + return true +} + func (c *AgentClient) permissionRules() []map[string]string { rules := []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}} tools := c.Tools