diff --git a/docs/model-results.md b/docs/model-results.md index 576d6b65..f04a3366 100644 --- a/docs/model-results.md +++ b/docs/model-results.md @@ -82,10 +82,11 @@ Format recovery has a separate bounded budget, even with `retries=0`: 2. If the local pass cannot repair it, make at most one fresh, deny-all, retained text-generation request with only the malformed response and precise parsing error. The original implementing/reviewing agent is never re-prompted. The - formatter's deadline is derived from the payload size and the attempt's own - remaining budget rather than a fixed short constant, so a large result is not - guaranteed to time out; it stays within the stage deadline, is floored for - ordinary payloads, and keeps the 32 KiB input/output bound. + formatter's deadline is derived from the payload size, the stage timeout and + the attempt's own remaining budget rather than a fixed short constant, so a + large or slower result is not guaranteed to time out; it stays within the + stage deadline (clamped to `remaining` minus a margin), is floored for ordinary + payloads, and keeps the 32 KiB input/output bound. 3. Accept only insertion of missing commas/colons or trailing closing delimiters, with at most 16 insertions. Every existing string, number, literal and structural delimiter must remain verbatim and in order. Validate the full @@ -106,8 +107,14 @@ rerun project edits, commands, or the entire stage to repair final formatting. Configured execution retries remain for retryable text-generation transport errors; an explicit read-only stage may additionally re-emit a malformed final result and retry a post-admission deadline or transport failure up to `retries`. -A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy -read/glob/grep/edit/shell bundle), never has admitted tool work replayed. The +A failure to reach the harness before any prompt or tool call is admitted +(connecting, or creating the session) is a distinct reach failure: nothing was +admitted, so it is retried within `retries` for any stage, mutating or not, with +a fresh session and no replayed prompt or tool call. A post-admission readback, +active-session poll or agent-execution failure is retried only on an explicit +read-only stage. A stage whose tools include `edit`/`shell`, or that has no tools +block (the legacy read/glob/grep/edit/shell bundle), never has admitted tool work +replayed. The deterministic local pass and the fresh deny-all, tool-free formatter session never re-prompt or replay the admitted stage session, so a shell-enabled review stage gains malformed-result recovery without any replay of admitted work. @@ -121,11 +128,18 @@ and a recognized safe error category (such as session not found or service unavailable). Arbitrary stderr, provider messages and response bodies are not copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`) is not evidence that the agent prompt failed to start, and does not trigger replay -of stage work. +of stage work. When the context expires while a request is still in flight, the +error keeps both the sanitized transport detail (operation/status/exit) and +`errors.Is(err, context.DeadlineExceeded)`, so a deadline that fired during a hung +transport call is not replaced by a bare context error; a deadline measured at the +attempt boundary with no transport in flight stays the stage overrun it is. Native `StructuredOutputError` is named directly, rather than reduced to a generic -command failure. Native HTTP evidence records the actual HTTP operation/status; -an `exit_code` of -1 there is explicitly marked as not a subprocess exit. -Cleanup failures preserve the original generation error and active ownership. +command failure, and an unrecognized provider execution-error name is reported as +a bounded, sanitized identifier plus any 4xx/5xx status instead of a blanket +summary (a name containing arbitrary bytes stays the generic message). Native HTTP +evidence records the actual HTTP operation/status; an `exit_code` of -1 there is +explicitly marked as not a subprocess exit. Cleanup failures preserve the original +generation error and active ownership. For existing native sessions with format-bearing messages, the HTTP readback workaround is read-only CLI `opencode export`. The exported session ID must match @@ -144,6 +158,11 @@ diagnostic: retry repeats work. - **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback failures, including the sanitized `malformed or incomplete` readback message. + A single classifier covers the two transport shapes: a pre-admission reach + failure (connecting or creating the session) and a post-admission failure + (readback/poll/agent execution). Both share the one `HARNESS TRANSPORT` label; + only the pre-admission shape is retried on a mutating stage, because it replays + nothing. A structured-output validation failure is never reported as a harness transport failure, and vice versa. Neither label includes a provider response body, prompt, @@ -166,8 +185,10 @@ isolated formatter hit its own 30-second deadline. The malformed payload was retrieved read-only with `pipelines inspect 55 --json` and reproduced the exact byte offset 7837. It is an unescaped quote/colon inside a string value - an ambiguous case - so the deterministic pass correctly reports it unrecoverable -rather than guessing, while the new derived deadline removes the fixed-deadline -failure mode for larger payloads. These are source-level findings covered by mocked +rather than guessing, while the stage-timeout-proportional derived deadline +(floored, capped and clamped to the remaining attempt budget) removes the +fixed-30-second-deadline failure mode for larger or slower payloads. These are +source-level findings covered by mocked `internal/stage` fixtures. An updated checkout or installed skill is **not** evidence that a user's running server contains the fix; deploying/restarting is a separate authorized action. diff --git a/docs/pipeline-performance-review.md b/docs/pipeline-performance-review.md index ec203403..d3ac3fd6 100644 --- a/docs/pipeline-performance-review.md +++ b/docs/pipeline-performance-review.md @@ -428,3 +428,31 @@ blocked-class cause distribution and the transience of transport errors are hypotheses, not measurements. Raw captures are under `/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and `/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo). + +## 7. Transport and repair recovery (source-level, 2026-10-17) + +Read-only pass over the §1 diagnostics after the harness-transport and +format-repair change. No run was started, retried or cancelled; no server, ntfy, +lane or device was restarted or redeployed; no provider inference was performed. +Each shape is mapped to the code path that produces it. The per-run +pre-admission vs post-admission and read-only vs mutating split was **not** +re-inspected with `pipelines inspect`, so every per-run row is labelled *not +verified* and the counts are not summed into a "recovered" total. + +| failure shape (§1 class, quoted diagnostic) | run ids | code path | can recover | remains unrecoverable | +|---|---|---|---|---| +| execution error at readback (`OpenCode agent reported an execution error`) | 73,74,75,77,80,149,150,152,195,202 | post-admission `completedAgentResult` / `opencode.ExecutionError` | a stage that is explicit read-only may retry within `retries`(*not verified per run*) | a mutating/legacy stage never replays admitted work; a non-transient provider error stays terminal | +| `context deadline exceeded` | 33,36,38,59,101,138 | attempt-boundary deadline, or a transport call that expired | a deadline during an in-flight request now keeps the transport cause and, when pre-admission, retries for any stage; a read-only stage retries a post-admission one | a genuine stage overrun on a mutating/legacy stage stays terminal; whether each run was a hung call or a real overrun is *not verified* | +| readback failure (`final assistant readback had no completed result`) | 161 | post-admission `completedAgentResult` | explicit read-only stage may retry | mutating/legacy stage never replays | +| native V1 `/wait` command failed (`CLI exit code 1`) | 4 | post-admission `openCodeError` on `/wait` | explicit read-only stage may retry | mutating/legacy stage never replays | +| repair timeout (`isolated format repair failed … context deadline exceeded`) | 55,81,174,179,183,221 | `repairDeadline` (payload bytes + stage timeout, floored, capped, clamped to remaining − margin) | a slow-but-repairable payload now gets a longer bounded budget | run 55's payload is the ambiguous unescaped-quote/colon class (byte offset 7837) → unrecoverable; any payload past the 32 KiB bound stays unrecoverable | + +Mapping notes: the execution-error and readback shapes are read back *after* the +prompt was admitted, so they are post-admission transport failures and the +no-tool-replay rule keeps them terminal on a mutating stage. The +`session_transport_command_failed` shape (run 4) is the native V1 `/wait` call +issued after the prompt, not a session-create reach failure. The repair-timeout +shapes are `FINAL RESULT VALIDATION` (repair-stage), not transport. The new code +also reports an unknown provider execution-error name as a bounded sanitized +identifier plus any HTTP status, so future occurrences of runs 73–202 are more +precisely diagnostic even when they are not retried. diff --git a/internal/author/skill/references/debugging.md b/internal/author/skill/references/debugging.md index 35f5e398..9ab1df5d 100644 --- a/internal/author/skill/references/debugging.md +++ b/internal/author/skill/references/debugging.md @@ -20,14 +20,18 @@ Each playbook is *symptom → how to confirm → what to do*. the model may have done work but no final result was read. - **Confirm** `inspect --json` shows the category on the failed stage. A request failure, an active-session poll failure and a final-readback failure all carry this - one category. + one category. A request that failed while the context was still in flight shows + both the sanitized transport detail and `context deadline exceeded`; an execution + error names the sanitized error name/tag and any HTTP status, not a provider body. - **Do** Treat it as a transport failure, not a task failure. Distinguish a - pre-admission failure (nothing executed) from a post-admission one (tools may have - run). An explicit read-only stage may retry; a stage whose tools include `edit`/ - `shell`, or that has no tools block, **never replays admitted work**. Do not re-run - a full agent stage merely to recover a response's formatting. If it recurs, check - the service and harness health (`references/operations.md`); do not weaken the - stage to get past it. + pre-admission reach failure (connecting or creating the session; nothing executed) + from a post-admission one (tools may have run). A pre-admission reach failure + replays nothing, so it is retried automatically within `retries` for any stage; a + post-admission failure is retried only on an explicit read-only stage. A stage + whose tools include `edit`/`shell`, or that has no tools block, **never replays + admitted work**. Do not re-run a full agent stage merely to recover a response's + formatting. If it recurs, check the service and harness health + (`references/operations.md`); do not weaken the stage to get past it. ## Malformed model results @@ -39,8 +43,9 @@ Each playbook is *symptom → how to confirm → what to do*. - **Do** Recovery is bounded and work-preserving. A deterministic local pass first appends missing closers, drops one trailing comma, or inserts the single comma/colon the parser expected; only if that is ambiguous does one deny-all - formatter request run, with a deadline derived from the payload size and the - attempt's remaining budget. Each path records a `repaired`, `partial` or + formatter request run, with a deadline derived from the payload size, the stage + timeout and the attempt's remaining budget, clamped so it never extends the + stage's own deadline. Each path records a `repaired`, `partial` or `unrecoverable` status in `format_repairs`; a partial or unrecoverable payload is never accepted as complete. On an explicit read-only stage a malformed result may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block diff --git a/internal/author/skill/references/operations.md b/internal/author/skill/references/operations.md index 6c5561c1..f01f5ff2 100644 --- a/internal/author/skill/references/operations.md +++ b/internal/author/skill/references/operations.md @@ -350,7 +350,11 @@ admitted tool-using stage: completed edits and tool side effects are retained an not undone, so a full retry repeats work. An explicit read-only stage is the one exception: a malformed result there mutates nothing, so it may be re-emitted up to `retries` with its `format_repairs` evidence retained, and a post-admission -deadline or transport failure may be retried. A stage whose tools include +deadline or transport failure may be retried. A failure to reach the harness +before any prompt or tool call is admitted (connecting, or creating the session) +replays nothing, so it is retried within `retries` for any stage, mutating or not. +A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell bundle), is never replayed. Neither category exposes provider/model bodies or -credentials. +credentials. The isolated formatter's deadline is proportional to the stage +timeout and clamped so it never extends the stage's own deadline. diff --git a/internal/author/skill/references/verification.md b/internal/author/skill/references/verification.md index f5a1dc9b..6e8be281 100644 --- a/internal/author/skill/references/verification.md +++ b/internal/author/skill/references/verification.md @@ -80,7 +80,10 @@ specific authorization. Retained model-stage failures now carry a fixed sanitized category. A strictly invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a byte offset) is labeled `FINAL RESULT VALIDATION`; request, V2 active-session poll -and final-readback failures are labeled `HARNESS TRANSPORT`. A validation failure +and final-readback failures are labeled `HARNESS TRANSPORT`. A pre-admission +request failure replays nothing and is retried for any stage; a post-admission +failure is retried only on an explicit read-only stage (see the 2026-10-17 section +below). A validation failure preserves the admitted session, transcript and evidence and performs no second tool execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history readback of exactly 262144 bytes of incomplete JSON (`invalid agent messages`), a @@ -100,8 +103,9 @@ Source-level, mocked `internal/stage` regression coverage (not live-provider evidence and not a deployment): malformed final-result recovery now tries a deterministic local pass first (append missing closers, drop one trailing comma, or insert the single comma/colon the parser expected), then at most one deny-all -formatter request whose deadline is derived from the payload size and the -attempt's remaining budget rather than a fixed 30-second constant. Every path +formatter request whose deadline is derived from the payload size, the stage +timeout and the attempt's remaining budget rather than a fixed 30-second +constant. Every path records `repaired`, `partial` or `unrecoverable` in `format_repairs`; a `partial` or `unrecoverable` payload is never returned as a completed result, and an ambiguous payload (an unescaped quote/colon inside a string value) is never @@ -112,6 +116,26 @@ unchanged. The run-55 `ui_reviewer` payload was retrieved read-only with ambiguous error class, so it is reported unrecoverable rather than repaired. This does not update or restart any running server. +## Harness transport reach retry and repair deadline: 2026-10-17 + +Source-level, mocked `internal/stage`, `internal/opencode` and `internal/server` +regression coverage (not live-provider evidence and not a deployment): the +harness transport path now distinguishes a pre-admission reach failure +(connecting or creating the session) from a post-admission readback/poll/agent +failure under the one `HARNESS TRANSPORT` category. A pre-admission reach failure +replays nothing, so it is retried within `retries` for any stage with a fresh +session and no replayed prompt; a post-admission failure is retried only on an +explicit read-only stage, and an `edit`/`shell`/legacy stage still never replays +admitted work. A context that expires during a request keeps both the sanitized +transport detail (operation/status/exit) and +`errors.Is(err, context.DeadlineExceeded)`; a bare attempt-boundary deadline is +not treated as a reach failure and stays terminal for a mutating stage. An +unrecognized provider execution-error name is reported as a bounded, sanitized +identifier plus any HTTP status. The isolated formatter deadline is now +proportional to the stage timeout (floored, capped, and clamped to +`remaining - margin` so it can never extend the stage/attempt deadline). These are +mocked fixtures; a deployed server is not updated by this change. + ## Failure-handling source changes: 2026-10-15 Source-level, mocked `internal/stage` and `internal/server` regression coverage diff --git a/internal/opencode/format.go b/internal/opencode/format.go index 3ad7bff1..3d36e51f 100644 --- a/internal/opencode/format.go +++ b/internal/opencode/format.go @@ -81,6 +81,15 @@ func ExecutionError(data json.RawMessage) error { } safeNames := map[string]bool{"APIError": true, "ProviderAuthError": true, "ContextOverflowError": true, "MessageAbortedError": true, "OutputLengthError": true, "UnknownError": true, "ServiceUnavailableError": true} if !safeNames[name] { + // An unrecognized provider error name is still reported, but only as a + // bounded, safe identifier plus any HTTP status. No provider body, + // message, prompt or credential is copied. + if safe := sanitizedExecutionName(name); safe != "" { + if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 { + return fmt.Errorf("OpenCode agent reported an execution error: %s (HTTP %d)", safe, failure.Data.StatusCode) + } + return fmt.Errorf("OpenCode agent reported an execution error: %s", safe) + } return fmt.Errorf("OpenCode agent reported an execution error") } if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 { @@ -88,3 +97,23 @@ func ExecutionError(data json.RawMessage) error { } return fmt.Errorf("OpenCode agent execution error: %s", name) } + +// sanitizedExecutionName reduces an unknown provider error name/tag to a +// bounded, safe identifier so the diagnostic can name the failure shape without +// copying an arbitrary message, prompt or credential. A name containing any +// other byte returns empty and is reported as a generic execution error. +func sanitizedExecutionName(name string) string { + if len(name) > 64 { + name = name[:64] + } + var b strings.Builder + for _, r := range name { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_', r == '.', r == '-': + b.WriteRune(r) + default: + return "" + } + } + return b.String() +} diff --git a/internal/opencode/format_test.go b/internal/opencode/format_test.go index 0d728ae1..c8cdab69 100644 --- a/internal/opencode/format_test.go +++ b/internal/opencode/format_test.go @@ -90,3 +90,26 @@ func TestNativeStructuredFailureIsClearAndDoesNotReplayGeneration(t *testing.T) t.Fatal("wait failure discarded provider response", err, string(retained)) } } + +// TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody proves an +// unrecognized provider error name is reported as a bounded, safe identifier +// plus any HTTP status instead of a blanket summary, while no provider message, +// prompt or credential is copied and known names keep their exact wording. +func TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody(t *testing.T) { + err := ExecutionError(json.RawMessage(`{"name":"TotallyUnknownError","data":{"statusCode":502,"message":"secret provider body"}}`)) + if err == nil || !strings.Contains(err.Error(), "TotallyUnknownError") || !strings.Contains(err.Error(), "HTTP 502") { + t.Fatalf("unknown error name/status not reported: %v", err) + } + if strings.Contains(err.Error(), "secret provider body") { + t.Fatalf("diagnostic leaked the provider body: %v", err) + } + if err := ExecutionError(json.RawMessage(`{"_tag":"SomeTagError"}`)); err == nil || !strings.Contains(err.Error(), "SomeTagError") { + t.Fatalf("unknown tag not reported: %v", err) + } + if err := ExecutionError(json.RawMessage(`{"name":"Injected Error: Bearer secret-token"}`)); err == nil || strings.Contains(err.Error(), "secret-token") || strings.Contains(err.Error(), "Bearer") || err.Error() != "OpenCode agent reported an execution error" { + t.Fatalf("unsafe name was copied into the diagnostic: %v", err) + } + if err := ExecutionError(json.RawMessage(`{"name":"APIError","data":{"statusCode":500}}`)); err == nil || !strings.Contains(err.Error(), "APIError (HTTP 500)") { + t.Fatalf("known safe name changed: %v", err) + } +} diff --git a/internal/server/retry_test.go b/internal/server/retry_test.go index 930cf7b5..acde318d 100644 --- a/internal/server/retry_test.go +++ b/internal/server/retry_test.go @@ -247,3 +247,97 @@ func TestRetryRejectsUnavailableOriginalDirectoryAndNonfailedRuns(t *testing.T) t.Fatal("nonexistent run accepted") } } + +// TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot proves, +// through the real runner and CLI fixture, that a read-only stage retries a +// post-admission transport (final-readback) failure with a fresh session while +// an edit stage never replays the admitted prompt, and that the failed step's +// diagnostic retains the underlying transport cause. +func TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + dir := t.TempDir() + executable := filepath.Join(dir, "opencode") + t.Setenv("PIPELINES_OPENCODE", executable) + scriptTemplate := `#!/bin/sh +if [ "$1" = "--help" ]; then echo ' api'; exit 0; fi +printf '%s %s\n' "$2" "$3" >> "@STATE@/calls" +case "$3" in + /api/session) + n=0; if [ -f "@STATE@/sessions" ]; then n=$(cat "@STATE@/sessions"); fi + n=$((n+1)); echo "$n" > "@STATE@/sessions" + printf '{"data":{"id":"ses_bounded_%s"}}' "$n" ;; + */prompt) echo '{}' ;; + /api/session/active) echo '{"data":{}}' ;; + */message*) + r=0; if [ -f "@STATE@/readbacks" ]; then r=$(cat "@STATE@/readbacks"); fi + r=$((r+1)); echo "$r" > "@STATE@/readbacks" + if [ "$r" = "1" ]; then + echo '{"data":[{"finish":"stop","content":' + else + echo '{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}' + fi ;; + */interrupt) echo '{}' ;; + *) exit 99 ;; +esac +` + store, err := pipeline.Open(filepath.Join(dir, "runs.db")) + if err != nil { + t.Fatal(err) + } + defer store.Close() + r, err := newRunner(ctx, store) + if err != nil { + t.Fatal(err) + } + defer func() { cancel(); r.wg.Wait() }() + toolsBlock := func(tools string) string { + return "# stage worker\n\n```output\nResult.\n```\n\n```prompt\nDo the work.\n```\n\n```tools\n" + tools + "\n```\n\n```edges\nsuccess -> #success\nerror -> #failure\n```\n\n```configs\ntype=model\nprovider=opencode-go\nmodel=deepseek-v4.1-flash\nretries=1\ntimeout=30s\n```\n" + } + runStage := func(name, tools string) (status, sessions, diagnostic string) { + t.Helper() + workspace := t.TempDir() + script := strings.ReplaceAll(scriptTemplate, "@STATE@", workspace) + if err := os.WriteFile(executable, []byte(script), 0700); err != nil { + t.Fatal(err) + } + if _, err := store.Submit(ctx, name, toolsBlock(tools)); err != nil { + t.Fatal(err) + } + items, _ := store.List(ctx) + var id int64 + for _, item := range items { + if item.Name == name { + id = item.ID + } + } + started, err := r.start(protocol.Message{PipelineID: id, Version: 1, RequestKey: name, Workspace: workspace, Input: map[string]any{"prompt": "task"}}) + if err != nil { + t.Fatal(err) + } + r.wg.Wait() + detail, err := store.GetRun(ctx, started.ID) + if err != nil { + t.Fatal(err) + } + if len(detail.Steps) > 0 { + diagnostic = detail.Steps[0].Diagnostic + } + if data, err := os.ReadFile(filepath.Join(workspace, "sessions")); err == nil { + sessions = strings.TrimSpace(string(data)) + } + return detail.Status, sessions, diagnostic + } + + status, sessions, diagnostic := runStage("readonly", "read") + if status != "succeeded" || sessions != "2" || diagnostic != "" { + t.Fatalf("read-only stage did not retry the transport readback: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic) + } + status, sessions, diagnostic = runStage("edit", "edit") + if status != "failed" || sessions != "1" { + t.Fatalf("edit stage replayed admitted work: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic) + } + if !strings.Contains(diagnostic, "HARNESS TRANSPORT") || !strings.Contains(diagnostic, "malformed or incomplete") { + t.Fatalf("step diagnostic lost the underlying transport cause: %q", diagnostic) + } +} diff --git a/internal/stage/agent.go b/internal/stage/agent.go index c3d4286a..e0c3974d 100644 --- a/internal/stage/agent.go +++ b/internal/stage/agent.go @@ -55,7 +55,8 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s if c.command == nil { connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend) if err != nil { - return "", err + // Connecting to the service failed before any session existed. + return "", harnessUnreachable(err) } defer connection.Close() c.connection = connection @@ -64,7 +65,9 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s rules := c.permissionRules() data, err := c.call(ctx, "post", "/api/session", map[string]any{"title": "Pipeline audited agent stage", "agent": "build", "location": map[string]string{"directory": c.Directory}, "model": map[string]string{"providerID": config.Provider, "id": config.Model}, "permissions": rules}) if err != nil { - return "", err + // No prompt or tool call was admitted: a request failure here is a + // reachable retry rather than a reason to end the run. + return "", harnessUnreachable(err) } var created struct { Data struct { diff --git a/internal/stage/model.go b/internal/stage/model.go index 4d1c4bfb..6128d1b5 100644 --- a/internal/stage/model.go +++ b/internal/stage/model.go @@ -27,15 +27,42 @@ const ( // diagnostics. It states the retained side effects and the cost of a full retry. const finalResultGuidance = "completed edits and tool side effects are retained and not undone; a full retry repeats work" +// harnessTransportFailure is the single classifier shared by both transport +// sentinels and the typed OpenCode error: a request, poll or readback failure +// that reports as the fixed HARNESS TRANSPORT category. A bare context deadline +// is classified separately so a genuine stage overrun is never relabeled as a +// transport failure. +func harnessTransportFailure(err error) bool { + var openCode *openCodeError + return errors.Is(err, ErrHarnessUnreachable) || errors.Is(err, ErrHarnessTransport) || errors.As(err, &openCode) +} + +// harnessReachFailure reports whether err is a pre-admission harness reach +// failure. It is distinct from a post-admission ErrHarnessTransport failure: no +// session or prompt was admitted, so the retry replays no work. +func harnessReachFailure(err error) bool { + return errors.Is(err, ErrHarnessUnreachable) +} + +// admittedWorker is implemented by a client that can admit tool work before its +// final result is read, so the retry loop can refuse to replay it without a +// per-client type switch. +type admittedWorker interface{ workAdmitted() bool } + +// workAdmitted reports whether the client admitted any prompt or tool call, so +// the retry loop may retry a transport failure on any stage only when it is +// false, and only a read-only stage otherwise. +func (c *AgentClient) workAdmitted() bool { return c.WorkStarted } +func (c *PiClient) workAdmitted() bool { return c.WorkStarted } + // modelFailureCategory classifies a stage failure using error identity only, so // the label stays sanitized and bounded. Validation is checked first: a // structured-output failure is never reported as a transport failure and vice versa. func modelFailureCategory(err error) string { - var openCode *openCodeError switch { case errors.Is(err, ErrInvalidModelResult): return finalResultValidation - case errors.Is(err, ErrHarnessTransport), errors.As(err, &openCode), errors.Is(err, context.DeadlineExceeded): + case harnessTransportFailure(err), errors.Is(err, context.DeadlineExceeded): return harnessTransport default: return "" @@ -294,15 +321,19 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu break } replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err) - // Never blindly replay admitted tool work: only a read-only stage may - // retry a post-admission deadline or transport failure. - if agent, ok := client.(*AgentClient); ok && agent.WorkStarted && !replayPostAdmission { + // A reach failure observed before any work was admitted replays + // nothing, so it is retryable within the existing budget for any + // stage. After admission, only an explicit read-only stage may + // retry; admitted tool work is never replayed. + workStarted := false + if worker, ok := client.(admittedWorker); ok { + workStarted = worker.workAdmitted() + } + if workStarted && !replayPostAdmission { break } - if pi, ok := client.(*PiClient); ok && pi.WorkStarted && !replayPostAdmission { - break - } - if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission) { + preAdmissionReach := !workStarted && harnessReachFailure(err) + if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission && !preAdmissionReach) { break } } diff --git a/internal/stage/model_prompt.go b/internal/stage/model_prompt.go index 05f7e170..f6cc3db2 100644 --- a/internal/stage/model_prompt.go +++ b/internal/stage/model_prompt.go @@ -18,6 +18,23 @@ var ErrInvalidModelResult = errors.New("invalid model stage result") // validation and never replay admitted tool work. var ErrHarnessTransport = errors.New("harness transport failure") +// ErrHarnessUnreachable marks a request/transport failure observed before the +// harness admitted any agent work: connecting to the service, or creating the +// session. Nothing was executed or admitted, so re-attempting replays no prompt +// or tool call and is safe for any stage, mutating or not. Post-admission +// readback, poll and agent-execution failures stay ErrHarnessTransport so the +// single fixed HARNESS TRANSPORT category and the no-tool-replay rule are +// unchanged. +var ErrHarnessUnreachable = errors.New("harness unreachable") + +// harnessUnreachable marks err as a pre-admission reach failure. It is the single +// wrap shared by the connect and session-create paths of both client types, so +// they cannot drift: the sentinel and the underlying cause are always both +// preserved. +func harnessUnreachable(err error) error { + return fmt.Errorf("%w: %w", ErrHarnessUnreachable, err) +} + func modelPrompt(def Definition, spec ModelSpec, input Values) (string, error) { outcomes := make([]string, 0, len(def.Edges)) for name := range def.Edges { diff --git a/internal/stage/opencode.go b/internal/stage/opencode.go index 3f7fdbd1..38d7722e 100644 --- a/internal/stage/opencode.go +++ b/internal/stage/opencode.go @@ -97,10 +97,14 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any) evidenceStatus = c.connection.LastHTTPStatus } c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: evidenceStatus, NativeOperation: c.connection.LastHTTPOperation, Diagnostic: err.Error(), Stdout: string(data)}, started) - if ctx.Err() != nil { - return nil, ctx.Err() + transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()} + if ctxErr := ctx.Err(); ctxErr != nil { + // Keep the transport shape (operation/status) with the context + // cause: a deadline that fired while the request was in flight is + // both, and neither may be discarded. + return nil, errors.Join(ctxErr, transport) } - return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()} + return nil, transport } return data, nil } @@ -146,14 +150,18 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any) code = exit.ExitCode() } c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: cmd.Path, Directory: directory, ExitCode: code, HTTPStatus: status, Diagnostic: safeCommandDiagnostic(err, stdout.Bytes()), Stdout: stdout.String(), Stderr: stderr.String()}, started) - if ctx.Err() != nil { - return nil, ctx.Err() - } // Do not embed provider responses, prompts, credentials, or CLI logs in errors. // Keep the raw cause so errors.Is(err, exec.ErrWaitDelay) still holds while // Error() stays sanitized. This branch is only reached with a non-nil Run // error, so valid-looking partial stdout is never returned as success. - return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())} + transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())} + if ctxErr := ctx.Err(); ctxErr != nil { + // Keep the sanitized transport detail (operation/status/exit) together + // with errors.Is(err, context.DeadlineExceeded) so the true cause is + // not replaced by a bare context error. + return nil, errors.Join(ctxErr, transport) + } + return nil, transport } if stdout.overflow { return nil, fmt.Errorf("OpenCode response exceeds 1 MiB limit") @@ -259,7 +267,8 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp if c.command == nil { connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend) if err != nil { - return "", err + // Connecting to the service failed before any session existed. + return "", harnessUnreachable(err) } defer connection.Close() c.connection = connection @@ -288,7 +297,9 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp "permissions": []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}}, }) if err != nil { - return "", err + // No prompt or tool call was admitted: a request failure here is a + // reachable retry rather than a reason to end the run. + return "", harnessUnreachable(err) } var session struct { Data struct { diff --git a/internal/stage/result_repair.go b/internal/stage/result_repair.go index 948faae7..70e2ed60 100644 --- a/internal/stage/result_repair.go +++ b/internal/stage/result_repair.go @@ -129,14 +129,15 @@ func repairModelResult(ctx context.Context, client ModelGenerator, config ModelC return Result{}, fmt.Errorf("%w; no isolated format repair available; recovery status %s", parseError, repairStatusUnrecoverable) } // One formatter request, independent of execution/transport retries. The - // deadline is derived from the payload size and the attempt's own remaining - // budget rather than a fixed short constant, so a larger result is not - // guaranteed to time out while the attempt context remains the outer bound. + // deadline is derived from the payload size, the stage timeout and the + // attempt's own remaining budget rather than a fixed short constant, so a + // larger or slower result is not guaranteed to time out while the attempt + // context remains the outer bound. remaining := time.Hour if deadline, ok := ctx.Deadline(); ok { remaining = time.Until(deadline) } - repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining)) + repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining, config.Timeout)) defer cancel() candidate, err := formatter.RepairFormat(repairCtx, config, original, parseError) if err := ctx.Err(); err != nil { @@ -291,9 +292,10 @@ func lexemeOffsets(text string, tokens []string) ([]int, bool) { } // Isolated formatter deadline bounds. The allowance grows by one second per KiB -// of payload up to a ceiling, is floored for ordinary payloads, and is clamped -// to the attempt's remaining budget so the stage/attempt context stays the outer -// bound and a large result is not guaranteed to time out. +// of payload up to a ceiling and by a share of the stage timeout, is floored for +// ordinary payloads, and is clamped to the attempt's remaining budget so the +// stage/attempt context stays the outer bound and a large or slow result is not +// guaranteed to time out. const ( repairDeadlineFloor = 30 * time.Second repairDeadlineMax = 10 * time.Minute @@ -301,13 +303,13 @@ const ( ) // repairDeadline derives the formatter deadline. It is O(1) and monotonic -// non-decreasing in payloadBytes; it never exceeds remaining and, when the -// remaining budget allows, never drops below repairDeadlineFloor. -func repairDeadline(payloadBytes int, remaining time.Duration) time.Duration { +// non-decreasing in payloadBytes and stageTimeout; it never exceeds remaining +// and, when the remaining budget allows, never drops below repairDeadlineFloor. +func repairDeadline(payloadBytes int, remaining, stageTimeout time.Duration) time.Duration { if remaining <= 0 { return 0 } - allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second + allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second + stageTimeout/4 if allowance > repairDeadlineMax { allowance = repairDeadlineMax } diff --git a/internal/stage/result_repair_test.go b/internal/stage/result_repair_test.go index 88d4b5f6..d507f892 100644 --- a/internal/stage/result_repair_test.go +++ b/internal/stage/result_repair_test.go @@ -306,26 +306,37 @@ func TestPresentationStrippingBeforeDeterministicRepair(t *testing.T) { // TestRepairDeadlineIsMonotonicFlooredAndClamped proves the derived deadline, not // a fixed constant, bounds the single formatter request. func TestRepairDeadlineIsMonotonicFlooredAndClamped(t *testing.T) { - if got := repairDeadline(0, time.Hour); got != repairDeadlineFloor { + if got := repairDeadline(0, time.Hour, time.Minute); got < repairDeadlineFloor || got > repairDeadlineMax { t.Fatalf("floor not applied: %s", got) } - if got := repairDeadline(1<<30, 24*time.Hour); got != repairDeadlineMax { + if got := repairDeadline(1<<30, 24*time.Hour, time.Hour); got != repairDeadlineMax { t.Fatalf("ceiling not applied: %s", got) } - if got := repairDeadline(10, 0); got != 0 { + if got := repairDeadline(10, 0, time.Hour); got != 0 { t.Fatalf("zero remaining should yield zero: %s", got) } - if clamped := repairDeadline(1<<20, 5*time.Second); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second { + if clamped := repairDeadline(1<<20, 5*time.Second, time.Hour); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second { t.Fatalf("remaining budget was not the bound: %s", clamped) } previous := time.Duration(0) for size := 0; size <= 64<<10; size += 1024 { - got := repairDeadline(size, time.Hour) + got := repairDeadline(size, time.Hour, time.Minute) if got < previous { - t.Fatalf("deadline not monotonic at %d: %s < %s", size, got, previous) + t.Fatalf("deadline not monotonic in payload at %d: %s < %s", size, got, previous) } previous = got } + previous = time.Duration(0) + for stage := time.Duration(0); stage <= 2*time.Hour; stage += time.Minute { + got := repairDeadline(64<<10, time.Hour, stage) + if got < previous { + t.Fatalf("deadline not monotonic in stage timeout at %s: %s < %s", stage, got, previous) + } + previous = got + } + if got := repairDeadline(8<<10, time.Hour, time.Hour); got < repairDeadlineFloor { + t.Fatalf("ordinary payload was not floored: %s", got) + } } // TestShellStageRecoverableMalformedResultDoesNotFailRun proves a shell-enabled diff --git a/internal/stage/transport_retry_test.go b/internal/stage/transport_retry_test.go new file mode 100644 index 00000000..3e80336d --- /dev/null +++ b/internal/stage/transport_retry_test.go @@ -0,0 +1,197 @@ +package stage + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "time" + + "git.a-wan.net/alan/pipelines/internal/opencode" +) + +// TestPreAdmissionReachFailureRetriesOnMutatingStage proves a failure to reach +// the harness before any prompt or tool call is admitted retries within the +// existing budget even on an edit stage, with a fresh session and no replayed +// prompt. Nothing was admitted, so no admitted work can be replayed. +func TestPreAdmissionReachFailureRetriesOnMutatingStage(t *testing.T) { + def := modelDefinition() + def.Blocks["tools"] = "edit" + def.Configs["retries"] = "1" + var sessions, prompts atomic.Int64 + agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) { + path := args[2] + switch { + case path == "/api/session": + if sessions.Add(1) == 1 { + return nil, errors.New("connection refused") + } + return []byte(`{"data":{"id":"ses_reach_2"}}`), nil + case strings.HasSuffix(path, "/prompt"): + prompts.Add(1) + return []byte(`{}`), nil + case path == "/api/session/active": + return []byte(`{"data":{}}`), nil + case strings.HasSuffix(path, finalAssistantReadback): + return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}`), nil + } + return nil, fmt.Errorf("unexpected path %s", path) + }}} + result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil) + if err != nil || result.Output["summary"] != "done" { + t.Fatalf("pre-admission reach failure was not retried to success: %v %v", result, err) + } + if sessions.Load() != 2 || prompts.Load() != 1 || agent.SessionID != "ses_reach_2" { + t.Fatalf("reach retry did not use exactly one fresh session: sessions=%d prompts=%d session=%q", sessions.Load(), prompts.Load(), agent.SessionID) + } +} + +// TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage proves a readback +// transport failure observed after the prompt was admitted is retried only on an +// explicit read-only stage; an edit, shell or legacy-bundle stage never replays +// the admitted prompt and ends with a HARNESS TRANSPORT failure. +func TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage(t *testing.T) { + run := func(name string, setTools bool, tools string) (Result, error, int64, int64) { + t.Helper() + def := modelDefinition() + if setTools { + def.Blocks["tools"] = tools + } + def.Configs["retries"] = "1" + var sessions, prompts, readbacks atomic.Int64 + agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) { + path := args[2] + switch { + case path == "/api/session": + n := sessions.Add(1) + return []byte(fmt.Sprintf(`{"data":{"id":"ses_rb_%d"}}`, n)), nil + case strings.HasSuffix(path, "/prompt"): + prompts.Add(1) + return []byte(`{}`), nil + case path == "/api/session/active": + return []byte(`{"data":{}}`), nil + case strings.HasSuffix(path, finalAssistantReadback): + if readbacks.Add(1) == 1 { + // Truncated readback: a post-admission transport failure. + return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":`), nil + } + return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"ok\"}}"}]}]}`), nil + case strings.HasSuffix(path, "/interrupt"): + return []byte(`{}`), nil + } + return nil, fmt.Errorf("unexpected path %s", path) + }}} + result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil) + return result, err, sessions.Load(), prompts.Load() + } + + result, err, sessions, prompts := run("read-only", true, "read") + if err != nil || result.Output["summary"] != "ok" || sessions != 2 || prompts != 2 { + t.Fatalf("read-only readback failure was not retried: %v %v sessions=%d prompts=%d", result, err, sessions, prompts) + } + for _, tc := range []struct { + name string + setTools bool + tools string + }{ + {"edit", true, "edit"}, + {"shell", true, "shell"}, + {"legacy", false, ""}, + } { + result, err, sessions, prompts := run(tc.name, tc.setTools, tc.tools) + if !errors.Is(err, ErrHarnessTransport) || sessions != 1 || prompts != 1 { + t.Fatalf("%s stage replayed admitted work on a readback failure: err=%v sessions=%d prompts=%d", tc.name, err, sessions, prompts) + } + if result.Output != nil { + t.Fatalf("%s stage retained a partial result", tc.name) + } + if !strings.Contains(err.Error(), "HARNESS TRANSPORT") { + t.Fatalf("%s failure lost the transport category: %v", tc.name, err) + } + } +} + +// TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline proves an +// expired context during a real CLI request keeps both the sanitized transport +// detail (operation/exit) and errors.Is(err, context.DeadlineExceeded) instead +// of returning a bare context error that discards the cause. +func TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline(t *testing.T) { + dir := t.TempDir() + executable := filepath.Join(dir, "opencode") + if err := os.WriteFile(executable, []byte("#!/bin/sh\nexec sleep 30\n"), 0700); err != nil { + t.Fatal(err) + } + t.Setenv("PIPELINES_OPENCODE", executable) + client := OpenCodeClient{Directory: dir} + ctx, cancel := context.WithTimeout(context.Background(), 40*time.Millisecond) + defer cancel() + _, err := client.call(ctx, "post", "/api/session", nil) + if err == nil || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("expired transport call lost the context cause: %v", err) + } + var typed *openCodeError + if !errors.As(err, &typed) || typed.operation != "post /api/session" { + t.Fatalf("expired transport call discarded the typed transport error: %v", err) + } + if !strings.Contains(err.Error(), "OpenCode post /api/session") { + t.Fatalf("expired transport call discarded the operation detail: %v", err) + } + if !harnessTransportFailure(err) { + t.Fatalf("expired transport call was not classified as transport: %v", err) + } +} + +// TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline proves +// the native V1 branch keeps both the typed transport error (the operation +// detail) and errors.Is(err, context.DeadlineExceeded) when the request failed +// under an expired context, instead of returning a bare context error that +// discards the transport shape. The native request path folds an in-flight +// deadline into ctx.Err(), so the retained transport detail is the operation and +// the parsed HTTP status is 0; the important invariant is that both shapes +// survive the join. +func TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline(t *testing.T) { + client := OpenCodeClient{Backend: "v1", connection: &opencode.Connection{Backend: "v1", URL: "http://127.0.0.1:1"}} + ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Second)) + defer cancel() + _, err := client.call(ctx, "post", "/api/session/ses_native/generate", map[string]any{"prompt": "task"}) + if err == nil || !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("expired native transport call lost the context cause: %v", err) + } + var typed *openCodeError + if !errors.As(err, &typed) || typed.operation != "post /api/session/ses_native/generate" { + t.Fatalf("expired native transport call discarded the typed transport error: %v", err) + } + if !strings.Contains(err.Error(), "post /api/session/ses_native/generate") { + t.Fatalf("expired native transport call discarded the operation detail: %v", err) + } + if !harnessTransportFailure(err) { + t.Fatalf("expired native transport call was not classified as transport: %v", err) + } +} + +// TestHarnessTransportSentinelsShareOneClassifierAndCategory proves the two +// transport shapes share one classifier and the single HARNESS TRANSPORT label +// while a bare stage deadline stays distinct from a pre-admission reach failure. +func TestHarnessTransportSentinelsShareOneClassifierAndCategory(t *testing.T) { + unreachable := harnessUnreachable(errors.New("connect refused")) + if !harnessTransportFailure(unreachable) || !harnessReachFailure(unreachable) { + t.Fatal("pre-admission reach failure was not classified") + } + if !strings.Contains(wrapModelStageFailure("worker", unreachable).Error(), "HARNESS TRANSPORT") { + t.Fatalf("pre-admission reach failure lost the transport category: %v", wrapModelStageFailure("worker", unreachable)) + } + post := fmt.Errorf("%w: final assistant readback was malformed or incomplete", ErrHarnessTransport) + if !harnessTransportFailure(post) || harnessReachFailure(post) { + t.Fatal("post-admission transport failure was misclassified as a pre-admission reach failure") + } + if !strings.Contains(wrapModelStageFailure("worker", post).Error(), "HARNESS TRANSPORT") { + t.Fatal("post-admission transport failure lost the transport category") + } + if harnessTransportFailure(context.DeadlineExceeded) || harnessReachFailure(context.DeadlineExceeded) { + t.Fatal("a bare stage deadline must not be a transport or reach failure") + } +}