Merge lane-srv3: keep the lane's reviewed work

This commit is contained in:
2026-10-17 14:57:52 +01:00
15 changed files with 563 additions and 63 deletions
+33 -12
View File
@@ -82,10 +82,11 @@ Format recovery has a separate bounded budget, even with `retries=0`:
2. If the local pass cannot repair it, make at most one fresh, deny-all, retained
text-generation request with only the malformed response and precise parsing
error. The original implementing/reviewing agent is never re-prompted. The
formatter's deadline is derived from the payload size and the attempt's own
remaining budget rather than a fixed short constant, so a large result is not
guaranteed to time out; it stays within the stage deadline, is floored for
ordinary payloads, and keeps the 32 KiB input/output bound.
formatter's deadline is derived from the payload size, the stage timeout and
the attempt's own remaining budget rather than a fixed short constant, so a
large or slower result is not guaranteed to time out; it stays within the
stage deadline (clamped to `remaining` minus a margin), is floored for ordinary
payloads, and keeps the 32 KiB input/output bound.
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
with at most 16 insertions. Every existing string, number, literal and
structural delimiter must remain verbatim and in order. Validate the full
@@ -106,8 +107,14 @@ rerun project edits, commands, or the entire stage to repair final formatting.
Configured execution retries remain for retryable text-generation transport
errors; an explicit read-only stage may additionally re-emit a malformed final
result and retry a post-admission deadline or transport failure up to `retries`.
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
read/glob/grep/edit/shell bundle), never has admitted tool work replayed. The
A failure to reach the harness before any prompt or tool call is admitted
(connecting, or creating the session) is a distinct reach failure: nothing was
admitted, so it is retried within `retries` for any stage, mutating or not, with
a fresh session and no replayed prompt or tool call. A post-admission readback,
active-session poll or agent-execution failure is retried only on an explicit
read-only stage. A stage whose tools include `edit`/`shell`, or that has no tools
block (the legacy read/glob/grep/edit/shell bundle), never has admitted tool work
replayed. The
deterministic local pass and the fresh deny-all, tool-free formatter session never
re-prompt or replay the admitted stage session, so a shell-enabled review stage
gains malformed-result recovery without any replay of admitted work.
@@ -121,11 +128,18 @@ and a recognized safe error category (such as session not found or service
unavailable). Arbitrary stderr, provider messages and response bodies are not
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
is not evidence that the agent prompt failed to start, and does not trigger replay
of stage work.
of stage work. When the context expires while a request is still in flight, the
error keeps both the sanitized transport detail (operation/status/exit) and
`errors.Is(err, context.DeadlineExceeded)`, so a deadline that fired during a hung
transport call is not replaced by a bare context error; a deadline measured at the
attempt boundary with no transport in flight stays the stage overrun it is.
Native `StructuredOutputError` is named directly, rather than reduced to a generic
command failure. Native HTTP evidence records the actual HTTP operation/status;
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
Cleanup failures preserve the original generation error and active ownership.
command failure, and an unrecognized provider execution-error name is reported as
a bounded, sanitized identifier plus any 4xx/5xx status instead of a blanket
summary (a name containing arbitrary bytes stays the generic message). Native HTTP
evidence records the actual HTTP operation/status; an `exit_code` of -1 there is
explicitly marked as not a subprocess exit. Cleanup failures preserve the original
generation error and active ownership.
For existing native sessions with format-bearing messages, the HTTP readback
workaround is read-only CLI `opencode export`. The exported session ID must match
@@ -144,6 +158,11 @@ diagnostic:
retry repeats work.
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
failures, including the sanitized `malformed or incomplete` readback message.
A single classifier covers the two transport shapes: a pre-admission reach
failure (connecting or creating the session) and a post-admission failure
(readback/poll/agent execution). Both share the one `HARNESS TRANSPORT` label;
only the pre-admission shape is retried on a mutating stage, because it replays
nothing.
A structured-output validation failure is never reported as a harness transport
failure, and vice versa. Neither label includes a provider response body, prompt,
@@ -166,8 +185,10 @@ isolated formatter hit its own 30-second deadline. The malformed payload was
retrieved read-only with `pipelines inspect 55 --json` and reproduced the exact
byte offset 7837. It is an unescaped quote/colon inside a string value - an
ambiguous case - so the deterministic pass correctly reports it unrecoverable
rather than guessing, while the new derived deadline removes the fixed-deadline
failure mode for larger payloads. These are source-level findings covered by mocked
rather than guessing, while the stage-timeout-proportional derived deadline
(floored, capped and clamped to the remaining attempt budget) removes the
fixed-30-second-deadline failure mode for larger or slower payloads. These are
source-level findings covered by mocked
`internal/stage` fixtures. An updated checkout or installed skill is **not**
evidence that a user's running server contains the fix; deploying/restarting is a
separate authorized action.
+28
View File
@@ -428,3 +428,31 @@ blocked-class cause distribution and the transience of transport errors are
hypotheses, not measurements. Raw captures are under
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).
## 7. Transport and repair recovery (source-level, 2026-10-17)
Read-only pass over the §1 diagnostics after the harness-transport and
format-repair change. No run was started, retried or cancelled; no server, ntfy,
lane or device was restarted or redeployed; no provider inference was performed.
Each shape is mapped to the code path that produces it. The per-run
pre-admission vs post-admission and read-only vs mutating split was **not**
re-inspected with `pipelines inspect`, so every per-run row is labelled *not
verified* and the counts are not summed into a "recovered" total.
| failure shape (§1 class, quoted diagnostic) | run ids | code path | can recover | remains unrecoverable |
|---|---|---|---|---|
| execution error at readback (`OpenCode agent reported an execution error`) | 73,74,75,77,80,149,150,152,195,202 | post-admission `completedAgentResult` / `opencode.ExecutionError` | a stage that is explicit read-only may retry within `retries`(*not verified per run*) | a mutating/legacy stage never replays admitted work; a non-transient provider error stays terminal |
| `context deadline exceeded` | 33,36,38,59,101,138 | attempt-boundary deadline, or a transport call that expired | a deadline during an in-flight request now keeps the transport cause and, when pre-admission, retries for any stage; a read-only stage retries a post-admission one | a genuine stage overrun on a mutating/legacy stage stays terminal; whether each run was a hung call or a real overrun is *not verified* |
| readback failure (`final assistant readback had no completed result`) | 161 | post-admission `completedAgentResult` | explicit read-only stage may retry | mutating/legacy stage never replays |
| native V1 `/wait` command failed (`CLI exit code 1`) | 4 | post-admission `openCodeError` on `/wait` | explicit read-only stage may retry | mutating/legacy stage never replays |
| repair timeout (`isolated format repair failed … context deadline exceeded`) | 55,81,174,179,183,221 | `repairDeadline` (payload bytes + stage timeout, floored, capped, clamped to remaining − margin) | a slow-but-repairable payload now gets a longer bounded budget | run 55's payload is the ambiguous unescaped-quote/colon class (byte offset 7837) → unrecoverable; any payload past the 32 KiB bound stays unrecoverable |
Mapping notes: the execution-error and readback shapes are read back *after* the
prompt was admitted, so they are post-admission transport failures and the
no-tool-replay rule keeps them terminal on a mutating stage. The
`session_transport_command_failed` shape (run 4) is the native V1 `/wait` call
issued after the prompt, not a session-create reach failure. The repair-timeout
shapes are `FINAL RESULT VALIDATION` (repair-stage), not transport. The new code
also reports an unknown provider execution-error name as a bounded sanitized
identifier plus any HTTP status, so future occurrences of runs 73–202 are more
precisely diagnostic even when they are not retried.
+14 -9
View File
@@ -20,14 +20,18 @@ Each playbook is *symptom → how to confirm → what to do*.
the model may have done work but no final result was read.
- **Confirm** `inspect --json` shows the category on the failed stage. A request
failure, an active-session poll failure and a final-readback failure all carry this
one category.
one category. A request that failed while the context was still in flight shows
both the sanitized transport detail and `context deadline exceeded`; an execution
error names the sanitized error name/tag and any HTTP status, not a provider body.
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
pre-admission failure (nothing executed) from a post-admission one (tools may have
run). An explicit read-only stage may retry; a stage whose tools include `edit`/
`shell`, or that has no tools block, **never replays admitted work**. Do not re-run
a full agent stage merely to recover a response's formatting. If it recurs, check
the service and harness health (`references/operations.md`); do not weaken the
stage to get past it.
pre-admission reach failure (connecting or creating the session; nothing executed)
from a post-admission one (tools may have run). A pre-admission reach failure
replays nothing, so it is retried automatically within `retries` for any stage; a
post-admission failure is retried only on an explicit read-only stage. A stage
whose tools include `edit`/`shell`, or that has no tools block, **never replays
admitted work**. Do not re-run a full agent stage merely to recover a response's
formatting. If it recurs, check the service and harness health
(`references/operations.md`); do not weaken the stage to get past it.
## Malformed model results
@@ -39,8 +43,9 @@ Each playbook is *symptom → how to confirm → what to do*.
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
appends missing closers, drops one trailing comma, or inserts the single
comma/colon the parser expected; only if that is ambiguous does one deny-all
formatter request run, with a deadline derived from the payload size and the
attempt's remaining budget. Each path records a `repaired`, `partial` or
formatter request run, with a deadline derived from the payload size, the stage
timeout and the attempt's remaining budget, clamped so it never extends the
stage's own deadline. Each path records a `repaired`, `partial` or
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
is never accepted as complete. On an explicit read-only stage a malformed result
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
@@ -350,7 +350,11 @@ admitted tool-using stage: completed edits and tool side effects are retained an
not undone, so a full retry repeats work. An explicit read-only stage is the one
exception: a malformed result there mutates nothing, so it may be re-emitted up
to `retries` with its `format_repairs` evidence retained, and a post-admission
deadline or transport failure may be retried. A stage whose tools include
deadline or transport failure may be retried. A failure to reach the harness
before any prompt or tool call is admitted (connecting, or creating the session)
replays nothing, so it is retried within `retries` for any stage, mutating or not.
A stage whose tools include
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
bundle), is never replayed. Neither category exposes provider/model bodies or
credentials.
credentials. The isolated formatter's deadline is proportional to the stage
timeout and clamped so it never extends the stage's own deadline.
@@ -80,7 +80,10 @@ specific authorization.
Retained model-stage failures now carry a fixed sanitized category. A strictly
invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a
byte offset) is labeled `FINAL RESULT VALIDATION`; request, V2 active-session poll
and final-readback failures are labeled `HARNESS TRANSPORT`. A validation failure
and final-readback failures are labeled `HARNESS TRANSPORT`. A pre-admission
request failure replays nothing and is retried for any stage; a post-admission
failure is retried only on an explicit read-only stage (see the 2026-10-17 section
below). A validation failure
preserves the admitted session, transcript and evidence and performs no second tool
execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history
readback of exactly 262144 bytes of incomplete JSON (`invalid agent messages`), a
@@ -100,8 +103,9 @@ Source-level, mocked `internal/stage` regression coverage (not live-provider
evidence and not a deployment): malformed final-result recovery now tries a
deterministic local pass first (append missing closers, drop one trailing comma,
or insert the single comma/colon the parser expected), then at most one deny-all
formatter request whose deadline is derived from the payload size and the
attempt's remaining budget rather than a fixed 30-second constant. Every path
formatter request whose deadline is derived from the payload size, the stage
timeout and the attempt's remaining budget rather than a fixed 30-second
constant. Every path
records `repaired`, `partial` or `unrecoverable` in `format_repairs`; a `partial`
or `unrecoverable` payload is never returned as a completed result, and an
ambiguous payload (an unescaped quote/colon inside a string value) is never
@@ -112,6 +116,26 @@ unchanged. The run-55 `ui_reviewer` payload was retrieved read-only with
ambiguous error class, so it is reported unrecoverable rather than repaired. This
does not update or restart any running server.
## Harness transport reach retry and repair deadline: 2026-10-17
Source-level, mocked `internal/stage`, `internal/opencode` and `internal/server`
regression coverage (not live-provider evidence and not a deployment): the
harness transport path now distinguishes a pre-admission reach failure
(connecting or creating the session) from a post-admission readback/poll/agent
failure under the one `HARNESS TRANSPORT` category. A pre-admission reach failure
replays nothing, so it is retried within `retries` for any stage with a fresh
session and no replayed prompt; a post-admission failure is retried only on an
explicit read-only stage, and an `edit`/`shell`/legacy stage still never replays
admitted work. A context that expires during a request keeps both the sanitized
transport detail (operation/status/exit) and
`errors.Is(err, context.DeadlineExceeded)`; a bare attempt-boundary deadline is
not treated as a reach failure and stays terminal for a mutating stage. An
unrecognized provider execution-error name is reported as a bounded, sanitized
identifier plus any HTTP status. The isolated formatter deadline is now
proportional to the stage timeout (floored, capped, and clamped to
`remaining - margin` so it can never extend the stage/attempt deadline). These are
mocked fixtures; a deployed server is not updated by this change.
## Failure-handling source changes: 2026-10-15
Source-level, mocked `internal/stage` and `internal/server` regression coverage
+29
View File
@@ -81,6 +81,15 @@ func ExecutionError(data json.RawMessage) error {
}
safeNames := map[string]bool{"APIError": true, "ProviderAuthError": true, "ContextOverflowError": true, "MessageAbortedError": true, "OutputLengthError": true, "UnknownError": true, "ServiceUnavailableError": true}
if !safeNames[name] {
// An unrecognized provider error name is still reported, but only as a
// bounded, safe identifier plus any HTTP status. No provider body,
// message, prompt or credential is copied.
if safe := sanitizedExecutionName(name); safe != "" {
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
return fmt.Errorf("OpenCode agent reported an execution error: %s (HTTP %d)", safe, failure.Data.StatusCode)
}
return fmt.Errorf("OpenCode agent reported an execution error: %s", safe)
}
return fmt.Errorf("OpenCode agent reported an execution error")
}
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
@@ -88,3 +97,23 @@ func ExecutionError(data json.RawMessage) error {
}
return fmt.Errorf("OpenCode agent execution error: %s", name)
}
// sanitizedExecutionName reduces an unknown provider error name/tag to a
// bounded, safe identifier so the diagnostic can name the failure shape without
// copying an arbitrary message, prompt or credential. A name containing any
// other byte returns empty and is reported as a generic execution error.
func sanitizedExecutionName(name string) string {
if len(name) > 64 {
name = name[:64]
}
var b strings.Builder
for _, r := range name {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_', r == '.', r == '-':
b.WriteRune(r)
default:
return ""
}
}
return b.String()
}
+23
View File
@@ -90,3 +90,26 @@ func TestNativeStructuredFailureIsClearAndDoesNotReplayGeneration(t *testing.T)
t.Fatal("wait failure discarded provider response", err, string(retained))
}
}
// TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody proves an
// unrecognized provider error name is reported as a bounded, safe identifier
// plus any HTTP status instead of a blanket summary, while no provider message,
// prompt or credential is copied and known names keep their exact wording.
func TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody(t *testing.T) {
err := ExecutionError(json.RawMessage(`{"name":"TotallyUnknownError","data":{"statusCode":502,"message":"secret provider body"}}`))
if err == nil || !strings.Contains(err.Error(), "TotallyUnknownError") || !strings.Contains(err.Error(), "HTTP 502") {
t.Fatalf("unknown error name/status not reported: %v", err)
}
if strings.Contains(err.Error(), "secret provider body") {
t.Fatalf("diagnostic leaked the provider body: %v", err)
}
if err := ExecutionError(json.RawMessage(`{"_tag":"SomeTagError"}`)); err == nil || !strings.Contains(err.Error(), "SomeTagError") {
t.Fatalf("unknown tag not reported: %v", err)
}
if err := ExecutionError(json.RawMessage(`{"name":"Injected Error: Bearer secret-token"}`)); err == nil || strings.Contains(err.Error(), "secret-token") || strings.Contains(err.Error(), "Bearer") || err.Error() != "OpenCode agent reported an execution error" {
t.Fatalf("unsafe name was copied into the diagnostic: %v", err)
}
if err := ExecutionError(json.RawMessage(`{"name":"APIError","data":{"statusCode":500}}`)); err == nil || !strings.Contains(err.Error(), "APIError (HTTP 500)") {
t.Fatalf("known safe name changed: %v", err)
}
}
+94
View File
@@ -247,3 +247,97 @@ func TestRetryRejectsUnavailableOriginalDirectoryAndNonfailedRuns(t *testing.T)
t.Fatal("nonexistent run accepted")
}
}
// TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot proves,
// through the real runner and CLI fixture, that a read-only stage retries a
// post-admission transport (final-readback) failure with a fresh session while
// an edit stage never replays the admitted prompt, and that the failed step's
// diagnostic retains the underlying transport cause.
func TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
dir := t.TempDir()
executable := filepath.Join(dir, "opencode")
t.Setenv("PIPELINES_OPENCODE", executable)
scriptTemplate := `#!/bin/sh
if [ "$1" = "--help" ]; then echo ' api'; exit 0; fi
printf '%s %s\n' "$2" "$3" >> "@STATE@/calls"
case "$3" in
/api/session)
n=0; if [ -f "@STATE@/sessions" ]; then n=$(cat "@STATE@/sessions"); fi
n=$((n+1)); echo "$n" > "@STATE@/sessions"
printf '{"data":{"id":"ses_bounded_%s"}}' "$n" ;;
*/prompt) echo '{}' ;;
/api/session/active) echo '{"data":{}}' ;;
*/message*)
r=0; if [ -f "@STATE@/readbacks" ]; then r=$(cat "@STATE@/readbacks"); fi
r=$((r+1)); echo "$r" > "@STATE@/readbacks"
if [ "$r" = "1" ]; then
echo '{"data":[{"finish":"stop","content":'
else
echo '{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}'
fi ;;
*/interrupt) echo '{}' ;;
*) exit 99 ;;
esac
`
store, err := pipeline.Open(filepath.Join(dir, "runs.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
r, err := newRunner(ctx, store)
if err != nil {
t.Fatal(err)
}
defer func() { cancel(); r.wg.Wait() }()
toolsBlock := func(tools string) string {
return "# stage worker\n\n```output\nResult.\n```\n\n```prompt\nDo the work.\n```\n\n```tools\n" + tools + "\n```\n\n```edges\nsuccess -> #success\nerror -> #failure\n```\n\n```configs\ntype=model\nprovider=opencode-go\nmodel=deepseek-v4.1-flash\nretries=1\ntimeout=30s\n```\n"
}
runStage := func(name, tools string) (status, sessions, diagnostic string) {
t.Helper()
workspace := t.TempDir()
script := strings.ReplaceAll(scriptTemplate, "@STATE@", workspace)
if err := os.WriteFile(executable, []byte(script), 0700); err != nil {
t.Fatal(err)
}
if _, err := store.Submit(ctx, name, toolsBlock(tools)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
var id int64
for _, item := range items {
if item.Name == name {
id = item.ID
}
}
started, err := r.start(protocol.Message{PipelineID: id, Version: 1, RequestKey: name, Workspace: workspace, Input: map[string]any{"prompt": "task"}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
detail, err := store.GetRun(ctx, started.ID)
if err != nil {
t.Fatal(err)
}
if len(detail.Steps) > 0 {
diagnostic = detail.Steps[0].Diagnostic
}
if data, err := os.ReadFile(filepath.Join(workspace, "sessions")); err == nil {
sessions = strings.TrimSpace(string(data))
}
return detail.Status, sessions, diagnostic
}
status, sessions, diagnostic := runStage("readonly", "read")
if status != "succeeded" || sessions != "2" || diagnostic != "" {
t.Fatalf("read-only stage did not retry the transport readback: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic)
}
status, sessions, diagnostic = runStage("edit", "edit")
if status != "failed" || sessions != "1" {
t.Fatalf("edit stage replayed admitted work: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic)
}
if !strings.Contains(diagnostic, "HARNESS TRANSPORT") || !strings.Contains(diagnostic, "malformed or incomplete") {
t.Fatalf("step diagnostic lost the underlying transport cause: %q", diagnostic)
}
}
+5 -2
View File
@@ -55,7 +55,8 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
if c.command == nil {
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
if err != nil {
return "", err
// Connecting to the service failed before any session existed.
return "", harnessUnreachable(err)
}
defer connection.Close()
c.connection = connection
@@ -64,7 +65,9 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
rules := c.permissionRules()
data, err := c.call(ctx, "post", "/api/session", map[string]any{"title": "Pipeline audited agent stage", "agent": "build", "location": map[string]string{"directory": c.Directory}, "model": map[string]string{"providerID": config.Provider, "id": config.Model}, "permissions": rules})
if err != nil {
return "", err
// No prompt or tool call was admitted: a request failure here is a
// reachable retry rather than a reason to end the run.
return "", harnessUnreachable(err)
}
var created struct {
Data struct {
+40 -9
View File
@@ -27,15 +27,42 @@ const (
// diagnostics. It states the retained side effects and the cost of a full retry.
const finalResultGuidance = "completed edits and tool side effects are retained and not undone; a full retry repeats work"
// harnessTransportFailure is the single classifier shared by both transport
// sentinels and the typed OpenCode error: a request, poll or readback failure
// that reports as the fixed HARNESS TRANSPORT category. A bare context deadline
// is classified separately so a genuine stage overrun is never relabeled as a
// transport failure.
func harnessTransportFailure(err error) bool {
var openCode *openCodeError
return errors.Is(err, ErrHarnessUnreachable) || errors.Is(err, ErrHarnessTransport) || errors.As(err, &openCode)
}
// harnessReachFailure reports whether err is a pre-admission harness reach
// failure. It is distinct from a post-admission ErrHarnessTransport failure: no
// session or prompt was admitted, so the retry replays no work.
func harnessReachFailure(err error) bool {
return errors.Is(err, ErrHarnessUnreachable)
}
// admittedWorker is implemented by a client that can admit tool work before its
// final result is read, so the retry loop can refuse to replay it without a
// per-client type switch.
type admittedWorker interface{ workAdmitted() bool }
// workAdmitted reports whether the client admitted any prompt or tool call, so
// the retry loop may retry a transport failure on any stage only when it is
// false, and only a read-only stage otherwise.
func (c *AgentClient) workAdmitted() bool { return c.WorkStarted }
func (c *PiClient) workAdmitted() bool { return c.WorkStarted }
// modelFailureCategory classifies a stage failure using error identity only, so
// the label stays sanitized and bounded. Validation is checked first: a
// structured-output failure is never reported as a transport failure and vice versa.
func modelFailureCategory(err error) string {
var openCode *openCodeError
switch {
case errors.Is(err, ErrInvalidModelResult):
return finalResultValidation
case errors.Is(err, ErrHarnessTransport), errors.As(err, &openCode), errors.Is(err, context.DeadlineExceeded):
case harnessTransportFailure(err), errors.Is(err, context.DeadlineExceeded):
return harnessTransport
default:
return ""
@@ -294,15 +321,19 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
break
}
replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err)
// Never blindly replay admitted tool work: only a read-only stage may
// retry a post-admission deadline or transport failure.
if agent, ok := client.(*AgentClient); ok && agent.WorkStarted && !replayPostAdmission {
// A reach failure observed before any work was admitted replays
// nothing, so it is retryable within the existing budget for any
// stage. After admission, only an explicit read-only stage may
// retry; admitted tool work is never replayed.
workStarted := false
if worker, ok := client.(admittedWorker); ok {
workStarted = worker.workAdmitted()
}
if workStarted && !replayPostAdmission {
break
}
if pi, ok := client.(*PiClient); ok && pi.WorkStarted && !replayPostAdmission {
break
}
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission) {
preAdmissionReach := !workStarted && harnessReachFailure(err)
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission && !preAdmissionReach) {
break
}
}
+17
View File
@@ -18,6 +18,23 @@ var ErrInvalidModelResult = errors.New("invalid model stage result")
// validation and never replay admitted tool work.
var ErrHarnessTransport = errors.New("harness transport failure")
// ErrHarnessUnreachable marks a request/transport failure observed before the
// harness admitted any agent work: connecting to the service, or creating the
// session. Nothing was executed or admitted, so re-attempting replays no prompt
// or tool call and is safe for any stage, mutating or not. Post-admission
// readback, poll and agent-execution failures stay ErrHarnessTransport so the
// single fixed HARNESS TRANSPORT category and the no-tool-replay rule are
// unchanged.
var ErrHarnessUnreachable = errors.New("harness unreachable")
// harnessUnreachable marks err as a pre-admission reach failure. It is the single
// wrap shared by the connect and session-create paths of both client types, so
// they cannot drift: the sentinel and the underlying cause are always both
// preserved.
func harnessUnreachable(err error) error {
return fmt.Errorf("%w: %w", ErrHarnessUnreachable, err)
}
func modelPrompt(def Definition, spec ModelSpec, input Values) (string, error) {
outcomes := make([]string, 0, len(def.Edges))
for name := range def.Edges {
+20 -9
View File
@@ -97,10 +97,14 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
evidenceStatus = c.connection.LastHTTPStatus
}
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: evidenceStatus, NativeOperation: c.connection.LastHTTPOperation, Diagnostic: err.Error(), Stdout: string(data)}, started)
if ctx.Err() != nil {
return nil, ctx.Err()
transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
if ctxErr := ctx.Err(); ctxErr != nil {
// Keep the transport shape (operation/status) with the context
// cause: a deadline that fired while the request was in flight is
// both, and neither may be discarded.
return nil, errors.Join(ctxErr, transport)
}
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
return nil, transport
}
return data, nil
}
@@ -146,14 +150,18 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
code = exit.ExitCode()
}
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: cmd.Path, Directory: directory, ExitCode: code, HTTPStatus: status, Diagnostic: safeCommandDiagnostic(err, stdout.Bytes()), Stdout: stdout.String(), Stderr: stderr.String()}, started)
if ctx.Err() != nil {
return nil, ctx.Err()
}
// Do not embed provider responses, prompts, credentials, or CLI logs in errors.
// Keep the raw cause so errors.Is(err, exec.ErrWaitDelay) still holds while
// Error() stays sanitized. This branch is only reached with a non-nil Run
// error, so valid-looking partial stdout is never returned as success.
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())}
transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())}
if ctxErr := ctx.Err(); ctxErr != nil {
// Keep the sanitized transport detail (operation/status/exit) together
// with errors.Is(err, context.DeadlineExceeded) so the true cause is
// not replaced by a bare context error.
return nil, errors.Join(ctxErr, transport)
}
return nil, transport
}
if stdout.overflow {
return nil, fmt.Errorf("OpenCode response exceeds 1 MiB limit")
@@ -259,7 +267,8 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp
if c.command == nil {
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
if err != nil {
return "", err
// Connecting to the service failed before any session existed.
return "", harnessUnreachable(err)
}
defer connection.Close()
c.connection = connection
@@ -288,7 +297,9 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp
"permissions": []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}},
})
if err != nil {
return "", err
// No prompt or tool call was admitted: a request failure here is a
// reachable retry rather than a reason to end the run.
return "", harnessUnreachable(err)
}
var session struct {
Data struct {
+13 -11
View File
@@ -129,14 +129,15 @@ func repairModelResult(ctx context.Context, client ModelGenerator, config ModelC
return Result{}, fmt.Errorf("%w; no isolated format repair available; recovery status %s", parseError, repairStatusUnrecoverable)
}
// One formatter request, independent of execution/transport retries. The
// deadline is derived from the payload size and the attempt's own remaining
// budget rather than a fixed short constant, so a larger result is not
// guaranteed to time out while the attempt context remains the outer bound.
// deadline is derived from the payload size, the stage timeout and the
// attempt's own remaining budget rather than a fixed short constant, so a
// larger or slower result is not guaranteed to time out while the attempt
// context remains the outer bound.
remaining := time.Hour
if deadline, ok := ctx.Deadline(); ok {
remaining = time.Until(deadline)
}
repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining))
repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining, config.Timeout))
defer cancel()
candidate, err := formatter.RepairFormat(repairCtx, config, original, parseError)
if err := ctx.Err(); err != nil {
@@ -291,9 +292,10 @@ func lexemeOffsets(text string, tokens []string) ([]int, bool) {
}
// Isolated formatter deadline bounds. The allowance grows by one second per KiB
// of payload up to a ceiling, is floored for ordinary payloads, and is clamped
// to the attempt's remaining budget so the stage/attempt context stays the outer
// bound and a large result is not guaranteed to time out.
// of payload up to a ceiling and by a share of the stage timeout, is floored for
// ordinary payloads, and is clamped to the attempt's remaining budget so the
// stage/attempt context stays the outer bound and a large or slow result is not
// guaranteed to time out.
const (
repairDeadlineFloor = 30 * time.Second
repairDeadlineMax = 10 * time.Minute
@@ -301,13 +303,13 @@ const (
)
// repairDeadline derives the formatter deadline. It is O(1) and monotonic
// non-decreasing in payloadBytes; it never exceeds remaining and, when the
// remaining budget allows, never drops below repairDeadlineFloor.
func repairDeadline(payloadBytes int, remaining time.Duration) time.Duration {
// non-decreasing in payloadBytes and stageTimeout; it never exceeds remaining
// and, when the remaining budget allows, never drops below repairDeadlineFloor.
func repairDeadline(payloadBytes int, remaining, stageTimeout time.Duration) time.Duration {
if remaining <= 0 {
return 0
}
allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second
allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second + stageTimeout/4
if allowance > repairDeadlineMax {
allowance = repairDeadlineMax
}
+17 -6
View File
@@ -306,26 +306,37 @@ func TestPresentationStrippingBeforeDeterministicRepair(t *testing.T) {
// TestRepairDeadlineIsMonotonicFlooredAndClamped proves the derived deadline, not
// a fixed constant, bounds the single formatter request.
func TestRepairDeadlineIsMonotonicFlooredAndClamped(t *testing.T) {
if got := repairDeadline(0, time.Hour); got != repairDeadlineFloor {
if got := repairDeadline(0, time.Hour, time.Minute); got < repairDeadlineFloor || got > repairDeadlineMax {
t.Fatalf("floor not applied: %s", got)
}
if got := repairDeadline(1<<30, 24*time.Hour); got != repairDeadlineMax {
if got := repairDeadline(1<<30, 24*time.Hour, time.Hour); got != repairDeadlineMax {
t.Fatalf("ceiling not applied: %s", got)
}
if got := repairDeadline(10, 0); got != 0 {
if got := repairDeadline(10, 0, time.Hour); got != 0 {
t.Fatalf("zero remaining should yield zero: %s", got)
}
if clamped := repairDeadline(1<<20, 5*time.Second); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second {
if clamped := repairDeadline(1<<20, 5*time.Second, time.Hour); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second {
t.Fatalf("remaining budget was not the bound: %s", clamped)
}
previous := time.Duration(0)
for size := 0; size <= 64<<10; size += 1024 {
got := repairDeadline(size, time.Hour)
got := repairDeadline(size, time.Hour, time.Minute)
if got < previous {
t.Fatalf("deadline not monotonic at %d: %s < %s", size, got, previous)
t.Fatalf("deadline not monotonic in payload at %d: %s < %s", size, got, previous)
}
previous = got
}
previous = time.Duration(0)
for stage := time.Duration(0); stage <= 2*time.Hour; stage += time.Minute {
got := repairDeadline(64<<10, time.Hour, stage)
if got < previous {
t.Fatalf("deadline not monotonic in stage timeout at %s: %s < %s", stage, got, previous)
}
previous = got
}
if got := repairDeadline(8<<10, time.Hour, time.Hour); got < repairDeadlineFloor {
t.Fatalf("ordinary payload was not floored: %s", got)
}
}
// TestShellStageRecoverableMalformedResultDoesNotFailRun proves a shell-enabled
+197
View File
@@ -0,0 +1,197 @@
package stage
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/opencode"
)
// TestPreAdmissionReachFailureRetriesOnMutatingStage proves a failure to reach
// the harness before any prompt or tool call is admitted retries within the
// existing budget even on an edit stage, with a fresh session and no replayed
// prompt. Nothing was admitted, so no admitted work can be replayed.
func TestPreAdmissionReachFailureRetriesOnMutatingStage(t *testing.T) {
def := modelDefinition()
def.Blocks["tools"] = "edit"
def.Configs["retries"] = "1"
var sessions, prompts atomic.Int64
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
path := args[2]
switch {
case path == "/api/session":
if sessions.Add(1) == 1 {
return nil, errors.New("connection refused")
}
return []byte(`{"data":{"id":"ses_reach_2"}}`), nil
case strings.HasSuffix(path, "/prompt"):
prompts.Add(1)
return []byte(`{}`), nil
case path == "/api/session/active":
return []byte(`{"data":{}}`), nil
case strings.HasSuffix(path, finalAssistantReadback):
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}`), nil
}
return nil, fmt.Errorf("unexpected path %s", path)
}}}
result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
if err != nil || result.Output["summary"] != "done" {
t.Fatalf("pre-admission reach failure was not retried to success: %v %v", result, err)
}
if sessions.Load() != 2 || prompts.Load() != 1 || agent.SessionID != "ses_reach_2" {
t.Fatalf("reach retry did not use exactly one fresh session: sessions=%d prompts=%d session=%q", sessions.Load(), prompts.Load(), agent.SessionID)
}
}
// TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage proves a readback
// transport failure observed after the prompt was admitted is retried only on an
// explicit read-only stage; an edit, shell or legacy-bundle stage never replays
// the admitted prompt and ends with a HARNESS TRANSPORT failure.
func TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage(t *testing.T) {
run := func(name string, setTools bool, tools string) (Result, error, int64, int64) {
t.Helper()
def := modelDefinition()
if setTools {
def.Blocks["tools"] = tools
}
def.Configs["retries"] = "1"
var sessions, prompts, readbacks atomic.Int64
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
path := args[2]
switch {
case path == "/api/session":
n := sessions.Add(1)
return []byte(fmt.Sprintf(`{"data":{"id":"ses_rb_%d"}}`, n)), nil
case strings.HasSuffix(path, "/prompt"):
prompts.Add(1)
return []byte(`{}`), nil
case path == "/api/session/active":
return []byte(`{"data":{}}`), nil
case strings.HasSuffix(path, finalAssistantReadback):
if readbacks.Add(1) == 1 {
// Truncated readback: a post-admission transport failure.
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":`), nil
}
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"ok\"}}"}]}]}`), nil
case strings.HasSuffix(path, "/interrupt"):
return []byte(`{}`), nil
}
return nil, fmt.Errorf("unexpected path %s", path)
}}}
result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
return result, err, sessions.Load(), prompts.Load()
}
result, err, sessions, prompts := run("read-only", true, "read")
if err != nil || result.Output["summary"] != "ok" || sessions != 2 || prompts != 2 {
t.Fatalf("read-only readback failure was not retried: %v %v sessions=%d prompts=%d", result, err, sessions, prompts)
}
for _, tc := range []struct {
name string
setTools bool
tools string
}{
{"edit", true, "edit"},
{"shell", true, "shell"},
{"legacy", false, ""},
} {
result, err, sessions, prompts := run(tc.name, tc.setTools, tc.tools)
if !errors.Is(err, ErrHarnessTransport) || sessions != 1 || prompts != 1 {
t.Fatalf("%s stage replayed admitted work on a readback failure: err=%v sessions=%d prompts=%d", tc.name, err, sessions, prompts)
}
if result.Output != nil {
t.Fatalf("%s stage retained a partial result", tc.name)
}
if !strings.Contains(err.Error(), "HARNESS TRANSPORT") {
t.Fatalf("%s failure lost the transport category: %v", tc.name, err)
}
}
}
// TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline proves an
// expired context during a real CLI request keeps both the sanitized transport
// detail (operation/exit) and errors.Is(err, context.DeadlineExceeded) instead
// of returning a bare context error that discards the cause.
func TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline(t *testing.T) {
dir := t.TempDir()
executable := filepath.Join(dir, "opencode")
if err := os.WriteFile(executable, []byte("#!/bin/sh\nexec sleep 30\n"), 0700); err != nil {
t.Fatal(err)
}
t.Setenv("PIPELINES_OPENCODE", executable)
client := OpenCodeClient{Directory: dir}
ctx, cancel := context.WithTimeout(context.Background(), 40*time.Millisecond)
defer cancel()
_, err := client.call(ctx, "post", "/api/session", nil)
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("expired transport call lost the context cause: %v", err)
}
var typed *openCodeError
if !errors.As(err, &typed) || typed.operation != "post /api/session" {
t.Fatalf("expired transport call discarded the typed transport error: %v", err)
}
if !strings.Contains(err.Error(), "OpenCode post /api/session") {
t.Fatalf("expired transport call discarded the operation detail: %v", err)
}
if !harnessTransportFailure(err) {
t.Fatalf("expired transport call was not classified as transport: %v", err)
}
}
// TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline proves
// the native V1 branch keeps both the typed transport error (the operation
// detail) and errors.Is(err, context.DeadlineExceeded) when the request failed
// under an expired context, instead of returning a bare context error that
// discards the transport shape. The native request path folds an in-flight
// deadline into ctx.Err(), so the retained transport detail is the operation and
// the parsed HTTP status is 0; the important invariant is that both shapes
// survive the join.
func TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline(t *testing.T) {
client := OpenCodeClient{Backend: "v1", connection: &opencode.Connection{Backend: "v1", URL: "http://127.0.0.1:1"}}
ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Second))
defer cancel()
_, err := client.call(ctx, "post", "/api/session/ses_native/generate", map[string]any{"prompt": "task"})
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("expired native transport call lost the context cause: %v", err)
}
var typed *openCodeError
if !errors.As(err, &typed) || typed.operation != "post /api/session/ses_native/generate" {
t.Fatalf("expired native transport call discarded the typed transport error: %v", err)
}
if !strings.Contains(err.Error(), "post /api/session/ses_native/generate") {
t.Fatalf("expired native transport call discarded the operation detail: %v", err)
}
if !harnessTransportFailure(err) {
t.Fatalf("expired native transport call was not classified as transport: %v", err)
}
}
// TestHarnessTransportSentinelsShareOneClassifierAndCategory proves the two
// transport shapes share one classifier and the single HARNESS TRANSPORT label
// while a bare stage deadline stays distinct from a pre-admission reach failure.
func TestHarnessTransportSentinelsShareOneClassifierAndCategory(t *testing.T) {
unreachable := harnessUnreachable(errors.New("connect refused"))
if !harnessTransportFailure(unreachable) || !harnessReachFailure(unreachable) {
t.Fatal("pre-admission reach failure was not classified")
}
if !strings.Contains(wrapModelStageFailure("worker", unreachable).Error(), "HARNESS TRANSPORT") {
t.Fatalf("pre-admission reach failure lost the transport category: %v", wrapModelStageFailure("worker", unreachable))
}
post := fmt.Errorf("%w: final assistant readback was malformed or incomplete", ErrHarnessTransport)
if !harnessTransportFailure(post) || harnessReachFailure(post) {
t.Fatal("post-admission transport failure was misclassified as a pre-admission reach failure")
}
if !strings.Contains(wrapModelStageFailure("worker", post).Error(), "HARNESS TRANSPORT") {
t.Fatal("post-admission transport failure lost the transport category")
}
if harnessTransportFailure(context.DeadlineExceeded) || harnessReachFailure(context.DeadlineExceeded) {
t.Fatal("a bare stage deadline must not be a transport or reach failure")
}
}