Merge lane-srv3: keep the lane's reviewed work
This commit is contained in:
+33
-12
@@ -82,10 +82,11 @@ Format recovery has a separate bounded budget, even with `retries=0`:
|
|||||||
2. If the local pass cannot repair it, make at most one fresh, deny-all, retained
|
2. If the local pass cannot repair it, make at most one fresh, deny-all, retained
|
||||||
text-generation request with only the malformed response and precise parsing
|
text-generation request with only the malformed response and precise parsing
|
||||||
error. The original implementing/reviewing agent is never re-prompted. The
|
error. The original implementing/reviewing agent is never re-prompted. The
|
||||||
formatter's deadline is derived from the payload size and the attempt's own
|
formatter's deadline is derived from the payload size, the stage timeout and
|
||||||
remaining budget rather than a fixed short constant, so a large result is not
|
the attempt's own remaining budget rather than a fixed short constant, so a
|
||||||
guaranteed to time out; it stays within the stage deadline, is floored for
|
large or slower result is not guaranteed to time out; it stays within the
|
||||||
ordinary payloads, and keeps the 32 KiB input/output bound.
|
stage deadline (clamped to `remaining` minus a margin), is floored for ordinary
|
||||||
|
payloads, and keeps the 32 KiB input/output bound.
|
||||||
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
|
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
|
||||||
with at most 16 insertions. Every existing string, number, literal and
|
with at most 16 insertions. Every existing string, number, literal and
|
||||||
structural delimiter must remain verbatim and in order. Validate the full
|
structural delimiter must remain verbatim and in order. Validate the full
|
||||||
@@ -106,8 +107,14 @@ rerun project edits, commands, or the entire stage to repair final formatting.
|
|||||||
Configured execution retries remain for retryable text-generation transport
|
Configured execution retries remain for retryable text-generation transport
|
||||||
errors; an explicit read-only stage may additionally re-emit a malformed final
|
errors; an explicit read-only stage may additionally re-emit a malformed final
|
||||||
result and retry a post-admission deadline or transport failure up to `retries`.
|
result and retry a post-admission deadline or transport failure up to `retries`.
|
||||||
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
|
A failure to reach the harness before any prompt or tool call is admitted
|
||||||
read/glob/grep/edit/shell bundle), never has admitted tool work replayed. The
|
(connecting, or creating the session) is a distinct reach failure: nothing was
|
||||||
|
admitted, so it is retried within `retries` for any stage, mutating or not, with
|
||||||
|
a fresh session and no replayed prompt or tool call. A post-admission readback,
|
||||||
|
active-session poll or agent-execution failure is retried only on an explicit
|
||||||
|
read-only stage. A stage whose tools include `edit`/`shell`, or that has no tools
|
||||||
|
block (the legacy read/glob/grep/edit/shell bundle), never has admitted tool work
|
||||||
|
replayed. The
|
||||||
deterministic local pass and the fresh deny-all, tool-free formatter session never
|
deterministic local pass and the fresh deny-all, tool-free formatter session never
|
||||||
re-prompt or replay the admitted stage session, so a shell-enabled review stage
|
re-prompt or replay the admitted stage session, so a shell-enabled review stage
|
||||||
gains malformed-result recovery without any replay of admitted work.
|
gains malformed-result recovery without any replay of admitted work.
|
||||||
@@ -121,11 +128,18 @@ and a recognized safe error category (such as session not found or service
|
|||||||
unavailable). Arbitrary stderr, provider messages and response bodies are not
|
unavailable). Arbitrary stderr, provider messages and response bodies are not
|
||||||
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
|
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
|
||||||
is not evidence that the agent prompt failed to start, and does not trigger replay
|
is not evidence that the agent prompt failed to start, and does not trigger replay
|
||||||
of stage work.
|
of stage work. When the context expires while a request is still in flight, the
|
||||||
|
error keeps both the sanitized transport detail (operation/status/exit) and
|
||||||
|
`errors.Is(err, context.DeadlineExceeded)`, so a deadline that fired during a hung
|
||||||
|
transport call is not replaced by a bare context error; a deadline measured at the
|
||||||
|
attempt boundary with no transport in flight stays the stage overrun it is.
|
||||||
Native `StructuredOutputError` is named directly, rather than reduced to a generic
|
Native `StructuredOutputError` is named directly, rather than reduced to a generic
|
||||||
command failure. Native HTTP evidence records the actual HTTP operation/status;
|
command failure, and an unrecognized provider execution-error name is reported as
|
||||||
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
|
a bounded, sanitized identifier plus any 4xx/5xx status instead of a blanket
|
||||||
Cleanup failures preserve the original generation error and active ownership.
|
summary (a name containing arbitrary bytes stays the generic message). Native HTTP
|
||||||
|
evidence records the actual HTTP operation/status; an `exit_code` of -1 there is
|
||||||
|
explicitly marked as not a subprocess exit. Cleanup failures preserve the original
|
||||||
|
generation error and active ownership.
|
||||||
|
|
||||||
For existing native sessions with format-bearing messages, the HTTP readback
|
For existing native sessions with format-bearing messages, the HTTP readback
|
||||||
workaround is read-only CLI `opencode export`. The exported session ID must match
|
workaround is read-only CLI `opencode export`. The exported session ID must match
|
||||||
@@ -144,6 +158,11 @@ diagnostic:
|
|||||||
retry repeats work.
|
retry repeats work.
|
||||||
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
|
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
|
||||||
failures, including the sanitized `malformed or incomplete` readback message.
|
failures, including the sanitized `malformed or incomplete` readback message.
|
||||||
|
A single classifier covers the two transport shapes: a pre-admission reach
|
||||||
|
failure (connecting or creating the session) and a post-admission failure
|
||||||
|
(readback/poll/agent execution). Both share the one `HARNESS TRANSPORT` label;
|
||||||
|
only the pre-admission shape is retried on a mutating stage, because it replays
|
||||||
|
nothing.
|
||||||
|
|
||||||
A structured-output validation failure is never reported as a harness transport
|
A structured-output validation failure is never reported as a harness transport
|
||||||
failure, and vice versa. Neither label includes a provider response body, prompt,
|
failure, and vice versa. Neither label includes a provider response body, prompt,
|
||||||
@@ -166,8 +185,10 @@ isolated formatter hit its own 30-second deadline. The malformed payload was
|
|||||||
retrieved read-only with `pipelines inspect 55 --json` and reproduced the exact
|
retrieved read-only with `pipelines inspect 55 --json` and reproduced the exact
|
||||||
byte offset 7837. It is an unescaped quote/colon inside a string value - an
|
byte offset 7837. It is an unescaped quote/colon inside a string value - an
|
||||||
ambiguous case - so the deterministic pass correctly reports it unrecoverable
|
ambiguous case - so the deterministic pass correctly reports it unrecoverable
|
||||||
rather than guessing, while the new derived deadline removes the fixed-deadline
|
rather than guessing, while the stage-timeout-proportional derived deadline
|
||||||
failure mode for larger payloads. These are source-level findings covered by mocked
|
(floored, capped and clamped to the remaining attempt budget) removes the
|
||||||
|
fixed-30-second-deadline failure mode for larger or slower payloads. These are
|
||||||
|
source-level findings covered by mocked
|
||||||
`internal/stage` fixtures. An updated checkout or installed skill is **not**
|
`internal/stage` fixtures. An updated checkout or installed skill is **not**
|
||||||
evidence that a user's running server contains the fix; deploying/restarting is a
|
evidence that a user's running server contains the fix; deploying/restarting is a
|
||||||
separate authorized action.
|
separate authorized action.
|
||||||
|
|||||||
@@ -428,3 +428,31 @@ blocked-class cause distribution and the transience of transport errors are
|
|||||||
hypotheses, not measurements. Raw captures are under
|
hypotheses, not measurements. Raw captures are under
|
||||||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and
|
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and
|
||||||
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).
|
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).
|
||||||
|
|
||||||
|
## 7. Transport and repair recovery (source-level, 2026-10-17)
|
||||||
|
|
||||||
|
Read-only pass over the §1 diagnostics after the harness-transport and
|
||||||
|
format-repair change. No run was started, retried or cancelled; no server, ntfy,
|
||||||
|
lane or device was restarted or redeployed; no provider inference was performed.
|
||||||
|
Each shape is mapped to the code path that produces it. The per-run
|
||||||
|
pre-admission vs post-admission and read-only vs mutating split was **not**
|
||||||
|
re-inspected with `pipelines inspect`, so every per-run row is labelled *not
|
||||||
|
verified* and the counts are not summed into a "recovered" total.
|
||||||
|
|
||||||
|
| failure shape (§1 class, quoted diagnostic) | run ids | code path | can recover | remains unrecoverable |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| execution error at readback (`OpenCode agent reported an execution error`) | 73,74,75,77,80,149,150,152,195,202 | post-admission `completedAgentResult` / `opencode.ExecutionError` | a stage that is explicit read-only may retry within `retries`(*not verified per run*) | a mutating/legacy stage never replays admitted work; a non-transient provider error stays terminal |
|
||||||
|
| `context deadline exceeded` | 33,36,38,59,101,138 | attempt-boundary deadline, or a transport call that expired | a deadline during an in-flight request now keeps the transport cause and, when pre-admission, retries for any stage; a read-only stage retries a post-admission one | a genuine stage overrun on a mutating/legacy stage stays terminal; whether each run was a hung call or a real overrun is *not verified* |
|
||||||
|
| readback failure (`final assistant readback had no completed result`) | 161 | post-admission `completedAgentResult` | explicit read-only stage may retry | mutating/legacy stage never replays |
|
||||||
|
| native V1 `/wait` command failed (`CLI exit code 1`) | 4 | post-admission `openCodeError` on `/wait` | explicit read-only stage may retry | mutating/legacy stage never replays |
|
||||||
|
| repair timeout (`isolated format repair failed … context deadline exceeded`) | 55,81,174,179,183,221 | `repairDeadline` (payload bytes + stage timeout, floored, capped, clamped to remaining − margin) | a slow-but-repairable payload now gets a longer bounded budget | run 55's payload is the ambiguous unescaped-quote/colon class (byte offset 7837) → unrecoverable; any payload past the 32 KiB bound stays unrecoverable |
|
||||||
|
|
||||||
|
Mapping notes: the execution-error and readback shapes are read back *after* the
|
||||||
|
prompt was admitted, so they are post-admission transport failures and the
|
||||||
|
no-tool-replay rule keeps them terminal on a mutating stage. The
|
||||||
|
`session_transport_command_failed` shape (run 4) is the native V1 `/wait` call
|
||||||
|
issued after the prompt, not a session-create reach failure. The repair-timeout
|
||||||
|
shapes are `FINAL RESULT VALIDATION` (repair-stage), not transport. The new code
|
||||||
|
also reports an unknown provider execution-error name as a bounded sanitized
|
||||||
|
identifier plus any HTTP status, so future occurrences of runs 73–202 are more
|
||||||
|
precisely diagnostic even when they are not retried.
|
||||||
|
|||||||
@@ -20,14 +20,18 @@ Each playbook is *symptom → how to confirm → what to do*.
|
|||||||
the model may have done work but no final result was read.
|
the model may have done work but no final result was read.
|
||||||
- **Confirm** `inspect --json` shows the category on the failed stage. A request
|
- **Confirm** `inspect --json` shows the category on the failed stage. A request
|
||||||
failure, an active-session poll failure and a final-readback failure all carry this
|
failure, an active-session poll failure and a final-readback failure all carry this
|
||||||
one category.
|
one category. A request that failed while the context was still in flight shows
|
||||||
|
both the sanitized transport detail and `context deadline exceeded`; an execution
|
||||||
|
error names the sanitized error name/tag and any HTTP status, not a provider body.
|
||||||
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
|
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
|
||||||
pre-admission failure (nothing executed) from a post-admission one (tools may have
|
pre-admission reach failure (connecting or creating the session; nothing executed)
|
||||||
run). An explicit read-only stage may retry; a stage whose tools include `edit`/
|
from a post-admission one (tools may have run). A pre-admission reach failure
|
||||||
`shell`, or that has no tools block, **never replays admitted work**. Do not re-run
|
replays nothing, so it is retried automatically within `retries` for any stage; a
|
||||||
a full agent stage merely to recover a response's formatting. If it recurs, check
|
post-admission failure is retried only on an explicit read-only stage. A stage
|
||||||
the service and harness health (`references/operations.md`); do not weaken the
|
whose tools include `edit`/`shell`, or that has no tools block, **never replays
|
||||||
stage to get past it.
|
admitted work**. Do not re-run a full agent stage merely to recover a response's
|
||||||
|
formatting. If it recurs, check the service and harness health
|
||||||
|
(`references/operations.md`); do not weaken the stage to get past it.
|
||||||
|
|
||||||
## Malformed model results
|
## Malformed model results
|
||||||
|
|
||||||
@@ -39,8 +43,9 @@ Each playbook is *symptom → how to confirm → what to do*.
|
|||||||
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
|
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
|
||||||
appends missing closers, drops one trailing comma, or inserts the single
|
appends missing closers, drops one trailing comma, or inserts the single
|
||||||
comma/colon the parser expected; only if that is ambiguous does one deny-all
|
comma/colon the parser expected; only if that is ambiguous does one deny-all
|
||||||
formatter request run, with a deadline derived from the payload size and the
|
formatter request run, with a deadline derived from the payload size, the stage
|
||||||
attempt's remaining budget. Each path records a `repaired`, `partial` or
|
timeout and the attempt's remaining budget, clamped so it never extends the
|
||||||
|
stage's own deadline. Each path records a `repaired`, `partial` or
|
||||||
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
|
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
|
||||||
is never accepted as complete. On an explicit read-only stage a malformed result
|
is never accepted as complete. On an explicit read-only stage a malformed result
|
||||||
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
|
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
|
||||||
|
|||||||
@@ -350,7 +350,11 @@ admitted tool-using stage: completed edits and tool side effects are retained an
|
|||||||
not undone, so a full retry repeats work. An explicit read-only stage is the one
|
not undone, so a full retry repeats work. An explicit read-only stage is the one
|
||||||
exception: a malformed result there mutates nothing, so it may be re-emitted up
|
exception: a malformed result there mutates nothing, so it may be re-emitted up
|
||||||
to `retries` with its `format_repairs` evidence retained, and a post-admission
|
to `retries` with its `format_repairs` evidence retained, and a post-admission
|
||||||
deadline or transport failure may be retried. A stage whose tools include
|
deadline or transport failure may be retried. A failure to reach the harness
|
||||||
|
before any prompt or tool call is admitted (connecting, or creating the session)
|
||||||
|
replays nothing, so it is retried within `retries` for any stage, mutating or not.
|
||||||
|
A stage whose tools include
|
||||||
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
|
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
|
||||||
bundle), is never replayed. Neither category exposes provider/model bodies or
|
bundle), is never replayed. Neither category exposes provider/model bodies or
|
||||||
credentials.
|
credentials. The isolated formatter's deadline is proportional to the stage
|
||||||
|
timeout and clamped so it never extends the stage's own deadline.
|
||||||
|
|||||||
@@ -80,7 +80,10 @@ specific authorization.
|
|||||||
Retained model-stage failures now carry a fixed sanitized category. A strictly
|
Retained model-stage failures now carry a fixed sanitized category. A strictly
|
||||||
invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a
|
invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a
|
||||||
byte offset) is labeled `FINAL RESULT VALIDATION`; request, V2 active-session poll
|
byte offset) is labeled `FINAL RESULT VALIDATION`; request, V2 active-session poll
|
||||||
and final-readback failures are labeled `HARNESS TRANSPORT`. A validation failure
|
and final-readback failures are labeled `HARNESS TRANSPORT`. A pre-admission
|
||||||
|
request failure replays nothing and is retried for any stage; a post-admission
|
||||||
|
failure is retried only on an explicit read-only stage (see the 2026-10-17 section
|
||||||
|
below). A validation failure
|
||||||
preserves the admitted session, transcript and evidence and performs no second tool
|
preserves the admitted session, transcript and evidence and performs no second tool
|
||||||
execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history
|
execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history
|
||||||
readback of exactly 262144 bytes of incomplete JSON (`invalid agent messages`), a
|
readback of exactly 262144 bytes of incomplete JSON (`invalid agent messages`), a
|
||||||
@@ -100,8 +103,9 @@ Source-level, mocked `internal/stage` regression coverage (not live-provider
|
|||||||
evidence and not a deployment): malformed final-result recovery now tries a
|
evidence and not a deployment): malformed final-result recovery now tries a
|
||||||
deterministic local pass first (append missing closers, drop one trailing comma,
|
deterministic local pass first (append missing closers, drop one trailing comma,
|
||||||
or insert the single comma/colon the parser expected), then at most one deny-all
|
or insert the single comma/colon the parser expected), then at most one deny-all
|
||||||
formatter request whose deadline is derived from the payload size and the
|
formatter request whose deadline is derived from the payload size, the stage
|
||||||
attempt's remaining budget rather than a fixed 30-second constant. Every path
|
timeout and the attempt's remaining budget rather than a fixed 30-second
|
||||||
|
constant. Every path
|
||||||
records `repaired`, `partial` or `unrecoverable` in `format_repairs`; a `partial`
|
records `repaired`, `partial` or `unrecoverable` in `format_repairs`; a `partial`
|
||||||
or `unrecoverable` payload is never returned as a completed result, and an
|
or `unrecoverable` payload is never returned as a completed result, and an
|
||||||
ambiguous payload (an unescaped quote/colon inside a string value) is never
|
ambiguous payload (an unescaped quote/colon inside a string value) is never
|
||||||
@@ -112,6 +116,26 @@ unchanged. The run-55 `ui_reviewer` payload was retrieved read-only with
|
|||||||
ambiguous error class, so it is reported unrecoverable rather than repaired. This
|
ambiguous error class, so it is reported unrecoverable rather than repaired. This
|
||||||
does not update or restart any running server.
|
does not update or restart any running server.
|
||||||
|
|
||||||
|
## Harness transport reach retry and repair deadline: 2026-10-17
|
||||||
|
|
||||||
|
Source-level, mocked `internal/stage`, `internal/opencode` and `internal/server`
|
||||||
|
regression coverage (not live-provider evidence and not a deployment): the
|
||||||
|
harness transport path now distinguishes a pre-admission reach failure
|
||||||
|
(connecting or creating the session) from a post-admission readback/poll/agent
|
||||||
|
failure under the one `HARNESS TRANSPORT` category. A pre-admission reach failure
|
||||||
|
replays nothing, so it is retried within `retries` for any stage with a fresh
|
||||||
|
session and no replayed prompt; a post-admission failure is retried only on an
|
||||||
|
explicit read-only stage, and an `edit`/`shell`/legacy stage still never replays
|
||||||
|
admitted work. A context that expires during a request keeps both the sanitized
|
||||||
|
transport detail (operation/status/exit) and
|
||||||
|
`errors.Is(err, context.DeadlineExceeded)`; a bare attempt-boundary deadline is
|
||||||
|
not treated as a reach failure and stays terminal for a mutating stage. An
|
||||||
|
unrecognized provider execution-error name is reported as a bounded, sanitized
|
||||||
|
identifier plus any HTTP status. The isolated formatter deadline is now
|
||||||
|
proportional to the stage timeout (floored, capped, and clamped to
|
||||||
|
`remaining - margin` so it can never extend the stage/attempt deadline). These are
|
||||||
|
mocked fixtures; a deployed server is not updated by this change.
|
||||||
|
|
||||||
## Failure-handling source changes: 2026-10-15
|
## Failure-handling source changes: 2026-10-15
|
||||||
|
|
||||||
Source-level, mocked `internal/stage` and `internal/server` regression coverage
|
Source-level, mocked `internal/stage` and `internal/server` regression coverage
|
||||||
|
|||||||
@@ -81,6 +81,15 @@ func ExecutionError(data json.RawMessage) error {
|
|||||||
}
|
}
|
||||||
safeNames := map[string]bool{"APIError": true, "ProviderAuthError": true, "ContextOverflowError": true, "MessageAbortedError": true, "OutputLengthError": true, "UnknownError": true, "ServiceUnavailableError": true}
|
safeNames := map[string]bool{"APIError": true, "ProviderAuthError": true, "ContextOverflowError": true, "MessageAbortedError": true, "OutputLengthError": true, "UnknownError": true, "ServiceUnavailableError": true}
|
||||||
if !safeNames[name] {
|
if !safeNames[name] {
|
||||||
|
// An unrecognized provider error name is still reported, but only as a
|
||||||
|
// bounded, safe identifier plus any HTTP status. No provider body,
|
||||||
|
// message, prompt or credential is copied.
|
||||||
|
if safe := sanitizedExecutionName(name); safe != "" {
|
||||||
|
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
|
||||||
|
return fmt.Errorf("OpenCode agent reported an execution error: %s (HTTP %d)", safe, failure.Data.StatusCode)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("OpenCode agent reported an execution error: %s", safe)
|
||||||
|
}
|
||||||
return fmt.Errorf("OpenCode agent reported an execution error")
|
return fmt.Errorf("OpenCode agent reported an execution error")
|
||||||
}
|
}
|
||||||
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
|
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
|
||||||
@@ -88,3 +97,23 @@ func ExecutionError(data json.RawMessage) error {
|
|||||||
}
|
}
|
||||||
return fmt.Errorf("OpenCode agent execution error: %s", name)
|
return fmt.Errorf("OpenCode agent execution error: %s", name)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sanitizedExecutionName reduces an unknown provider error name/tag to a
|
||||||
|
// bounded, safe identifier so the diagnostic can name the failure shape without
|
||||||
|
// copying an arbitrary message, prompt or credential. A name containing any
|
||||||
|
// other byte returns empty and is reported as a generic execution error.
|
||||||
|
func sanitizedExecutionName(name string) string {
|
||||||
|
if len(name) > 64 {
|
||||||
|
name = name[:64]
|
||||||
|
}
|
||||||
|
var b strings.Builder
|
||||||
|
for _, r := range name {
|
||||||
|
switch {
|
||||||
|
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_', r == '.', r == '-':
|
||||||
|
b.WriteRune(r)
|
||||||
|
default:
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
|||||||
@@ -90,3 +90,26 @@ func TestNativeStructuredFailureIsClearAndDoesNotReplayGeneration(t *testing.T)
|
|||||||
t.Fatal("wait failure discarded provider response", err, string(retained))
|
t.Fatal("wait failure discarded provider response", err, string(retained))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody proves an
|
||||||
|
// unrecognized provider error name is reported as a bounded, safe identifier
|
||||||
|
// plus any HTTP status instead of a blanket summary, while no provider message,
|
||||||
|
// prompt or credential is copied and known names keep their exact wording.
|
||||||
|
func TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody(t *testing.T) {
|
||||||
|
err := ExecutionError(json.RawMessage(`{"name":"TotallyUnknownError","data":{"statusCode":502,"message":"secret provider body"}}`))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "TotallyUnknownError") || !strings.Contains(err.Error(), "HTTP 502") {
|
||||||
|
t.Fatalf("unknown error name/status not reported: %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "secret provider body") {
|
||||||
|
t.Fatalf("diagnostic leaked the provider body: %v", err)
|
||||||
|
}
|
||||||
|
if err := ExecutionError(json.RawMessage(`{"_tag":"SomeTagError"}`)); err == nil || !strings.Contains(err.Error(), "SomeTagError") {
|
||||||
|
t.Fatalf("unknown tag not reported: %v", err)
|
||||||
|
}
|
||||||
|
if err := ExecutionError(json.RawMessage(`{"name":"Injected Error: Bearer secret-token"}`)); err == nil || strings.Contains(err.Error(), "secret-token") || strings.Contains(err.Error(), "Bearer") || err.Error() != "OpenCode agent reported an execution error" {
|
||||||
|
t.Fatalf("unsafe name was copied into the diagnostic: %v", err)
|
||||||
|
}
|
||||||
|
if err := ExecutionError(json.RawMessage(`{"name":"APIError","data":{"statusCode":500}}`)); err == nil || !strings.Contains(err.Error(), "APIError (HTTP 500)") {
|
||||||
|
t.Fatalf("known safe name changed: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -247,3 +247,97 @@ func TestRetryRejectsUnavailableOriginalDirectoryAndNonfailedRuns(t *testing.T)
|
|||||||
t.Fatal("nonexistent run accepted")
|
t.Fatal("nonexistent run accepted")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot proves,
|
||||||
|
// through the real runner and CLI fixture, that a read-only stage retries a
|
||||||
|
// post-admission transport (final-readback) failure with a fresh session while
|
||||||
|
// an edit stage never replays the admitted prompt, and that the failed step's
|
||||||
|
// diagnostic retains the underlying transport cause.
|
||||||
|
func TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
dir := t.TempDir()
|
||||||
|
executable := filepath.Join(dir, "opencode")
|
||||||
|
t.Setenv("PIPELINES_OPENCODE", executable)
|
||||||
|
scriptTemplate := `#!/bin/sh
|
||||||
|
if [ "$1" = "--help" ]; then echo ' api'; exit 0; fi
|
||||||
|
printf '%s %s\n' "$2" "$3" >> "@STATE@/calls"
|
||||||
|
case "$3" in
|
||||||
|
/api/session)
|
||||||
|
n=0; if [ -f "@STATE@/sessions" ]; then n=$(cat "@STATE@/sessions"); fi
|
||||||
|
n=$((n+1)); echo "$n" > "@STATE@/sessions"
|
||||||
|
printf '{"data":{"id":"ses_bounded_%s"}}' "$n" ;;
|
||||||
|
*/prompt) echo '{}' ;;
|
||||||
|
/api/session/active) echo '{"data":{}}' ;;
|
||||||
|
*/message*)
|
||||||
|
r=0; if [ -f "@STATE@/readbacks" ]; then r=$(cat "@STATE@/readbacks"); fi
|
||||||
|
r=$((r+1)); echo "$r" > "@STATE@/readbacks"
|
||||||
|
if [ "$r" = "1" ]; then
|
||||||
|
echo '{"data":[{"finish":"stop","content":'
|
||||||
|
else
|
||||||
|
echo '{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}'
|
||||||
|
fi ;;
|
||||||
|
*/interrupt) echo '{}' ;;
|
||||||
|
*) exit 99 ;;
|
||||||
|
esac
|
||||||
|
`
|
||||||
|
store, err := pipeline.Open(filepath.Join(dir, "runs.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer store.Close()
|
||||||
|
r, err := newRunner(ctx, store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() { cancel(); r.wg.Wait() }()
|
||||||
|
toolsBlock := func(tools string) string {
|
||||||
|
return "# stage worker\n\n```output\nResult.\n```\n\n```prompt\nDo the work.\n```\n\n```tools\n" + tools + "\n```\n\n```edges\nsuccess -> #success\nerror -> #failure\n```\n\n```configs\ntype=model\nprovider=opencode-go\nmodel=deepseek-v4.1-flash\nretries=1\ntimeout=30s\n```\n"
|
||||||
|
}
|
||||||
|
runStage := func(name, tools string) (status, sessions, diagnostic string) {
|
||||||
|
t.Helper()
|
||||||
|
workspace := t.TempDir()
|
||||||
|
script := strings.ReplaceAll(scriptTemplate, "@STATE@", workspace)
|
||||||
|
if err := os.WriteFile(executable, []byte(script), 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := store.Submit(ctx, name, toolsBlock(tools)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
items, _ := store.List(ctx)
|
||||||
|
var id int64
|
||||||
|
for _, item := range items {
|
||||||
|
if item.Name == name {
|
||||||
|
id = item.ID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
started, err := r.start(protocol.Message{PipelineID: id, Version: 1, RequestKey: name, Workspace: workspace, Input: map[string]any{"prompt": "task"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r.wg.Wait()
|
||||||
|
detail, err := store.GetRun(ctx, started.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(detail.Steps) > 0 {
|
||||||
|
diagnostic = detail.Steps[0].Diagnostic
|
||||||
|
}
|
||||||
|
if data, err := os.ReadFile(filepath.Join(workspace, "sessions")); err == nil {
|
||||||
|
sessions = strings.TrimSpace(string(data))
|
||||||
|
}
|
||||||
|
return detail.Status, sessions, diagnostic
|
||||||
|
}
|
||||||
|
|
||||||
|
status, sessions, diagnostic := runStage("readonly", "read")
|
||||||
|
if status != "succeeded" || sessions != "2" || diagnostic != "" {
|
||||||
|
t.Fatalf("read-only stage did not retry the transport readback: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic)
|
||||||
|
}
|
||||||
|
status, sessions, diagnostic = runStage("edit", "edit")
|
||||||
|
if status != "failed" || sessions != "1" {
|
||||||
|
t.Fatalf("edit stage replayed admitted work: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic)
|
||||||
|
}
|
||||||
|
if !strings.Contains(diagnostic, "HARNESS TRANSPORT") || !strings.Contains(diagnostic, "malformed or incomplete") {
|
||||||
|
t.Fatalf("step diagnostic lost the underlying transport cause: %q", diagnostic)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -55,7 +55,8 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
|
|||||||
if c.command == nil {
|
if c.command == nil {
|
||||||
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
|
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
// Connecting to the service failed before any session existed.
|
||||||
|
return "", harnessUnreachable(err)
|
||||||
}
|
}
|
||||||
defer connection.Close()
|
defer connection.Close()
|
||||||
c.connection = connection
|
c.connection = connection
|
||||||
@@ -64,7 +65,9 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
|
|||||||
rules := c.permissionRules()
|
rules := c.permissionRules()
|
||||||
data, err := c.call(ctx, "post", "/api/session", map[string]any{"title": "Pipeline audited agent stage", "agent": "build", "location": map[string]string{"directory": c.Directory}, "model": map[string]string{"providerID": config.Provider, "id": config.Model}, "permissions": rules})
|
data, err := c.call(ctx, "post", "/api/session", map[string]any{"title": "Pipeline audited agent stage", "agent": "build", "location": map[string]string{"directory": c.Directory}, "model": map[string]string{"providerID": config.Provider, "id": config.Model}, "permissions": rules})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
// No prompt or tool call was admitted: a request failure here is a
|
||||||
|
// reachable retry rather than a reason to end the run.
|
||||||
|
return "", harnessUnreachable(err)
|
||||||
}
|
}
|
||||||
var created struct {
|
var created struct {
|
||||||
Data struct {
|
Data struct {
|
||||||
|
|||||||
+40
-9
@@ -27,15 +27,42 @@ const (
|
|||||||
// diagnostics. It states the retained side effects and the cost of a full retry.
|
// diagnostics. It states the retained side effects and the cost of a full retry.
|
||||||
const finalResultGuidance = "completed edits and tool side effects are retained and not undone; a full retry repeats work"
|
const finalResultGuidance = "completed edits and tool side effects are retained and not undone; a full retry repeats work"
|
||||||
|
|
||||||
|
// harnessTransportFailure is the single classifier shared by both transport
|
||||||
|
// sentinels and the typed OpenCode error: a request, poll or readback failure
|
||||||
|
// that reports as the fixed HARNESS TRANSPORT category. A bare context deadline
|
||||||
|
// is classified separately so a genuine stage overrun is never relabeled as a
|
||||||
|
// transport failure.
|
||||||
|
func harnessTransportFailure(err error) bool {
|
||||||
|
var openCode *openCodeError
|
||||||
|
return errors.Is(err, ErrHarnessUnreachable) || errors.Is(err, ErrHarnessTransport) || errors.As(err, &openCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// harnessReachFailure reports whether err is a pre-admission harness reach
|
||||||
|
// failure. It is distinct from a post-admission ErrHarnessTransport failure: no
|
||||||
|
// session or prompt was admitted, so the retry replays no work.
|
||||||
|
func harnessReachFailure(err error) bool {
|
||||||
|
return errors.Is(err, ErrHarnessUnreachable)
|
||||||
|
}
|
||||||
|
|
||||||
|
// admittedWorker is implemented by a client that can admit tool work before its
|
||||||
|
// final result is read, so the retry loop can refuse to replay it without a
|
||||||
|
// per-client type switch.
|
||||||
|
type admittedWorker interface{ workAdmitted() bool }
|
||||||
|
|
||||||
|
// workAdmitted reports whether the client admitted any prompt or tool call, so
|
||||||
|
// the retry loop may retry a transport failure on any stage only when it is
|
||||||
|
// false, and only a read-only stage otherwise.
|
||||||
|
func (c *AgentClient) workAdmitted() bool { return c.WorkStarted }
|
||||||
|
func (c *PiClient) workAdmitted() bool { return c.WorkStarted }
|
||||||
|
|
||||||
// modelFailureCategory classifies a stage failure using error identity only, so
|
// modelFailureCategory classifies a stage failure using error identity only, so
|
||||||
// the label stays sanitized and bounded. Validation is checked first: a
|
// the label stays sanitized and bounded. Validation is checked first: a
|
||||||
// structured-output failure is never reported as a transport failure and vice versa.
|
// structured-output failure is never reported as a transport failure and vice versa.
|
||||||
func modelFailureCategory(err error) string {
|
func modelFailureCategory(err error) string {
|
||||||
var openCode *openCodeError
|
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, ErrInvalidModelResult):
|
case errors.Is(err, ErrInvalidModelResult):
|
||||||
return finalResultValidation
|
return finalResultValidation
|
||||||
case errors.Is(err, ErrHarnessTransport), errors.As(err, &openCode), errors.Is(err, context.DeadlineExceeded):
|
case harnessTransportFailure(err), errors.Is(err, context.DeadlineExceeded):
|
||||||
return harnessTransport
|
return harnessTransport
|
||||||
default:
|
default:
|
||||||
return ""
|
return ""
|
||||||
@@ -294,15 +321,19 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err)
|
replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err)
|
||||||
// Never blindly replay admitted tool work: only a read-only stage may
|
// A reach failure observed before any work was admitted replays
|
||||||
// retry a post-admission deadline or transport failure.
|
// nothing, so it is retryable within the existing budget for any
|
||||||
if agent, ok := client.(*AgentClient); ok && agent.WorkStarted && !replayPostAdmission {
|
// stage. After admission, only an explicit read-only stage may
|
||||||
|
// retry; admitted tool work is never replayed.
|
||||||
|
workStarted := false
|
||||||
|
if worker, ok := client.(admittedWorker); ok {
|
||||||
|
workStarted = worker.workAdmitted()
|
||||||
|
}
|
||||||
|
if workStarted && !replayPostAdmission {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
if pi, ok := client.(*PiClient); ok && pi.WorkStarted && !replayPostAdmission {
|
preAdmissionReach := !workStarted && harnessReachFailure(err)
|
||||||
break
|
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission && !preAdmissionReach) {
|
||||||
}
|
|
||||||
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission) {
|
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,6 +18,23 @@ var ErrInvalidModelResult = errors.New("invalid model stage result")
|
|||||||
// validation and never replay admitted tool work.
|
// validation and never replay admitted tool work.
|
||||||
var ErrHarnessTransport = errors.New("harness transport failure")
|
var ErrHarnessTransport = errors.New("harness transport failure")
|
||||||
|
|
||||||
|
// ErrHarnessUnreachable marks a request/transport failure observed before the
|
||||||
|
// harness admitted any agent work: connecting to the service, or creating the
|
||||||
|
// session. Nothing was executed or admitted, so re-attempting replays no prompt
|
||||||
|
// or tool call and is safe for any stage, mutating or not. Post-admission
|
||||||
|
// readback, poll and agent-execution failures stay ErrHarnessTransport so the
|
||||||
|
// single fixed HARNESS TRANSPORT category and the no-tool-replay rule are
|
||||||
|
// unchanged.
|
||||||
|
var ErrHarnessUnreachable = errors.New("harness unreachable")
|
||||||
|
|
||||||
|
// harnessUnreachable marks err as a pre-admission reach failure. It is the single
|
||||||
|
// wrap shared by the connect and session-create paths of both client types, so
|
||||||
|
// they cannot drift: the sentinel and the underlying cause are always both
|
||||||
|
// preserved.
|
||||||
|
func harnessUnreachable(err error) error {
|
||||||
|
return fmt.Errorf("%w: %w", ErrHarnessUnreachable, err)
|
||||||
|
}
|
||||||
|
|
||||||
func modelPrompt(def Definition, spec ModelSpec, input Values) (string, error) {
|
func modelPrompt(def Definition, spec ModelSpec, input Values) (string, error) {
|
||||||
outcomes := make([]string, 0, len(def.Edges))
|
outcomes := make([]string, 0, len(def.Edges))
|
||||||
for name := range def.Edges {
|
for name := range def.Edges {
|
||||||
|
|||||||
@@ -97,10 +97,14 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
|
|||||||
evidenceStatus = c.connection.LastHTTPStatus
|
evidenceStatus = c.connection.LastHTTPStatus
|
||||||
}
|
}
|
||||||
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: evidenceStatus, NativeOperation: c.connection.LastHTTPOperation, Diagnostic: err.Error(), Stdout: string(data)}, started)
|
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: evidenceStatus, NativeOperation: c.connection.LastHTTPOperation, Diagnostic: err.Error(), Stdout: string(data)}, started)
|
||||||
if ctx.Err() != nil {
|
transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
|
||||||
return nil, ctx.Err()
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
|
// Keep the transport shape (operation/status) with the context
|
||||||
|
// cause: a deadline that fired while the request was in flight is
|
||||||
|
// both, and neither may be discarded.
|
||||||
|
return nil, errors.Join(ctxErr, transport)
|
||||||
}
|
}
|
||||||
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
|
return nil, transport
|
||||||
}
|
}
|
||||||
return data, nil
|
return data, nil
|
||||||
}
|
}
|
||||||
@@ -146,14 +150,18 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
|
|||||||
code = exit.ExitCode()
|
code = exit.ExitCode()
|
||||||
}
|
}
|
||||||
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: cmd.Path, Directory: directory, ExitCode: code, HTTPStatus: status, Diagnostic: safeCommandDiagnostic(err, stdout.Bytes()), Stdout: stdout.String(), Stderr: stderr.String()}, started)
|
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: cmd.Path, Directory: directory, ExitCode: code, HTTPStatus: status, Diagnostic: safeCommandDiagnostic(err, stdout.Bytes()), Stdout: stdout.String(), Stderr: stderr.String()}, started)
|
||||||
if ctx.Err() != nil {
|
|
||||||
return nil, ctx.Err()
|
|
||||||
}
|
|
||||||
// Do not embed provider responses, prompts, credentials, or CLI logs in errors.
|
// Do not embed provider responses, prompts, credentials, or CLI logs in errors.
|
||||||
// Keep the raw cause so errors.Is(err, exec.ErrWaitDelay) still holds while
|
// Keep the raw cause so errors.Is(err, exec.ErrWaitDelay) still holds while
|
||||||
// Error() stays sanitized. This branch is only reached with a non-nil Run
|
// Error() stays sanitized. This branch is only reached with a non-nil Run
|
||||||
// error, so valid-looking partial stdout is never returned as success.
|
// error, so valid-looking partial stdout is never returned as success.
|
||||||
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())}
|
transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())}
|
||||||
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
|
// Keep the sanitized transport detail (operation/status/exit) together
|
||||||
|
// with errors.Is(err, context.DeadlineExceeded) so the true cause is
|
||||||
|
// not replaced by a bare context error.
|
||||||
|
return nil, errors.Join(ctxErr, transport)
|
||||||
|
}
|
||||||
|
return nil, transport
|
||||||
}
|
}
|
||||||
if stdout.overflow {
|
if stdout.overflow {
|
||||||
return nil, fmt.Errorf("OpenCode response exceeds 1 MiB limit")
|
return nil, fmt.Errorf("OpenCode response exceeds 1 MiB limit")
|
||||||
@@ -259,7 +267,8 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp
|
|||||||
if c.command == nil {
|
if c.command == nil {
|
||||||
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
|
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
// Connecting to the service failed before any session existed.
|
||||||
|
return "", harnessUnreachable(err)
|
||||||
}
|
}
|
||||||
defer connection.Close()
|
defer connection.Close()
|
||||||
c.connection = connection
|
c.connection = connection
|
||||||
@@ -288,7 +297,9 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp
|
|||||||
"permissions": []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}},
|
"permissions": []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
// No prompt or tool call was admitted: a request failure here is a
|
||||||
|
// reachable retry rather than a reason to end the run.
|
||||||
|
return "", harnessUnreachable(err)
|
||||||
}
|
}
|
||||||
var session struct {
|
var session struct {
|
||||||
Data struct {
|
Data struct {
|
||||||
|
|||||||
@@ -129,14 +129,15 @@ func repairModelResult(ctx context.Context, client ModelGenerator, config ModelC
|
|||||||
return Result{}, fmt.Errorf("%w; no isolated format repair available; recovery status %s", parseError, repairStatusUnrecoverable)
|
return Result{}, fmt.Errorf("%w; no isolated format repair available; recovery status %s", parseError, repairStatusUnrecoverable)
|
||||||
}
|
}
|
||||||
// One formatter request, independent of execution/transport retries. The
|
// One formatter request, independent of execution/transport retries. The
|
||||||
// deadline is derived from the payload size and the attempt's own remaining
|
// deadline is derived from the payload size, the stage timeout and the
|
||||||
// budget rather than a fixed short constant, so a larger result is not
|
// attempt's own remaining budget rather than a fixed short constant, so a
|
||||||
// guaranteed to time out while the attempt context remains the outer bound.
|
// larger or slower result is not guaranteed to time out while the attempt
|
||||||
|
// context remains the outer bound.
|
||||||
remaining := time.Hour
|
remaining := time.Hour
|
||||||
if deadline, ok := ctx.Deadline(); ok {
|
if deadline, ok := ctx.Deadline(); ok {
|
||||||
remaining = time.Until(deadline)
|
remaining = time.Until(deadline)
|
||||||
}
|
}
|
||||||
repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining))
|
repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining, config.Timeout))
|
||||||
defer cancel()
|
defer cancel()
|
||||||
candidate, err := formatter.RepairFormat(repairCtx, config, original, parseError)
|
candidate, err := formatter.RepairFormat(repairCtx, config, original, parseError)
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
@@ -291,9 +292,10 @@ func lexemeOffsets(text string, tokens []string) ([]int, bool) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Isolated formatter deadline bounds. The allowance grows by one second per KiB
|
// Isolated formatter deadline bounds. The allowance grows by one second per KiB
|
||||||
// of payload up to a ceiling, is floored for ordinary payloads, and is clamped
|
// of payload up to a ceiling and by a share of the stage timeout, is floored for
|
||||||
// to the attempt's remaining budget so the stage/attempt context stays the outer
|
// ordinary payloads, and is clamped to the attempt's remaining budget so the
|
||||||
// bound and a large result is not guaranteed to time out.
|
// stage/attempt context stays the outer bound and a large or slow result is not
|
||||||
|
// guaranteed to time out.
|
||||||
const (
|
const (
|
||||||
repairDeadlineFloor = 30 * time.Second
|
repairDeadlineFloor = 30 * time.Second
|
||||||
repairDeadlineMax = 10 * time.Minute
|
repairDeadlineMax = 10 * time.Minute
|
||||||
@@ -301,13 +303,13 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// repairDeadline derives the formatter deadline. It is O(1) and monotonic
|
// repairDeadline derives the formatter deadline. It is O(1) and monotonic
|
||||||
// non-decreasing in payloadBytes; it never exceeds remaining and, when the
|
// non-decreasing in payloadBytes and stageTimeout; it never exceeds remaining
|
||||||
// remaining budget allows, never drops below repairDeadlineFloor.
|
// and, when the remaining budget allows, never drops below repairDeadlineFloor.
|
||||||
func repairDeadline(payloadBytes int, remaining time.Duration) time.Duration {
|
func repairDeadline(payloadBytes int, remaining, stageTimeout time.Duration) time.Duration {
|
||||||
if remaining <= 0 {
|
if remaining <= 0 {
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second
|
allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second + stageTimeout/4
|
||||||
if allowance > repairDeadlineMax {
|
if allowance > repairDeadlineMax {
|
||||||
allowance = repairDeadlineMax
|
allowance = repairDeadlineMax
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -306,26 +306,37 @@ func TestPresentationStrippingBeforeDeterministicRepair(t *testing.T) {
|
|||||||
// TestRepairDeadlineIsMonotonicFlooredAndClamped proves the derived deadline, not
|
// TestRepairDeadlineIsMonotonicFlooredAndClamped proves the derived deadline, not
|
||||||
// a fixed constant, bounds the single formatter request.
|
// a fixed constant, bounds the single formatter request.
|
||||||
func TestRepairDeadlineIsMonotonicFlooredAndClamped(t *testing.T) {
|
func TestRepairDeadlineIsMonotonicFlooredAndClamped(t *testing.T) {
|
||||||
if got := repairDeadline(0, time.Hour); got != repairDeadlineFloor {
|
if got := repairDeadline(0, time.Hour, time.Minute); got < repairDeadlineFloor || got > repairDeadlineMax {
|
||||||
t.Fatalf("floor not applied: %s", got)
|
t.Fatalf("floor not applied: %s", got)
|
||||||
}
|
}
|
||||||
if got := repairDeadline(1<<30, 24*time.Hour); got != repairDeadlineMax {
|
if got := repairDeadline(1<<30, 24*time.Hour, time.Hour); got != repairDeadlineMax {
|
||||||
t.Fatalf("ceiling not applied: %s", got)
|
t.Fatalf("ceiling not applied: %s", got)
|
||||||
}
|
}
|
||||||
if got := repairDeadline(10, 0); got != 0 {
|
if got := repairDeadline(10, 0, time.Hour); got != 0 {
|
||||||
t.Fatalf("zero remaining should yield zero: %s", got)
|
t.Fatalf("zero remaining should yield zero: %s", got)
|
||||||
}
|
}
|
||||||
if clamped := repairDeadline(1<<20, 5*time.Second); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second {
|
if clamped := repairDeadline(1<<20, 5*time.Second, time.Hour); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second {
|
||||||
t.Fatalf("remaining budget was not the bound: %s", clamped)
|
t.Fatalf("remaining budget was not the bound: %s", clamped)
|
||||||
}
|
}
|
||||||
previous := time.Duration(0)
|
previous := time.Duration(0)
|
||||||
for size := 0; size <= 64<<10; size += 1024 {
|
for size := 0; size <= 64<<10; size += 1024 {
|
||||||
got := repairDeadline(size, time.Hour)
|
got := repairDeadline(size, time.Hour, time.Minute)
|
||||||
if got < previous {
|
if got < previous {
|
||||||
t.Fatalf("deadline not monotonic at %d: %s < %s", size, got, previous)
|
t.Fatalf("deadline not monotonic in payload at %d: %s < %s", size, got, previous)
|
||||||
}
|
}
|
||||||
previous = got
|
previous = got
|
||||||
}
|
}
|
||||||
|
previous = time.Duration(0)
|
||||||
|
for stage := time.Duration(0); stage <= 2*time.Hour; stage += time.Minute {
|
||||||
|
got := repairDeadline(64<<10, time.Hour, stage)
|
||||||
|
if got < previous {
|
||||||
|
t.Fatalf("deadline not monotonic in stage timeout at %s: %s < %s", stage, got, previous)
|
||||||
|
}
|
||||||
|
previous = got
|
||||||
|
}
|
||||||
|
if got := repairDeadline(8<<10, time.Hour, time.Hour); got < repairDeadlineFloor {
|
||||||
|
t.Fatalf("ordinary payload was not floored: %s", got)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestShellStageRecoverableMalformedResultDoesNotFailRun proves a shell-enabled
|
// TestShellStageRecoverableMalformedResultDoesNotFailRun proves a shell-enabled
|
||||||
|
|||||||
@@ -0,0 +1,197 @@
|
|||||||
|
package stage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.a-wan.net/alan/pipelines/internal/opencode"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestPreAdmissionReachFailureRetriesOnMutatingStage proves a failure to reach
|
||||||
|
// the harness before any prompt or tool call is admitted retries within the
|
||||||
|
// existing budget even on an edit stage, with a fresh session and no replayed
|
||||||
|
// prompt. Nothing was admitted, so no admitted work can be replayed.
|
||||||
|
func TestPreAdmissionReachFailureRetriesOnMutatingStage(t *testing.T) {
|
||||||
|
def := modelDefinition()
|
||||||
|
def.Blocks["tools"] = "edit"
|
||||||
|
def.Configs["retries"] = "1"
|
||||||
|
var sessions, prompts atomic.Int64
|
||||||
|
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
|
||||||
|
path := args[2]
|
||||||
|
switch {
|
||||||
|
case path == "/api/session":
|
||||||
|
if sessions.Add(1) == 1 {
|
||||||
|
return nil, errors.New("connection refused")
|
||||||
|
}
|
||||||
|
return []byte(`{"data":{"id":"ses_reach_2"}}`), nil
|
||||||
|
case strings.HasSuffix(path, "/prompt"):
|
||||||
|
prompts.Add(1)
|
||||||
|
return []byte(`{}`), nil
|
||||||
|
case path == "/api/session/active":
|
||||||
|
return []byte(`{"data":{}}`), nil
|
||||||
|
case strings.HasSuffix(path, finalAssistantReadback):
|
||||||
|
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}`), nil
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("unexpected path %s", path)
|
||||||
|
}}}
|
||||||
|
result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
|
||||||
|
if err != nil || result.Output["summary"] != "done" {
|
||||||
|
t.Fatalf("pre-admission reach failure was not retried to success: %v %v", result, err)
|
||||||
|
}
|
||||||
|
if sessions.Load() != 2 || prompts.Load() != 1 || agent.SessionID != "ses_reach_2" {
|
||||||
|
t.Fatalf("reach retry did not use exactly one fresh session: sessions=%d prompts=%d session=%q", sessions.Load(), prompts.Load(), agent.SessionID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage proves a readback
|
||||||
|
// transport failure observed after the prompt was admitted is retried only on an
|
||||||
|
// explicit read-only stage; an edit, shell or legacy-bundle stage never replays
|
||||||
|
// the admitted prompt and ends with a HARNESS TRANSPORT failure.
|
||||||
|
func TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage(t *testing.T) {
|
||||||
|
run := func(name string, setTools bool, tools string) (Result, error, int64, int64) {
|
||||||
|
t.Helper()
|
||||||
|
def := modelDefinition()
|
||||||
|
if setTools {
|
||||||
|
def.Blocks["tools"] = tools
|
||||||
|
}
|
||||||
|
def.Configs["retries"] = "1"
|
||||||
|
var sessions, prompts, readbacks atomic.Int64
|
||||||
|
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
|
||||||
|
path := args[2]
|
||||||
|
switch {
|
||||||
|
case path == "/api/session":
|
||||||
|
n := sessions.Add(1)
|
||||||
|
return []byte(fmt.Sprintf(`{"data":{"id":"ses_rb_%d"}}`, n)), nil
|
||||||
|
case strings.HasSuffix(path, "/prompt"):
|
||||||
|
prompts.Add(1)
|
||||||
|
return []byte(`{}`), nil
|
||||||
|
case path == "/api/session/active":
|
||||||
|
return []byte(`{"data":{}}`), nil
|
||||||
|
case strings.HasSuffix(path, finalAssistantReadback):
|
||||||
|
if readbacks.Add(1) == 1 {
|
||||||
|
// Truncated readback: a post-admission transport failure.
|
||||||
|
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":`), nil
|
||||||
|
}
|
||||||
|
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"ok\"}}"}]}]}`), nil
|
||||||
|
case strings.HasSuffix(path, "/interrupt"):
|
||||||
|
return []byte(`{}`), nil
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("unexpected path %s", path)
|
||||||
|
}}}
|
||||||
|
result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
|
||||||
|
return result, err, sessions.Load(), prompts.Load()
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err, sessions, prompts := run("read-only", true, "read")
|
||||||
|
if err != nil || result.Output["summary"] != "ok" || sessions != 2 || prompts != 2 {
|
||||||
|
t.Fatalf("read-only readback failure was not retried: %v %v sessions=%d prompts=%d", result, err, sessions, prompts)
|
||||||
|
}
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
setTools bool
|
||||||
|
tools string
|
||||||
|
}{
|
||||||
|
{"edit", true, "edit"},
|
||||||
|
{"shell", true, "shell"},
|
||||||
|
{"legacy", false, ""},
|
||||||
|
} {
|
||||||
|
result, err, sessions, prompts := run(tc.name, tc.setTools, tc.tools)
|
||||||
|
if !errors.Is(err, ErrHarnessTransport) || sessions != 1 || prompts != 1 {
|
||||||
|
t.Fatalf("%s stage replayed admitted work on a readback failure: err=%v sessions=%d prompts=%d", tc.name, err, sessions, prompts)
|
||||||
|
}
|
||||||
|
if result.Output != nil {
|
||||||
|
t.Fatalf("%s stage retained a partial result", tc.name)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "HARNESS TRANSPORT") {
|
||||||
|
t.Fatalf("%s failure lost the transport category: %v", tc.name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline proves an
|
||||||
|
// expired context during a real CLI request keeps both the sanitized transport
|
||||||
|
// detail (operation/exit) and errors.Is(err, context.DeadlineExceeded) instead
|
||||||
|
// of returning a bare context error that discards the cause.
|
||||||
|
func TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
executable := filepath.Join(dir, "opencode")
|
||||||
|
if err := os.WriteFile(executable, []byte("#!/bin/sh\nexec sleep 30\n"), 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PIPELINES_OPENCODE", executable)
|
||||||
|
client := OpenCodeClient{Directory: dir}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 40*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
_, err := client.call(ctx, "post", "/api/session", nil)
|
||||||
|
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
t.Fatalf("expired transport call lost the context cause: %v", err)
|
||||||
|
}
|
||||||
|
var typed *openCodeError
|
||||||
|
if !errors.As(err, &typed) || typed.operation != "post /api/session" {
|
||||||
|
t.Fatalf("expired transport call discarded the typed transport error: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "OpenCode post /api/session") {
|
||||||
|
t.Fatalf("expired transport call discarded the operation detail: %v", err)
|
||||||
|
}
|
||||||
|
if !harnessTransportFailure(err) {
|
||||||
|
t.Fatalf("expired transport call was not classified as transport: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline proves
|
||||||
|
// the native V1 branch keeps both the typed transport error (the operation
|
||||||
|
// detail) and errors.Is(err, context.DeadlineExceeded) when the request failed
|
||||||
|
// under an expired context, instead of returning a bare context error that
|
||||||
|
// discards the transport shape. The native request path folds an in-flight
|
||||||
|
// deadline into ctx.Err(), so the retained transport detail is the operation and
|
||||||
|
// the parsed HTTP status is 0; the important invariant is that both shapes
|
||||||
|
// survive the join.
|
||||||
|
func TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline(t *testing.T) {
|
||||||
|
client := OpenCodeClient{Backend: "v1", connection: &opencode.Connection{Backend: "v1", URL: "http://127.0.0.1:1"}}
|
||||||
|
ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Second))
|
||||||
|
defer cancel()
|
||||||
|
_, err := client.call(ctx, "post", "/api/session/ses_native/generate", map[string]any{"prompt": "task"})
|
||||||
|
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
t.Fatalf("expired native transport call lost the context cause: %v", err)
|
||||||
|
}
|
||||||
|
var typed *openCodeError
|
||||||
|
if !errors.As(err, &typed) || typed.operation != "post /api/session/ses_native/generate" {
|
||||||
|
t.Fatalf("expired native transport call discarded the typed transport error: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "post /api/session/ses_native/generate") {
|
||||||
|
t.Fatalf("expired native transport call discarded the operation detail: %v", err)
|
||||||
|
}
|
||||||
|
if !harnessTransportFailure(err) {
|
||||||
|
t.Fatalf("expired native transport call was not classified as transport: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHarnessTransportSentinelsShareOneClassifierAndCategory proves the two
|
||||||
|
// transport shapes share one classifier and the single HARNESS TRANSPORT label
|
||||||
|
// while a bare stage deadline stays distinct from a pre-admission reach failure.
|
||||||
|
func TestHarnessTransportSentinelsShareOneClassifierAndCategory(t *testing.T) {
|
||||||
|
unreachable := harnessUnreachable(errors.New("connect refused"))
|
||||||
|
if !harnessTransportFailure(unreachable) || !harnessReachFailure(unreachable) {
|
||||||
|
t.Fatal("pre-admission reach failure was not classified")
|
||||||
|
}
|
||||||
|
if !strings.Contains(wrapModelStageFailure("worker", unreachable).Error(), "HARNESS TRANSPORT") {
|
||||||
|
t.Fatalf("pre-admission reach failure lost the transport category: %v", wrapModelStageFailure("worker", unreachable))
|
||||||
|
}
|
||||||
|
post := fmt.Errorf("%w: final assistant readback was malformed or incomplete", ErrHarnessTransport)
|
||||||
|
if !harnessTransportFailure(post) || harnessReachFailure(post) {
|
||||||
|
t.Fatal("post-admission transport failure was misclassified as a pre-admission reach failure")
|
||||||
|
}
|
||||||
|
if !strings.Contains(wrapModelStageFailure("worker", post).Error(), "HARNESS TRANSPORT") {
|
||||||
|
t.Fatal("post-admission transport failure lost the transport category")
|
||||||
|
}
|
||||||
|
if harnessTransportFailure(context.DeadlineExceeded) || harnessReachFailure(context.DeadlineExceeded) {
|
||||||
|
t.Fatal("a bare stage deadline must not be a transport or reach failure")
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user