Merge lane-srv3: keep the lane's reviewed work
This commit is contained in:
+33
-12
@@ -82,10 +82,11 @@ Format recovery has a separate bounded budget, even with `retries=0`:
|
||||
2. If the local pass cannot repair it, make at most one fresh, deny-all, retained
|
||||
text-generation request with only the malformed response and precise parsing
|
||||
error. The original implementing/reviewing agent is never re-prompted. The
|
||||
formatter's deadline is derived from the payload size and the attempt's own
|
||||
remaining budget rather than a fixed short constant, so a large result is not
|
||||
guaranteed to time out; it stays within the stage deadline, is floored for
|
||||
ordinary payloads, and keeps the 32 KiB input/output bound.
|
||||
formatter's deadline is derived from the payload size, the stage timeout and
|
||||
the attempt's own remaining budget rather than a fixed short constant, so a
|
||||
large or slower result is not guaranteed to time out; it stays within the
|
||||
stage deadline (clamped to `remaining` minus a margin), is floored for ordinary
|
||||
payloads, and keeps the 32 KiB input/output bound.
|
||||
3. Accept only insertion of missing commas/colons or trailing closing delimiters,
|
||||
with at most 16 insertions. Every existing string, number, literal and
|
||||
structural delimiter must remain verbatim and in order. Validate the full
|
||||
@@ -106,8 +107,14 @@ rerun project edits, commands, or the entire stage to repair final formatting.
|
||||
Configured execution retries remain for retryable text-generation transport
|
||||
errors; an explicit read-only stage may additionally re-emit a malformed final
|
||||
result and retry a post-admission deadline or transport failure up to `retries`.
|
||||
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
|
||||
read/glob/grep/edit/shell bundle), never has admitted tool work replayed. The
|
||||
A failure to reach the harness before any prompt or tool call is admitted
|
||||
(connecting, or creating the session) is a distinct reach failure: nothing was
|
||||
admitted, so it is retried within `retries` for any stage, mutating or not, with
|
||||
a fresh session and no replayed prompt or tool call. A post-admission readback,
|
||||
active-session poll or agent-execution failure is retried only on an explicit
|
||||
read-only stage. A stage whose tools include `edit`/`shell`, or that has no tools
|
||||
block (the legacy read/glob/grep/edit/shell bundle), never has admitted tool work
|
||||
replayed. The
|
||||
deterministic local pass and the fresh deny-all, tool-free formatter session never
|
||||
re-prompt or replay the admitted stage session, so a shell-enabled review stage
|
||||
gains malformed-result recovery without any replay of admitted work.
|
||||
@@ -121,11 +128,18 @@ and a recognized safe error category (such as session not found or service
|
||||
unavailable). Arbitrary stderr, provider messages and response bodies are not
|
||||
copied into the diagnostic. A failed V2 active-session poll (or native V1 `/wait`)
|
||||
is not evidence that the agent prompt failed to start, and does not trigger replay
|
||||
of stage work.
|
||||
of stage work. When the context expires while a request is still in flight, the
|
||||
error keeps both the sanitized transport detail (operation/status/exit) and
|
||||
`errors.Is(err, context.DeadlineExceeded)`, so a deadline that fired during a hung
|
||||
transport call is not replaced by a bare context error; a deadline measured at the
|
||||
attempt boundary with no transport in flight stays the stage overrun it is.
|
||||
Native `StructuredOutputError` is named directly, rather than reduced to a generic
|
||||
command failure. Native HTTP evidence records the actual HTTP operation/status;
|
||||
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
|
||||
Cleanup failures preserve the original generation error and active ownership.
|
||||
command failure, and an unrecognized provider execution-error name is reported as
|
||||
a bounded, sanitized identifier plus any 4xx/5xx status instead of a blanket
|
||||
summary (a name containing arbitrary bytes stays the generic message). Native HTTP
|
||||
evidence records the actual HTTP operation/status; an `exit_code` of -1 there is
|
||||
explicitly marked as not a subprocess exit. Cleanup failures preserve the original
|
||||
generation error and active ownership.
|
||||
|
||||
For existing native sessions with format-bearing messages, the HTTP readback
|
||||
workaround is read-only CLI `opencode export`. The exported session ID must match
|
||||
@@ -144,6 +158,11 @@ diagnostic:
|
||||
retry repeats work.
|
||||
- **HARNESS TRANSPORT** for request, V2 active-session poll and final-readback
|
||||
failures, including the sanitized `malformed or incomplete` readback message.
|
||||
A single classifier covers the two transport shapes: a pre-admission reach
|
||||
failure (connecting or creating the session) and a post-admission failure
|
||||
(readback/poll/agent execution). Both share the one `HARNESS TRANSPORT` label;
|
||||
only the pre-admission shape is retried on a mutating stage, because it replays
|
||||
nothing.
|
||||
|
||||
A structured-output validation failure is never reported as a harness transport
|
||||
failure, and vice versa. Neither label includes a provider response body, prompt,
|
||||
@@ -166,8 +185,10 @@ isolated formatter hit its own 30-second deadline. The malformed payload was
|
||||
retrieved read-only with `pipelines inspect 55 --json` and reproduced the exact
|
||||
byte offset 7837. It is an unescaped quote/colon inside a string value - an
|
||||
ambiguous case - so the deterministic pass correctly reports it unrecoverable
|
||||
rather than guessing, while the new derived deadline removes the fixed-deadline
|
||||
failure mode for larger payloads. These are source-level findings covered by mocked
|
||||
rather than guessing, while the stage-timeout-proportional derived deadline
|
||||
(floored, capped and clamped to the remaining attempt budget) removes the
|
||||
fixed-30-second-deadline failure mode for larger or slower payloads. These are
|
||||
source-level findings covered by mocked
|
||||
`internal/stage` fixtures. An updated checkout or installed skill is **not**
|
||||
evidence that a user's running server contains the fix; deploying/restarting is a
|
||||
separate authorized action.
|
||||
|
||||
@@ -428,3 +428,31 @@ blocked-class cause distribution and the transience of transport errors are
|
||||
hypotheses, not measurements. Raw captures are under
|
||||
`/Users/alan/.local/share/pipelines/data/artifacts/222/2/` and
|
||||
`/Users/alan/.local/share/pipelines/data/artifacts/222/4/` (outside the repo).
|
||||
|
||||
## 7. Transport and repair recovery (source-level, 2026-10-17)
|
||||
|
||||
Read-only pass over the §1 diagnostics after the harness-transport and
|
||||
format-repair change. No run was started, retried or cancelled; no server, ntfy,
|
||||
lane or device was restarted or redeployed; no provider inference was performed.
|
||||
Each shape is mapped to the code path that produces it. The per-run
|
||||
pre-admission vs post-admission and read-only vs mutating split was **not**
|
||||
re-inspected with `pipelines inspect`, so every per-run row is labelled *not
|
||||
verified* and the counts are not summed into a "recovered" total.
|
||||
|
||||
| failure shape (§1 class, quoted diagnostic) | run ids | code path | can recover | remains unrecoverable |
|
||||
|---|---|---|---|---|
|
||||
| execution error at readback (`OpenCode agent reported an execution error`) | 73,74,75,77,80,149,150,152,195,202 | post-admission `completedAgentResult` / `opencode.ExecutionError` | a stage that is explicit read-only may retry within `retries`(*not verified per run*) | a mutating/legacy stage never replays admitted work; a non-transient provider error stays terminal |
|
||||
| `context deadline exceeded` | 33,36,38,59,101,138 | attempt-boundary deadline, or a transport call that expired | a deadline during an in-flight request now keeps the transport cause and, when pre-admission, retries for any stage; a read-only stage retries a post-admission one | a genuine stage overrun on a mutating/legacy stage stays terminal; whether each run was a hung call or a real overrun is *not verified* |
|
||||
| readback failure (`final assistant readback had no completed result`) | 161 | post-admission `completedAgentResult` | explicit read-only stage may retry | mutating/legacy stage never replays |
|
||||
| native V1 `/wait` command failed (`CLI exit code 1`) | 4 | post-admission `openCodeError` on `/wait` | explicit read-only stage may retry | mutating/legacy stage never replays |
|
||||
| repair timeout (`isolated format repair failed … context deadline exceeded`) | 55,81,174,179,183,221 | `repairDeadline` (payload bytes + stage timeout, floored, capped, clamped to remaining − margin) | a slow-but-repairable payload now gets a longer bounded budget | run 55's payload is the ambiguous unescaped-quote/colon class (byte offset 7837) → unrecoverable; any payload past the 32 KiB bound stays unrecoverable |
|
||||
|
||||
Mapping notes: the execution-error and readback shapes are read back *after* the
|
||||
prompt was admitted, so they are post-admission transport failures and the
|
||||
no-tool-replay rule keeps them terminal on a mutating stage. The
|
||||
`session_transport_command_failed` shape (run 4) is the native V1 `/wait` call
|
||||
issued after the prompt, not a session-create reach failure. The repair-timeout
|
||||
shapes are `FINAL RESULT VALIDATION` (repair-stage), not transport. The new code
|
||||
also reports an unknown provider execution-error name as a bounded sanitized
|
||||
identifier plus any HTTP status, so future occurrences of runs 73–202 are more
|
||||
precisely diagnostic even when they are not retried.
|
||||
|
||||
@@ -20,14 +20,18 @@ Each playbook is *symptom → how to confirm → what to do*.
|
||||
the model may have done work but no final result was read.
|
||||
- **Confirm** `inspect --json` shows the category on the failed stage. A request
|
||||
failure, an active-session poll failure and a final-readback failure all carry this
|
||||
one category.
|
||||
one category. A request that failed while the context was still in flight shows
|
||||
both the sanitized transport detail and `context deadline exceeded`; an execution
|
||||
error names the sanitized error name/tag and any HTTP status, not a provider body.
|
||||
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
|
||||
pre-admission failure (nothing executed) from a post-admission one (tools may have
|
||||
run). An explicit read-only stage may retry; a stage whose tools include `edit`/
|
||||
`shell`, or that has no tools block, **never replays admitted work**. Do not re-run
|
||||
a full agent stage merely to recover a response's formatting. If it recurs, check
|
||||
the service and harness health (`references/operations.md`); do not weaken the
|
||||
stage to get past it.
|
||||
pre-admission reach failure (connecting or creating the session; nothing executed)
|
||||
from a post-admission one (tools may have run). A pre-admission reach failure
|
||||
replays nothing, so it is retried automatically within `retries` for any stage; a
|
||||
post-admission failure is retried only on an explicit read-only stage. A stage
|
||||
whose tools include `edit`/`shell`, or that has no tools block, **never replays
|
||||
admitted work**. Do not re-run a full agent stage merely to recover a response's
|
||||
formatting. If it recurs, check the service and harness health
|
||||
(`references/operations.md`); do not weaken the stage to get past it.
|
||||
|
||||
## Malformed model results
|
||||
|
||||
@@ -39,8 +43,9 @@ Each playbook is *symptom → how to confirm → what to do*.
|
||||
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
|
||||
appends missing closers, drops one trailing comma, or inserts the single
|
||||
comma/colon the parser expected; only if that is ambiguous does one deny-all
|
||||
formatter request run, with a deadline derived from the payload size and the
|
||||
attempt's remaining budget. Each path records a `repaired`, `partial` or
|
||||
formatter request run, with a deadline derived from the payload size, the stage
|
||||
timeout and the attempt's remaining budget, clamped so it never extends the
|
||||
stage's own deadline. Each path records a `repaired`, `partial` or
|
||||
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
|
||||
is never accepted as complete. On an explicit read-only stage a malformed result
|
||||
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
|
||||
|
||||
@@ -350,7 +350,11 @@ admitted tool-using stage: completed edits and tool side effects are retained an
|
||||
not undone, so a full retry repeats work. An explicit read-only stage is the one
|
||||
exception: a malformed result there mutates nothing, so it may be re-emitted up
|
||||
to `retries` with its `format_repairs` evidence retained, and a post-admission
|
||||
deadline or transport failure may be retried. A stage whose tools include
|
||||
deadline or transport failure may be retried. A failure to reach the harness
|
||||
before any prompt or tool call is admitted (connecting, or creating the session)
|
||||
replays nothing, so it is retried within `retries` for any stage, mutating or not.
|
||||
A stage whose tools include
|
||||
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
|
||||
bundle), is never replayed. Neither category exposes provider/model bodies or
|
||||
credentials.
|
||||
credentials. The isolated formatter's deadline is proportional to the stage
|
||||
timeout and clamped so it never extends the stage's own deadline.
|
||||
|
||||
@@ -80,7 +80,10 @@ specific authorization.
|
||||
Retained model-stage failures now carry a fixed sanitized category. A strictly
|
||||
invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a
|
||||
byte offset) is labeled `FINAL RESULT VALIDATION`; request, V2 active-session poll
|
||||
and final-readback failures are labeled `HARNESS TRANSPORT`. A validation failure
|
||||
and final-readback failures are labeled `HARNESS TRANSPORT`. A pre-admission
|
||||
request failure replays nothing and is retried for any stage; a post-admission
|
||||
failure is retried only on an explicit read-only stage (see the 2026-10-17 section
|
||||
below). A validation failure
|
||||
preserves the admitted session, transcript and evidence and performs no second tool
|
||||
execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history
|
||||
readback of exactly 262144 bytes of incomplete JSON (`invalid agent messages`), a
|
||||
@@ -100,8 +103,9 @@ Source-level, mocked `internal/stage` regression coverage (not live-provider
|
||||
evidence and not a deployment): malformed final-result recovery now tries a
|
||||
deterministic local pass first (append missing closers, drop one trailing comma,
|
||||
or insert the single comma/colon the parser expected), then at most one deny-all
|
||||
formatter request whose deadline is derived from the payload size and the
|
||||
attempt's remaining budget rather than a fixed 30-second constant. Every path
|
||||
formatter request whose deadline is derived from the payload size, the stage
|
||||
timeout and the attempt's remaining budget rather than a fixed 30-second
|
||||
constant. Every path
|
||||
records `repaired`, `partial` or `unrecoverable` in `format_repairs`; a `partial`
|
||||
or `unrecoverable` payload is never returned as a completed result, and an
|
||||
ambiguous payload (an unescaped quote/colon inside a string value) is never
|
||||
@@ -112,6 +116,26 @@ unchanged. The run-55 `ui_reviewer` payload was retrieved read-only with
|
||||
ambiguous error class, so it is reported unrecoverable rather than repaired. This
|
||||
does not update or restart any running server.
|
||||
|
||||
## Harness transport reach retry and repair deadline: 2026-10-17
|
||||
|
||||
Source-level, mocked `internal/stage`, `internal/opencode` and `internal/server`
|
||||
regression coverage (not live-provider evidence and not a deployment): the
|
||||
harness transport path now distinguishes a pre-admission reach failure
|
||||
(connecting or creating the session) from a post-admission readback/poll/agent
|
||||
failure under the one `HARNESS TRANSPORT` category. A pre-admission reach failure
|
||||
replays nothing, so it is retried within `retries` for any stage with a fresh
|
||||
session and no replayed prompt; a post-admission failure is retried only on an
|
||||
explicit read-only stage, and an `edit`/`shell`/legacy stage still never replays
|
||||
admitted work. A context that expires during a request keeps both the sanitized
|
||||
transport detail (operation/status/exit) and
|
||||
`errors.Is(err, context.DeadlineExceeded)`; a bare attempt-boundary deadline is
|
||||
not treated as a reach failure and stays terminal for a mutating stage. An
|
||||
unrecognized provider execution-error name is reported as a bounded, sanitized
|
||||
identifier plus any HTTP status. The isolated formatter deadline is now
|
||||
proportional to the stage timeout (floored, capped, and clamped to
|
||||
`remaining - margin` so it can never extend the stage/attempt deadline). These are
|
||||
mocked fixtures; a deployed server is not updated by this change.
|
||||
|
||||
## Failure-handling source changes: 2026-10-15
|
||||
|
||||
Source-level, mocked `internal/stage` and `internal/server` regression coverage
|
||||
|
||||
@@ -81,6 +81,15 @@ func ExecutionError(data json.RawMessage) error {
|
||||
}
|
||||
safeNames := map[string]bool{"APIError": true, "ProviderAuthError": true, "ContextOverflowError": true, "MessageAbortedError": true, "OutputLengthError": true, "UnknownError": true, "ServiceUnavailableError": true}
|
||||
if !safeNames[name] {
|
||||
// An unrecognized provider error name is still reported, but only as a
|
||||
// bounded, safe identifier plus any HTTP status. No provider body,
|
||||
// message, prompt or credential is copied.
|
||||
if safe := sanitizedExecutionName(name); safe != "" {
|
||||
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
|
||||
return fmt.Errorf("OpenCode agent reported an execution error: %s (HTTP %d)", safe, failure.Data.StatusCode)
|
||||
}
|
||||
return fmt.Errorf("OpenCode agent reported an execution error: %s", safe)
|
||||
}
|
||||
return fmt.Errorf("OpenCode agent reported an execution error")
|
||||
}
|
||||
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
|
||||
@@ -88,3 +97,23 @@ func ExecutionError(data json.RawMessage) error {
|
||||
}
|
||||
return fmt.Errorf("OpenCode agent execution error: %s", name)
|
||||
}
|
||||
|
||||
// sanitizedExecutionName reduces an unknown provider error name/tag to a
|
||||
// bounded, safe identifier so the diagnostic can name the failure shape without
|
||||
// copying an arbitrary message, prompt or credential. A name containing any
|
||||
// other byte returns empty and is reported as a generic execution error.
|
||||
func sanitizedExecutionName(name string) string {
|
||||
if len(name) > 64 {
|
||||
name = name[:64]
|
||||
}
|
||||
var b strings.Builder
|
||||
for _, r := range name {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_', r == '.', r == '-':
|
||||
b.WriteRune(r)
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -90,3 +90,26 @@ func TestNativeStructuredFailureIsClearAndDoesNotReplayGeneration(t *testing.T)
|
||||
t.Fatal("wait failure discarded provider response", err, string(retained))
|
||||
}
|
||||
}
|
||||
|
||||
// TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody proves an
|
||||
// unrecognized provider error name is reported as a bounded, safe identifier
|
||||
// plus any HTTP status instead of a blanket summary, while no provider message,
|
||||
// prompt or credential is copied and known names keep their exact wording.
|
||||
func TestExecutionErrorReportsSanitizedNameAndStatusWithoutBody(t *testing.T) {
|
||||
err := ExecutionError(json.RawMessage(`{"name":"TotallyUnknownError","data":{"statusCode":502,"message":"secret provider body"}}`))
|
||||
if err == nil || !strings.Contains(err.Error(), "TotallyUnknownError") || !strings.Contains(err.Error(), "HTTP 502") {
|
||||
t.Fatalf("unknown error name/status not reported: %v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret provider body") {
|
||||
t.Fatalf("diagnostic leaked the provider body: %v", err)
|
||||
}
|
||||
if err := ExecutionError(json.RawMessage(`{"_tag":"SomeTagError"}`)); err == nil || !strings.Contains(err.Error(), "SomeTagError") {
|
||||
t.Fatalf("unknown tag not reported: %v", err)
|
||||
}
|
||||
if err := ExecutionError(json.RawMessage(`{"name":"Injected Error: Bearer secret-token"}`)); err == nil || strings.Contains(err.Error(), "secret-token") || strings.Contains(err.Error(), "Bearer") || err.Error() != "OpenCode agent reported an execution error" {
|
||||
t.Fatalf("unsafe name was copied into the diagnostic: %v", err)
|
||||
}
|
||||
if err := ExecutionError(json.RawMessage(`{"name":"APIError","data":{"statusCode":500}}`)); err == nil || !strings.Contains(err.Error(), "APIError (HTTP 500)") {
|
||||
t.Fatalf("known safe name changed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -247,3 +247,97 @@ func TestRetryRejectsUnavailableOriginalDirectoryAndNonfailedRuns(t *testing.T)
|
||||
t.Fatal("nonexistent run accepted")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot proves,
|
||||
// through the real runner and CLI fixture, that a read-only stage retries a
|
||||
// post-admission transport (final-readback) failure with a fresh session while
|
||||
// an edit stage never replays the admitted prompt, and that the failed step's
|
||||
// diagnostic retains the underlying transport cause.
|
||||
func TestReadOnlyStageRetriesPostAdmissionTransportButMutatingStageDoesNot(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
dir := t.TempDir()
|
||||
executable := filepath.Join(dir, "opencode")
|
||||
t.Setenv("PIPELINES_OPENCODE", executable)
|
||||
scriptTemplate := `#!/bin/sh
|
||||
if [ "$1" = "--help" ]; then echo ' api'; exit 0; fi
|
||||
printf '%s %s\n' "$2" "$3" >> "@STATE@/calls"
|
||||
case "$3" in
|
||||
/api/session)
|
||||
n=0; if [ -f "@STATE@/sessions" ]; then n=$(cat "@STATE@/sessions"); fi
|
||||
n=$((n+1)); echo "$n" > "@STATE@/sessions"
|
||||
printf '{"data":{"id":"ses_bounded_%s"}}' "$n" ;;
|
||||
*/prompt) echo '{}' ;;
|
||||
/api/session/active) echo '{"data":{}}' ;;
|
||||
*/message*)
|
||||
r=0; if [ -f "@STATE@/readbacks" ]; then r=$(cat "@STATE@/readbacks"); fi
|
||||
r=$((r+1)); echo "$r" > "@STATE@/readbacks"
|
||||
if [ "$r" = "1" ]; then
|
||||
echo '{"data":[{"finish":"stop","content":'
|
||||
else
|
||||
echo '{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}'
|
||||
fi ;;
|
||||
*/interrupt) echo '{}' ;;
|
||||
*) exit 99 ;;
|
||||
esac
|
||||
`
|
||||
store, err := pipeline.Open(filepath.Join(dir, "runs.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
r, err := newRunner(ctx, store)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { cancel(); r.wg.Wait() }()
|
||||
toolsBlock := func(tools string) string {
|
||||
return "# stage worker\n\n```output\nResult.\n```\n\n```prompt\nDo the work.\n```\n\n```tools\n" + tools + "\n```\n\n```edges\nsuccess -> #success\nerror -> #failure\n```\n\n```configs\ntype=model\nprovider=opencode-go\nmodel=deepseek-v4.1-flash\nretries=1\ntimeout=30s\n```\n"
|
||||
}
|
||||
runStage := func(name, tools string) (status, sessions, diagnostic string) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
script := strings.ReplaceAll(scriptTemplate, "@STATE@", workspace)
|
||||
if err := os.WriteFile(executable, []byte(script), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := store.Submit(ctx, name, toolsBlock(tools)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := store.List(ctx)
|
||||
var id int64
|
||||
for _, item := range items {
|
||||
if item.Name == name {
|
||||
id = item.ID
|
||||
}
|
||||
}
|
||||
started, err := r.start(protocol.Message{PipelineID: id, Version: 1, RequestKey: name, Workspace: workspace, Input: map[string]any{"prompt": "task"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.wg.Wait()
|
||||
detail, err := store.GetRun(ctx, started.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(detail.Steps) > 0 {
|
||||
diagnostic = detail.Steps[0].Diagnostic
|
||||
}
|
||||
if data, err := os.ReadFile(filepath.Join(workspace, "sessions")); err == nil {
|
||||
sessions = strings.TrimSpace(string(data))
|
||||
}
|
||||
return detail.Status, sessions, diagnostic
|
||||
}
|
||||
|
||||
status, sessions, diagnostic := runStage("readonly", "read")
|
||||
if status != "succeeded" || sessions != "2" || diagnostic != "" {
|
||||
t.Fatalf("read-only stage did not retry the transport readback: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic)
|
||||
}
|
||||
status, sessions, diagnostic = runStage("edit", "edit")
|
||||
if status != "failed" || sessions != "1" {
|
||||
t.Fatalf("edit stage replayed admitted work: status=%q sessions=%q diagnostic=%q", status, sessions, diagnostic)
|
||||
}
|
||||
if !strings.Contains(diagnostic, "HARNESS TRANSPORT") || !strings.Contains(diagnostic, "malformed or incomplete") {
|
||||
t.Fatalf("step diagnostic lost the underlying transport cause: %q", diagnostic)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -55,7 +55,8 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
|
||||
if c.command == nil {
|
||||
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
|
||||
if err != nil {
|
||||
return "", err
|
||||
// Connecting to the service failed before any session existed.
|
||||
return "", harnessUnreachable(err)
|
||||
}
|
||||
defer connection.Close()
|
||||
c.connection = connection
|
||||
@@ -64,7 +65,9 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
|
||||
rules := c.permissionRules()
|
||||
data, err := c.call(ctx, "post", "/api/session", map[string]any{"title": "Pipeline audited agent stage", "agent": "build", "location": map[string]string{"directory": c.Directory}, "model": map[string]string{"providerID": config.Provider, "id": config.Model}, "permissions": rules})
|
||||
if err != nil {
|
||||
return "", err
|
||||
// No prompt or tool call was admitted: a request failure here is a
|
||||
// reachable retry rather than a reason to end the run.
|
||||
return "", harnessUnreachable(err)
|
||||
}
|
||||
var created struct {
|
||||
Data struct {
|
||||
|
||||
+40
-9
@@ -27,15 +27,42 @@ const (
|
||||
// diagnostics. It states the retained side effects and the cost of a full retry.
|
||||
const finalResultGuidance = "completed edits and tool side effects are retained and not undone; a full retry repeats work"
|
||||
|
||||
// harnessTransportFailure is the single classifier shared by both transport
|
||||
// sentinels and the typed OpenCode error: a request, poll or readback failure
|
||||
// that reports as the fixed HARNESS TRANSPORT category. A bare context deadline
|
||||
// is classified separately so a genuine stage overrun is never relabeled as a
|
||||
// transport failure.
|
||||
func harnessTransportFailure(err error) bool {
|
||||
var openCode *openCodeError
|
||||
return errors.Is(err, ErrHarnessUnreachable) || errors.Is(err, ErrHarnessTransport) || errors.As(err, &openCode)
|
||||
}
|
||||
|
||||
// harnessReachFailure reports whether err is a pre-admission harness reach
|
||||
// failure. It is distinct from a post-admission ErrHarnessTransport failure: no
|
||||
// session or prompt was admitted, so the retry replays no work.
|
||||
func harnessReachFailure(err error) bool {
|
||||
return errors.Is(err, ErrHarnessUnreachable)
|
||||
}
|
||||
|
||||
// admittedWorker is implemented by a client that can admit tool work before its
|
||||
// final result is read, so the retry loop can refuse to replay it without a
|
||||
// per-client type switch.
|
||||
type admittedWorker interface{ workAdmitted() bool }
|
||||
|
||||
// workAdmitted reports whether the client admitted any prompt or tool call, so
|
||||
// the retry loop may retry a transport failure on any stage only when it is
|
||||
// false, and only a read-only stage otherwise.
|
||||
func (c *AgentClient) workAdmitted() bool { return c.WorkStarted }
|
||||
func (c *PiClient) workAdmitted() bool { return c.WorkStarted }
|
||||
|
||||
// modelFailureCategory classifies a stage failure using error identity only, so
|
||||
// the label stays sanitized and bounded. Validation is checked first: a
|
||||
// structured-output failure is never reported as a transport failure and vice versa.
|
||||
func modelFailureCategory(err error) string {
|
||||
var openCode *openCodeError
|
||||
switch {
|
||||
case errors.Is(err, ErrInvalidModelResult):
|
||||
return finalResultValidation
|
||||
case errors.Is(err, ErrHarnessTransport), errors.As(err, &openCode), errors.Is(err, context.DeadlineExceeded):
|
||||
case harnessTransportFailure(err), errors.Is(err, context.DeadlineExceeded):
|
||||
return harnessTransport
|
||||
default:
|
||||
return ""
|
||||
@@ -294,15 +321,19 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
|
||||
break
|
||||
}
|
||||
replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err)
|
||||
// Never blindly replay admitted tool work: only a read-only stage may
|
||||
// retry a post-admission deadline or transport failure.
|
||||
if agent, ok := client.(*AgentClient); ok && agent.WorkStarted && !replayPostAdmission {
|
||||
// A reach failure observed before any work was admitted replays
|
||||
// nothing, so it is retryable within the existing budget for any
|
||||
// stage. After admission, only an explicit read-only stage may
|
||||
// retry; admitted tool work is never replayed.
|
||||
workStarted := false
|
||||
if worker, ok := client.(admittedWorker); ok {
|
||||
workStarted = worker.workAdmitted()
|
||||
}
|
||||
if workStarted && !replayPostAdmission {
|
||||
break
|
||||
}
|
||||
if pi, ok := client.(*PiClient); ok && pi.WorkStarted && !replayPostAdmission {
|
||||
break
|
||||
}
|
||||
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission) {
|
||||
preAdmissionReach := !workStarted && harnessReachFailure(err)
|
||||
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission && !preAdmissionReach) {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,23 @@ var ErrInvalidModelResult = errors.New("invalid model stage result")
|
||||
// validation and never replay admitted tool work.
|
||||
var ErrHarnessTransport = errors.New("harness transport failure")
|
||||
|
||||
// ErrHarnessUnreachable marks a request/transport failure observed before the
|
||||
// harness admitted any agent work: connecting to the service, or creating the
|
||||
// session. Nothing was executed or admitted, so re-attempting replays no prompt
|
||||
// or tool call and is safe for any stage, mutating or not. Post-admission
|
||||
// readback, poll and agent-execution failures stay ErrHarnessTransport so the
|
||||
// single fixed HARNESS TRANSPORT category and the no-tool-replay rule are
|
||||
// unchanged.
|
||||
var ErrHarnessUnreachable = errors.New("harness unreachable")
|
||||
|
||||
// harnessUnreachable marks err as a pre-admission reach failure. It is the single
|
||||
// wrap shared by the connect and session-create paths of both client types, so
|
||||
// they cannot drift: the sentinel and the underlying cause are always both
|
||||
// preserved.
|
||||
func harnessUnreachable(err error) error {
|
||||
return fmt.Errorf("%w: %w", ErrHarnessUnreachable, err)
|
||||
}
|
||||
|
||||
func modelPrompt(def Definition, spec ModelSpec, input Values) (string, error) {
|
||||
outcomes := make([]string, 0, len(def.Edges))
|
||||
for name := range def.Edges {
|
||||
|
||||
@@ -97,10 +97,14 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
|
||||
evidenceStatus = c.connection.LastHTTPStatus
|
||||
}
|
||||
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: evidenceStatus, NativeOperation: c.connection.LastHTTPOperation, Diagnostic: err.Error(), Stdout: string(data)}, started)
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
// Keep the transport shape (operation/status) with the context
|
||||
// cause: a deadline that fired while the request was in flight is
|
||||
// both, and neither may be discarded.
|
||||
return nil, errors.Join(ctxErr, transport)
|
||||
}
|
||||
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
|
||||
return nil, transport
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
@@ -146,14 +150,18 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
|
||||
code = exit.ExitCode()
|
||||
}
|
||||
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: cmd.Path, Directory: directory, ExitCode: code, HTTPStatus: status, Diagnostic: safeCommandDiagnostic(err, stdout.Bytes()), Stdout: stdout.String(), Stderr: stderr.String()}, started)
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
// Do not embed provider responses, prompts, credentials, or CLI logs in errors.
|
||||
// Keep the raw cause so errors.Is(err, exec.ErrWaitDelay) still holds while
|
||||
// Error() stays sanitized. This branch is only reached with a non-nil Run
|
||||
// error, so valid-looking partial stdout is never returned as success.
|
||||
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())}
|
||||
transport := &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: safeCommandDiagnostic(err, stdout.Bytes())}
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
// Keep the sanitized transport detail (operation/status/exit) together
|
||||
// with errors.Is(err, context.DeadlineExceeded) so the true cause is
|
||||
// not replaced by a bare context error.
|
||||
return nil, errors.Join(ctxErr, transport)
|
||||
}
|
||||
return nil, transport
|
||||
}
|
||||
if stdout.overflow {
|
||||
return nil, fmt.Errorf("OpenCode response exceeds 1 MiB limit")
|
||||
@@ -259,7 +267,8 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp
|
||||
if c.command == nil {
|
||||
connection, err := opencode.Open(ctx, opencode.Executable(), c.Directory, c.Backend)
|
||||
if err != nil {
|
||||
return "", err
|
||||
// Connecting to the service failed before any session existed.
|
||||
return "", harnessUnreachable(err)
|
||||
}
|
||||
defer connection.Close()
|
||||
c.connection = connection
|
||||
@@ -288,7 +297,9 @@ func (c *OpenCodeClient) Generate(ctx context.Context, config ModelConfig, promp
|
||||
"permissions": []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}},
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
// No prompt or tool call was admitted: a request failure here is a
|
||||
// reachable retry rather than a reason to end the run.
|
||||
return "", harnessUnreachable(err)
|
||||
}
|
||||
var session struct {
|
||||
Data struct {
|
||||
|
||||
@@ -129,14 +129,15 @@ func repairModelResult(ctx context.Context, client ModelGenerator, config ModelC
|
||||
return Result{}, fmt.Errorf("%w; no isolated format repair available; recovery status %s", parseError, repairStatusUnrecoverable)
|
||||
}
|
||||
// One formatter request, independent of execution/transport retries. The
|
||||
// deadline is derived from the payload size and the attempt's own remaining
|
||||
// budget rather than a fixed short constant, so a larger result is not
|
||||
// guaranteed to time out while the attempt context remains the outer bound.
|
||||
// deadline is derived from the payload size, the stage timeout and the
|
||||
// attempt's own remaining budget rather than a fixed short constant, so a
|
||||
// larger or slower result is not guaranteed to time out while the attempt
|
||||
// context remains the outer bound.
|
||||
remaining := time.Hour
|
||||
if deadline, ok := ctx.Deadline(); ok {
|
||||
remaining = time.Until(deadline)
|
||||
}
|
||||
repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining))
|
||||
repairCtx, cancel := context.WithTimeout(ctx, repairDeadline(len(original), remaining, config.Timeout))
|
||||
defer cancel()
|
||||
candidate, err := formatter.RepairFormat(repairCtx, config, original, parseError)
|
||||
if err := ctx.Err(); err != nil {
|
||||
@@ -291,9 +292,10 @@ func lexemeOffsets(text string, tokens []string) ([]int, bool) {
|
||||
}
|
||||
|
||||
// Isolated formatter deadline bounds. The allowance grows by one second per KiB
|
||||
// of payload up to a ceiling, is floored for ordinary payloads, and is clamped
|
||||
// to the attempt's remaining budget so the stage/attempt context stays the outer
|
||||
// bound and a large result is not guaranteed to time out.
|
||||
// of payload up to a ceiling and by a share of the stage timeout, is floored for
|
||||
// ordinary payloads, and is clamped to the attempt's remaining budget so the
|
||||
// stage/attempt context stays the outer bound and a large or slow result is not
|
||||
// guaranteed to time out.
|
||||
const (
|
||||
repairDeadlineFloor = 30 * time.Second
|
||||
repairDeadlineMax = 10 * time.Minute
|
||||
@@ -301,13 +303,13 @@ const (
|
||||
)
|
||||
|
||||
// repairDeadline derives the formatter deadline. It is O(1) and monotonic
|
||||
// non-decreasing in payloadBytes; it never exceeds remaining and, when the
|
||||
// remaining budget allows, never drops below repairDeadlineFloor.
|
||||
func repairDeadline(payloadBytes int, remaining time.Duration) time.Duration {
|
||||
// non-decreasing in payloadBytes and stageTimeout; it never exceeds remaining
|
||||
// and, when the remaining budget allows, never drops below repairDeadlineFloor.
|
||||
func repairDeadline(payloadBytes int, remaining, stageTimeout time.Duration) time.Duration {
|
||||
if remaining <= 0 {
|
||||
return 0
|
||||
}
|
||||
allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second
|
||||
allowance := repairDeadlineFloor + time.Duration(payloadBytes/1024)*time.Second + stageTimeout/4
|
||||
if allowance > repairDeadlineMax {
|
||||
allowance = repairDeadlineMax
|
||||
}
|
||||
|
||||
@@ -306,26 +306,37 @@ func TestPresentationStrippingBeforeDeterministicRepair(t *testing.T) {
|
||||
// TestRepairDeadlineIsMonotonicFlooredAndClamped proves the derived deadline, not
|
||||
// a fixed constant, bounds the single formatter request.
|
||||
func TestRepairDeadlineIsMonotonicFlooredAndClamped(t *testing.T) {
|
||||
if got := repairDeadline(0, time.Hour); got != repairDeadlineFloor {
|
||||
if got := repairDeadline(0, time.Hour, time.Minute); got < repairDeadlineFloor || got > repairDeadlineMax {
|
||||
t.Fatalf("floor not applied: %s", got)
|
||||
}
|
||||
if got := repairDeadline(1<<30, 24*time.Hour); got != repairDeadlineMax {
|
||||
if got := repairDeadline(1<<30, 24*time.Hour, time.Hour); got != repairDeadlineMax {
|
||||
t.Fatalf("ceiling not applied: %s", got)
|
||||
}
|
||||
if got := repairDeadline(10, 0); got != 0 {
|
||||
if got := repairDeadline(10, 0, time.Hour); got != 0 {
|
||||
t.Fatalf("zero remaining should yield zero: %s", got)
|
||||
}
|
||||
if clamped := repairDeadline(1<<20, 5*time.Second); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second {
|
||||
if clamped := repairDeadline(1<<20, 5*time.Second, time.Hour); clamped != 5*time.Second-repairDeadlineMargin || clamped > 5*time.Second {
|
||||
t.Fatalf("remaining budget was not the bound: %s", clamped)
|
||||
}
|
||||
previous := time.Duration(0)
|
||||
for size := 0; size <= 64<<10; size += 1024 {
|
||||
got := repairDeadline(size, time.Hour)
|
||||
got := repairDeadline(size, time.Hour, time.Minute)
|
||||
if got < previous {
|
||||
t.Fatalf("deadline not monotonic at %d: %s < %s", size, got, previous)
|
||||
t.Fatalf("deadline not monotonic in payload at %d: %s < %s", size, got, previous)
|
||||
}
|
||||
previous = got
|
||||
}
|
||||
previous = time.Duration(0)
|
||||
for stage := time.Duration(0); stage <= 2*time.Hour; stage += time.Minute {
|
||||
got := repairDeadline(64<<10, time.Hour, stage)
|
||||
if got < previous {
|
||||
t.Fatalf("deadline not monotonic in stage timeout at %s: %s < %s", stage, got, previous)
|
||||
}
|
||||
previous = got
|
||||
}
|
||||
if got := repairDeadline(8<<10, time.Hour, time.Hour); got < repairDeadlineFloor {
|
||||
t.Fatalf("ordinary payload was not floored: %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestShellStageRecoverableMalformedResultDoesNotFailRun proves a shell-enabled
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/opencode"
|
||||
)
|
||||
|
||||
// TestPreAdmissionReachFailureRetriesOnMutatingStage proves a failure to reach
|
||||
// the harness before any prompt or tool call is admitted retries within the
|
||||
// existing budget even on an edit stage, with a fresh session and no replayed
|
||||
// prompt. Nothing was admitted, so no admitted work can be replayed.
|
||||
func TestPreAdmissionReachFailureRetriesOnMutatingStage(t *testing.T) {
|
||||
def := modelDefinition()
|
||||
def.Blocks["tools"] = "edit"
|
||||
def.Configs["retries"] = "1"
|
||||
var sessions, prompts atomic.Int64
|
||||
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
|
||||
path := args[2]
|
||||
switch {
|
||||
case path == "/api/session":
|
||||
if sessions.Add(1) == 1 {
|
||||
return nil, errors.New("connection refused")
|
||||
}
|
||||
return []byte(`{"data":{"id":"ses_reach_2"}}`), nil
|
||||
case strings.HasSuffix(path, "/prompt"):
|
||||
prompts.Add(1)
|
||||
return []byte(`{}`), nil
|
||||
case path == "/api/session/active":
|
||||
return []byte(`{"data":{}}`), nil
|
||||
case strings.HasSuffix(path, finalAssistantReadback):
|
||||
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}`), nil
|
||||
}
|
||||
return nil, fmt.Errorf("unexpected path %s", path)
|
||||
}}}
|
||||
result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
|
||||
if err != nil || result.Output["summary"] != "done" {
|
||||
t.Fatalf("pre-admission reach failure was not retried to success: %v %v", result, err)
|
||||
}
|
||||
if sessions.Load() != 2 || prompts.Load() != 1 || agent.SessionID != "ses_reach_2" {
|
||||
t.Fatalf("reach retry did not use exactly one fresh session: sessions=%d prompts=%d session=%q", sessions.Load(), prompts.Load(), agent.SessionID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage proves a readback
|
||||
// transport failure observed after the prompt was admitted is retried only on an
|
||||
// explicit read-only stage; an edit, shell or legacy-bundle stage never replays
|
||||
// the admitted prompt and ends with a HARNESS TRANSPORT failure.
|
||||
func TestPostAdmissionReadbackFailureRetriesOnlyOnReadOnlyStage(t *testing.T) {
|
||||
run := func(name string, setTools bool, tools string) (Result, error, int64, int64) {
|
||||
t.Helper()
|
||||
def := modelDefinition()
|
||||
if setTools {
|
||||
def.Blocks["tools"] = tools
|
||||
}
|
||||
def.Configs["retries"] = "1"
|
||||
var sessions, prompts, readbacks atomic.Int64
|
||||
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
|
||||
path := args[2]
|
||||
switch {
|
||||
case path == "/api/session":
|
||||
n := sessions.Add(1)
|
||||
return []byte(fmt.Sprintf(`{"data":{"id":"ses_rb_%d"}}`, n)), nil
|
||||
case strings.HasSuffix(path, "/prompt"):
|
||||
prompts.Add(1)
|
||||
return []byte(`{}`), nil
|
||||
case path == "/api/session/active":
|
||||
return []byte(`{"data":{}}`), nil
|
||||
case strings.HasSuffix(path, finalAssistantReadback):
|
||||
if readbacks.Add(1) == 1 {
|
||||
// Truncated readback: a post-admission transport failure.
|
||||
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":`), nil
|
||||
}
|
||||
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"ok\"}}"}]}]}`), nil
|
||||
case strings.HasSuffix(path, "/interrupt"):
|
||||
return []byte(`{}`), nil
|
||||
}
|
||||
return nil, fmt.Errorf("unexpected path %s", path)
|
||||
}}}
|
||||
result, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
|
||||
return result, err, sessions.Load(), prompts.Load()
|
||||
}
|
||||
|
||||
result, err, sessions, prompts := run("read-only", true, "read")
|
||||
if err != nil || result.Output["summary"] != "ok" || sessions != 2 || prompts != 2 {
|
||||
t.Fatalf("read-only readback failure was not retried: %v %v sessions=%d prompts=%d", result, err, sessions, prompts)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
setTools bool
|
||||
tools string
|
||||
}{
|
||||
{"edit", true, "edit"},
|
||||
{"shell", true, "shell"},
|
||||
{"legacy", false, ""},
|
||||
} {
|
||||
result, err, sessions, prompts := run(tc.name, tc.setTools, tc.tools)
|
||||
if !errors.Is(err, ErrHarnessTransport) || sessions != 1 || prompts != 1 {
|
||||
t.Fatalf("%s stage replayed admitted work on a readback failure: err=%v sessions=%d prompts=%d", tc.name, err, sessions, prompts)
|
||||
}
|
||||
if result.Output != nil {
|
||||
t.Fatalf("%s stage retained a partial result", tc.name)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "HARNESS TRANSPORT") {
|
||||
t.Fatalf("%s failure lost the transport category: %v", tc.name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline proves an
|
||||
// expired context during a real CLI request keeps both the sanitized transport
|
||||
// detail (operation/exit) and errors.Is(err, context.DeadlineExceeded) instead
|
||||
// of returning a bare context error that discards the cause.
|
||||
func TestExpiredContextDuringCLITransportKeepsTransportDetailAndDeadline(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
executable := filepath.Join(dir, "opencode")
|
||||
if err := os.WriteFile(executable, []byte("#!/bin/sh\nexec sleep 30\n"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PIPELINES_OPENCODE", executable)
|
||||
client := OpenCodeClient{Directory: dir}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 40*time.Millisecond)
|
||||
defer cancel()
|
||||
_, err := client.call(ctx, "post", "/api/session", nil)
|
||||
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("expired transport call lost the context cause: %v", err)
|
||||
}
|
||||
var typed *openCodeError
|
||||
if !errors.As(err, &typed) || typed.operation != "post /api/session" {
|
||||
t.Fatalf("expired transport call discarded the typed transport error: %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "OpenCode post /api/session") {
|
||||
t.Fatalf("expired transport call discarded the operation detail: %v", err)
|
||||
}
|
||||
if !harnessTransportFailure(err) {
|
||||
t.Fatalf("expired transport call was not classified as transport: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline proves
|
||||
// the native V1 branch keeps both the typed transport error (the operation
|
||||
// detail) and errors.Is(err, context.DeadlineExceeded) when the request failed
|
||||
// under an expired context, instead of returning a bare context error that
|
||||
// discards the transport shape. The native request path folds an in-flight
|
||||
// deadline into ctx.Err(), so the retained transport detail is the operation and
|
||||
// the parsed HTTP status is 0; the important invariant is that both shapes
|
||||
// survive the join.
|
||||
func TestExpiredContextDuringNativeTransportKeepsTransportDetailAndDeadline(t *testing.T) {
|
||||
client := OpenCodeClient{Backend: "v1", connection: &opencode.Connection{Backend: "v1", URL: "http://127.0.0.1:1"}}
|
||||
ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Second))
|
||||
defer cancel()
|
||||
_, err := client.call(ctx, "post", "/api/session/ses_native/generate", map[string]any{"prompt": "task"})
|
||||
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("expired native transport call lost the context cause: %v", err)
|
||||
}
|
||||
var typed *openCodeError
|
||||
if !errors.As(err, &typed) || typed.operation != "post /api/session/ses_native/generate" {
|
||||
t.Fatalf("expired native transport call discarded the typed transport error: %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "post /api/session/ses_native/generate") {
|
||||
t.Fatalf("expired native transport call discarded the operation detail: %v", err)
|
||||
}
|
||||
if !harnessTransportFailure(err) {
|
||||
t.Fatalf("expired native transport call was not classified as transport: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHarnessTransportSentinelsShareOneClassifierAndCategory proves the two
|
||||
// transport shapes share one classifier and the single HARNESS TRANSPORT label
|
||||
// while a bare stage deadline stays distinct from a pre-admission reach failure.
|
||||
func TestHarnessTransportSentinelsShareOneClassifierAndCategory(t *testing.T) {
|
||||
unreachable := harnessUnreachable(errors.New("connect refused"))
|
||||
if !harnessTransportFailure(unreachable) || !harnessReachFailure(unreachable) {
|
||||
t.Fatal("pre-admission reach failure was not classified")
|
||||
}
|
||||
if !strings.Contains(wrapModelStageFailure("worker", unreachable).Error(), "HARNESS TRANSPORT") {
|
||||
t.Fatalf("pre-admission reach failure lost the transport category: %v", wrapModelStageFailure("worker", unreachable))
|
||||
}
|
||||
post := fmt.Errorf("%w: final assistant readback was malformed or incomplete", ErrHarnessTransport)
|
||||
if !harnessTransportFailure(post) || harnessReachFailure(post) {
|
||||
t.Fatal("post-admission transport failure was misclassified as a pre-admission reach failure")
|
||||
}
|
||||
if !strings.Contains(wrapModelStageFailure("worker", post).Error(), "HARNESS TRANSPORT") {
|
||||
t.Fatal("post-admission transport failure lost the transport category")
|
||||
}
|
||||
if harnessTransportFailure(context.DeadlineExceeded) || harnessReachFailure(context.DeadlineExceeded) {
|
||||
t.Fatal("a bare stage deadline must not be a transport or reach failure")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user