Make agent-stage retries safe and retain timeout failure evidence

Fix four stage failure-handling and observability defects (A2, A3, A6, A9).

- A2: add safeToReplay(def) as the single definition of retryable post-admission
  work - only an explicit tools block that selects neither edit nor shell. The
  agent/pi WorkStarted guards now break only when the stage is not safe to
  replay, so read-only stages genuinely retry a post-admission timeout while
  mutating and legacy-bundle stages still never replay admitted work.
- A6: re-emit a malformed final result on an explicit read-only stage up to the
  retry budget, preserving format_repairs evidence; mutating stages keep the
  immediate return and their session.
- A3: timed-out stages retain usable evidence. waitForSessionIdle records one
  bounded sanitized OperationEvidence on deadline; the agent deferred cleanup
  does a bounded best-effort latest-assistant readback and settles knowable V2
  usage; RetainSession adds a retained_session pointer to failed step
  transcripts so the session is visible beneath the BoundTranscript threshold.
- A9: a stage outcome routed to #failure now yields a non-empty run diagnostic
  ("stage <name> returned outcome <outcome>") while preferring an existing step
  diagnostic.

Adds focused stage and server tests for retry safety, malformed-result
re-emission, timeout evidence retention and run-level diagnostics; updates the
packaged skill references and docs/cli.md, docs/model-results.md.
This commit is contained in:
2026-10-15 18:27:22 +01:00
parent 0f5a188c6a
commit f57457aec9
13 changed files with 564 additions and 35 deletions
+6 -3
View File
@@ -396,9 +396,12 @@ Usage is polled during execution and read again afterward; an in-flight request
may exceed a limit before usage arrives. These are not provider-side hard caps.
Automatic retries cover retryable transport failures and per-attempt timeouts.
They do not replay admitted tool work, invalid results, or budget failures.
Cleanup must be confirmed before another attempt. Existing explicit `retries=0`
settings remain unchanged. The old 30-minute run deadline no longer truncates
An explicit read-only stage (a tools block listing neither `edit` nor `shell`)
may also retry a post-admission deadline or transport failure and re-emit a
malformed final result up to `retries`; a stage whose tools include `edit`/`shell`
or that has no tools block never replays admitted tool work or invalid results.
Budget failures always stop. Cleanup must be confirmed before another attempt.
Existing explicit `retries=0` settings remain unchanged. The old 30-minute run deadline no longer truncates
stage timeouts; the 30-step and five-visits-per-stage limits still apply.
Native ntfy stages also accept `timeout` (default `1h`), but never automatically
retry potentially delivered notifications and do not accept model budgets.
+7 -2
View File
@@ -47,7 +47,10 @@ Format recovery has a separate bounded budget, even with `retries=0`:
If recovery fails, report the original precise error and repair failure; never
rerun project edits, commands, or the entire stage to repair final formatting.
Configured execution retries remain for retryable text-generation transport
errors; audited tool-enabled execution is never automatically replayed.
errors; an explicit read-only stage may additionally re-emit a malformed final
result and retry a post-admission deadline or transport failure up to `retries`.
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
read/glob/grep/edit/shell bundle), never has admitted tool work replayed.
Repair records include the original/candidate response, parsing diagnostic,
method and acceptance flag in the bounded transcript preview. `pipelines
inspect <run-id> --json` and the TUI evidence section expose these records.
@@ -86,7 +89,9 @@ failure, and vice versa. Neither label includes a provider response body, prompt
credential or arbitrary model content. Duplicate nested fields are rejected by
strict parsing rather than merged or last-wins, and only a lossless insertion of
missing commas/colons or trailing closers is repaired, so an unknown evidence or
verdict value can never become `success` and admitted tool work is never replayed.
verdict value can never become `success`, and admitted tool work on a mutating or
legacy-bundle stage is never replayed (only an explicit read-only stage may
re-emit a malformed result within `retries`).
Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned
exactly 262144 bytes of incomplete JSON (`invalid agent messages`); a reviewer
+9 -4
View File
@@ -56,9 +56,11 @@ authentication and inference require `pipelines provider-check` on the execution
computer/container. Optional `retries` defaults to 2 (three total attempts),
and accepts integers 0–100; explicit 0 disables execution retries. Optional
`timeout` defaults to `1h` per attempt and accepts positive durations such as
`30m` or `90s`. Automatic retries only cover retryable transport failures and
timeouts; admitted tool work is never replayed automatically. Optional
`budget_usd=1.50` and `budget_tokens=50000` independently cap reported usage.
`30m` or `90s`. Automatic retries cover retryable transport failures and
timeouts; an explicit read-only stage may additionally retry a post-admission
deadline or transport failure, while a stage with `edit`/`shell` (or no tools
block) never replays admitted work. Optional `budget_usd=1.50` and
`budget_tokens=50000` independently cap reported usage.
They accumulate across attempts, format repair sessions, and repeated visits
to this stage in the same run. Omit either budget to leave that resource unlimited.
Both budgets must be positive; tokens include input, output, reasoning and cache
@@ -104,7 +106,10 @@ permission policy. Keep ordinary workflows language-neutral and unrestricted
within their run directory unless the user asks for a narrower scope.
Malformed final JSON has bounded, syntax-only recovery even with `retries=0`;
it never replays stage tools or changes verdict/evidence. Compatible native stable
it never changes verdict/evidence. On an explicit read-only stage a rejected
malformed result may be re-emitted up to `retries` with its `format_repairs`
evidence retained; a stage with `edit`/`shell` or no tools block never replays
stage tools or re-emits the result. Compatible native stable
backends request schema output with schema retries disabled; a retained deny-all,
no-inference compatibility probe identifies known format incompatibility. Incompatible
native backends and V2 require strict local envelope validation. Do not promise
@@ -196,5 +196,10 @@ VALIDATION` names the exact duplicate-field/syntax detail and byte offset for a
strictly invalid final result; `HARNESS TRANSPORT` covers request, active-session
poll and final-readback failures. A validation failure never re-prompts the
admitted tool-using stage: completed edits and tool side effects are retained and
not undone, so a full retry repeats work. Neither category exposes provider/model
bodies or credentials.
not undone, so a full retry repeats work. An explicit read-only stage is the one
exception: a malformed result there mutates nothing, so it may be re-emitted up
to `retries` with its `format_repairs` evidence retained, and a post-admission
deadline or transport failure may be retried. A stage whose tools include
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
bundle), is never replayed. Neither category exposes provider/model bodies or
credentials.
@@ -94,6 +94,20 @@ field is sent. A review-only fresh execution is a future recovery idea, **not
implemented**. Onboarding UI visual approval remains unresolved and must not be
reported complete.
## Failure-handling source changes: 2026-10-15
Source-level, mocked `internal/stage` and `internal/server` regression coverage
(not live-provider evidence): an explicit read-only tools stage now actually
retries a post-admission stage deadline or harness transport failure, creating a
fresh bounded session per attempt, while a stage with `edit`/`shell` or no tools
block still never replays admitted work; a malformed final result is re-emitted
up to `retries` only on such a read-only stage with its `format_repairs` evidence
retained. A timed-out agent stage settles knowable V2 usage and retains one
bounded, redacted latest-assistant readback plus a `retained_session` pointer in
its stored transcript. A declared outcome routed to `#failure` with no step
diagnostic now yields a run diagnostic like `stage <key> returned outcome
"blocked"`; an existing step diagnostic is still used verbatim.
## Dated source-level coverage: 2026-10-11
Source-level/mocked regression coverage (not live-provider evidence):
+66
View File
@@ -0,0 +1,66 @@
package server
import (
"context"
"errors"
"path/filepath"
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
)
// TestFailureRouteRunDiagnostic proves A9: a declared outcome routed to #failure
// with an empty step diagnostic still yields a meaningful run diagnostic, while
// an existing step diagnostic is preferred verbatim.
func TestFailureRouteRunDiagnostic(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "runs.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
source := strings.Replace(pipeline.Template([]string{"worker"}, true), "success -> #success", "success -> #success\nblocked -> #failure", 1)
if _, err := store.Submit(ctx, "diagnostic", source); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
r, err := newRunner(ctx, store)
if err != nil {
t.Fatal(err)
}
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
return stage.Result{Outcome: "blocked", Output: stage.Values{}}, nil
}
run, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "blocked", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
detail, err := store.GetRun(ctx, run.ID)
if err != nil {
t.Fatal(err)
}
if detail.Status != "failed" || detail.Outcome != "error" || !strings.Contains(detail.Diagnostic, `stage worker returned outcome "blocked"`) {
t.Fatalf("empty run diagnostic not replaced: %+v", detail)
}
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
return stage.Result{}, errors.New("fixture execution failed")
}
run, err = r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "execution", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
detail, err = store.GetRun(ctx, run.ID)
if err != nil {
t.Fatal(err)
}
if detail.Status != "failed" || detail.Diagnostic != "fixture execution failed" {
t.Fatalf("existing step diagnostic not preferred: %+v", detail)
}
}
+10 -1
View File
@@ -473,6 +473,11 @@ func (r *runner) run(ctx context.Context, id int64, d *protocol.PipelineDetail,
step.Outcome = "error"
step.Output = stage.Values{"error": err.Error(), "stage": key, "input": input}
}
// A failed or timed-out step carries the session pointer even when the
// bounded transcript would otherwise be only its stage_limits envelope.
if err != nil && stageSession != "" {
step.Transcript = stage.RetainSession(step.Transcript, stageSession)
}
step.Target = def.Edges[step.Outcome]
step.Control = &protocol.ControlEvidence{Visit: visits[key], MaxVisits: maxVisits, Memory: reads}
if err == nil && selected.Limits != nil && visits[key] >= maxVisits {
@@ -526,7 +531,11 @@ func (r *runner) run(ctx context.Context, id int64, d *protocol.PipelineDetail,
return
}
if step.Target == protocol.FailureEnd {
finish("failed", "error", step.Diagnostic)
diagnostic := step.Diagnostic
if diagnostic == "" {
diagnostic = fmt.Sprintf("stage %s returned outcome %q", step.Stage, step.Outcome)
}
finish("failed", "error", diagnostic)
return
}
if step.Target == "" {
+22
View File
@@ -93,6 +93,7 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 15*time.Second)
defer cancel()
if err != nil {
c.captureTimedOutEvidence(ctx, err, config, watch)
if stopErr := c.Interrupt(cleanup, c.SessionID); stopErr != nil {
err = fmt.Errorf("agent interrupt failed; retained session %s needs inspection: %w", c.SessionID, errors.Join(err, stopErr, ErrStageCleanup))
return
@@ -185,6 +186,27 @@ func (c OpenCodeClient) settleV2Usage(ctx context.Context, budget *Budget, id st
_ = budget.readUsage(id, data, false)
}
// captureTimedOutEvidence retains one bounded, redacted best-effort readback of
// the latest assistant message after a post-admission deadline or transport
// failure, and settles knowable V2 usage for an unbudgeted stage. A failed
// readback is ignored and never changes the returned error.
func (c *AgentClient) captureTimedOutEvidence(ctx context.Context, err error, config ModelConfig, watch *budgetWatch) {
if !c.WorkStarted || c.SessionID == "" || !(errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport)) {
return
}
readCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
data, readErr := c.call(readCtx, "get", "/api/session/"+c.SessionID+finalAssistantReadback, nil)
cancel()
if readErr == nil && len(data) > 0 {
c.Transcript = RedactEvidence(string(data))
}
// A budgeted stage already settles through watch.finish; only an unbudgeted
// V2 stage needs this bounded aggregate read.
if watch == nil && c.Backend == "v2" {
c.settleV2Usage(ctx, config.Budget, c.SessionID)
}
}
// finalAssistantReadback is the single bounded request used to select a V2
// final result: only the latest assistant message. Fetching the full assistant
// history also pulls large prior tool-read contents and can exceed the response
+24
View File
@@ -234,6 +234,30 @@ func BoundTranscript(transcript, sessionID string) string {
return string(encoded)
}
// RetainSession adds a retained_session pointer to an already-JSON-object
// transcript so a failed or timed-out step carries the session ID even below the
// 32 KiB BoundTranscript threshold. Non-object and empty transcripts are
// returned unchanged and the value is JSON-encoded, so the result stays valid.
func RetainSession(transcript, sessionID string) string {
if sessionID == "" {
return transcript
}
var data map[string]json.RawMessage
if json.Unmarshal([]byte(transcript), &data) != nil || data == nil {
return transcript
}
encoded, err := json.Marshal(sessionID)
if err != nil {
return transcript
}
data["retained_session"] = encoded
merged, err := json.Marshal(data)
if err != nil {
return transcript
}
return string(merged)
}
func (l *OperationLog) record(e OperationEvidence, started time.Time) {
if l == nil {
return
+59 -23
View File
@@ -59,6 +59,26 @@ func wrapModelStageFailure(key string, err error) error {
}
}
// replayablePostAdmission reports whether a post-admission error can be retried
// from the start without risking a repeated tool mutation: a stage context
// deadline or a harness transport failure. A structured-output validation
// failure is classified separately as ErrInvalidModelResult.
func replayablePostAdmission(err error) bool {
return errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport)
}
// toleratedModelResultRetry reports whether a malformed final result may be
// re-emitted. It requires an explicit read-only stage, remaining attempts,
// live work, available budget and a plain validation error; fail-closed
// accounting and cleanup errors still stop immediately.
func toleratedModelResultRetry(ctx context.Context, def Definition, config ModelConfig, attempt int, resultErr error) bool {
return errors.Is(resultErr, ErrInvalidModelResult) &&
!errors.Is(resultErr, ErrBudgetExceeded) &&
!errors.Is(resultErr, ErrBudgetAccounting) &&
!errors.Is(resultErr, ErrStageCleanup) &&
safeToReplay(def) && attempt < config.Retries && ctx.Err() == nil && config.Budget != nil && config.Budget.Check() == nil
}
var providerID = regexp.MustCompile(`^[A-Za-z0-9_.-]+$`)
var modelID = regexp.MustCompile(`^[A-Za-z0-9_./:-]+$`)
@@ -208,8 +228,9 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
return Result{}, fmt.Errorf("%w: generator cannot enforce usage limits", ErrBudgetAccounting)
}
}
// Replaying admitted tool work can repeat mutations. Only pre-admission
// transport errors may automatically retry an audited agent.
// Replaying admitted tool work can repeat mutations. Only a stage whose
// explicit tools cannot mutate the run directory (safeToReplay) may
// automatically retry work that started after admission.
if agent, ok := client.(*AgentClient); ok {
if tools, explicit, _ := ToolNames(def); explicit {
agent.Tools = tools
@@ -232,6 +253,7 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
if generateErr == nil && attemptCtx.Err() != nil {
generateErr = attemptCtx.Err()
}
replayPostAdmission := false
if generateErr == nil {
result, resultErr := decodeModelResult(stageResultText(text), def.Edges)
if resultErr != nil {
@@ -241,27 +263,41 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
resultErr = errors.Join(resultErr, attemptCtx.Err())
}
cancel()
return result, wrapModelStageFailure(def.Key, resultErr)
}
err = generateErr
if attemptCtx.Err() != nil && !errors.Is(err, ErrBudgetExceeded) && !errors.Is(err, ErrBudgetAccounting) {
err = errors.Join(err, attemptCtx.Err())
}
cancel()
if ctx.Err() != nil {
return Result{}, ctx.Err()
}
if errors.Is(err, ErrBudgetExceeded) || errors.Is(err, ErrBudgetAccounting) || errors.Is(err, ErrStageCleanup) {
break
}
if agent, ok := client.(*AgentClient); ok && agent.WorkStarted {
break
}
if pi, ok := client.(*PiClient); ok && pi.WorkStarted {
break
}
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded)) {
break
if resultErr == nil {
return result, nil
}
// A malformed final result mutates nothing on an explicit read-only
// stage, so it may be re-emitted within the retry budget. A mutating
// or legacy-bundle stage retains its admitted work and returns now.
if toleratedModelResultRetry(ctx, def, spec.Config, attempt, resultErr) {
err = resultErr
} else {
return result, wrapModelStageFailure(def.Key, resultErr)
}
} else {
err = generateErr
if attemptCtx.Err() != nil && !errors.Is(err, ErrBudgetExceeded) && !errors.Is(err, ErrBudgetAccounting) {
err = errors.Join(err, attemptCtx.Err())
}
cancel()
if ctx.Err() != nil {
return Result{}, ctx.Err()
}
if errors.Is(err, ErrBudgetExceeded) || errors.Is(err, ErrBudgetAccounting) || errors.Is(err, ErrStageCleanup) {
break
}
replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err)
// Never blindly replay admitted tool work: only a read-only stage may
// retry a post-admission deadline or transport failure.
if agent, ok := client.(*AgentClient); ok && agent.WorkStarted && !replayPostAdmission {
break
}
if pi, ok := client.(*PiClient); ok && pi.WorkStarted && !replayPostAdmission {
break
}
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission) {
break
}
}
delay := time.Duration(1<<min(attempt, 4)) * 100 * time.Millisecond
timer := time.NewTimer(delay)
+11
View File
@@ -196,6 +196,7 @@ func (c OpenCodeClient) waitForSessionIdle(ctx context.Context, sessionID string
if interval <= 0 {
interval = defaultSessionPollInterval
}
started := time.Now()
for {
active, err := c.sessionActive(ctx, sessionID)
if err != nil {
@@ -208,6 +209,16 @@ func (c OpenCodeClient) waitForSessionIdle(ctx context.Context, sessionID string
select {
case <-ctx.Done():
timer.Stop()
// A settled session is the only success signal; retain one bounded
// sanitized record so a timed-out wait is not an empty transcript.
c.Operations.record(OperationEvidence{
Operation: "wait for session idle",
Backend: c.Backend,
Executable: opencode.Executable(),
Directory: c.Directory,
ExitCode: -1,
Diagnostic: "stage deadline reached while waiting for the agent session to become idle",
}, started)
return ctx.Err()
case <-timer.C:
}
+311
View File
@@ -0,0 +1,311 @@
package stage
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"sync/atomic"
"testing"
"time"
)
// repairGenerator is a generatorFunc that also records format repairs, so a
// toleration test can assert the retained format_repairs evidence.
type repairGenerator struct {
generate func(context.Context, ModelConfig, string) (string, error)
repairs []FormatRepair
}
func (g *repairGenerator) Generate(ctx context.Context, config ModelConfig, prompt string) (string, error) {
return g.generate(ctx, config, prompt)
}
func (g *repairGenerator) RecordFormatRepair(repair FormatRepair) {
g.repairs = append(g.repairs, repair)
}
func TestSafeToReplayRequiresExplicitReadOnlyTools(t *testing.T) {
legacy := modelDefinition()
if safeToReplay(legacy) {
t.Fatal("absent tools block (legacy bundle) must not be replayable")
}
mutating := modelDefinition()
mutating.Blocks["tools"] = "read\nedit"
if safeToReplay(mutating) {
t.Fatal("edit-capable stage must not be replayable")
}
shell := modelDefinition()
shell.Blocks["tools"] = "shell"
if safeToReplay(shell) {
t.Fatal("shell-capable stage must not be replayable")
}
readonly := modelDefinition()
readonly.Blocks["tools"] = "read\nglob\ngrep\nwebsearch\nwebfetch"
if !safeToReplay(readonly) {
t.Fatal("explicit read-only stage should be replayable")
}
empty := modelDefinition()
empty.Blocks["tools"] = ""
if !safeToReplay(empty) {
t.Fatal("explicit empty tools block is side-effect free")
}
}
// TestReadOnlyAgentRetriesPostAdmissionDeadline proves A2: an explicit
// read-only agent stage retries a post-admission deadline on attempt 2 with a
// fresh bounded session, and the shared budget records both attempts.
func TestReadOnlyAgentRetriesPostAdmissionDeadline(t *testing.T) {
def := modelDefinition()
def.Blocks["tools"] = "read"
def.Configs["retries"] = "1"
def.Configs["timeout"] = "30ms"
var sessions, prompts atomic.Int64
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(ctx context.Context, _ string, args ...string) ([]byte, error) {
path := args[2]
switch {
case path == "/api/session":
n := sessions.Add(1)
return []byte(fmt.Sprintf(`{"data":{"id":"ses_read_%d"}}`, n)), nil
case strings.HasSuffix(path, "/prompt"):
if prompts.Add(1) == 1 {
<-ctx.Done()
return nil, ctx.Err()
}
return []byte(`{}`), nil
case path == "/api/session/active":
return []byte(`{"data":{}}`), nil
case strings.HasSuffix(path, finalAssistantReadback):
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}`), nil
case strings.HasSuffix(path, "/interrupt"):
return []byte(`{}`), nil
}
return nil, fmt.Errorf("unexpected path %s", path)
}}}
budget := NewBudget(0, 0)
result, err := (ModelStage{Client: agent}).Run(WithBudget(context.Background(), budget), def, nil)
if err != nil || result.Output["summary"] != "done" {
t.Fatalf("read-only deadline was not retried to success: %v %v", result, err)
}
if sessions.Load() != 2 || prompts.Load() != 2 || budget.Snapshot().Attempts != 2 {
t.Fatalf("expected two fresh attempts, got sessions=%d prompts=%d attempts=%d", sessions.Load(), prompts.Load(), budget.Snapshot().Attempts)
}
}
// TestMutatingAgentNeverRetriesPostAdmissionDeadline proves the narrowed
// guard: a mutating stage retains its admitted session and never replays.
func TestMutatingAgentNeverRetriesPostAdmissionDeadline(t *testing.T) {
for _, tools := range []string{"edit", "read\nshell", ""} {
t.Run("tools="+tools, func(t *testing.T) {
def := modelDefinition()
if tools != "" {
def.Blocks["tools"] = tools
}
def.Configs["retries"] = "2"
def.Configs["timeout"] = "30ms"
var sessions, prompts atomic.Int64
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(ctx context.Context, _ string, args ...string) ([]byte, error) {
path := args[2]
switch {
case path == "/api/session":
n := sessions.Add(1)
return []byte(fmt.Sprintf(`{"data":{"id":"ses_mut_%d"}}`, n)), nil
case strings.HasSuffix(path, "/prompt"):
prompts.Add(1)
<-ctx.Done()
return nil, ctx.Err()
case strings.HasSuffix(path, finalAssistantReadback):
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{}"}]}]}`), nil
case strings.HasSuffix(path, "/interrupt"):
return []byte(`{}`), nil
}
return nil, fmt.Errorf("unexpected path %s", path)
}}}
_, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("expected deadline failure, got %v", err)
}
if sessions.Load() != 1 || prompts.Load() != 1 || agent.SessionID != "ses_mut_1" {
t.Fatalf("mutating stage replayed admitted work: sessions=%d prompts=%d session=%q", sessions.Load(), prompts.Load(), agent.SessionID)
}
})
}
}
// TestMalformedResultToleratedOnlyForReadOnlyStages proves A6: a malformed final
// result is re-emitted on an explicit read-only stage (with format_repairs
// retained) but aborts immediately on a mutating or legacy stage.
func TestMalformedResultToleratedOnlyForReadOnlyStages(t *testing.T) {
malformed := `{"outcome":"success","output":{},"extra":true}`
t.Run("read-only", func(t *testing.T) {
def := modelDefinition()
def.Blocks["tools"] = "read"
def.Configs["retries"] = "1"
calls := 0
client := &repairGenerator{generate: func(context.Context, ModelConfig, string) (string, error) {
calls++
if calls == 1 {
return malformed, nil
}
return `{"outcome":"success","output":{"ok":true}}`, nil
}}
result, err := (ModelStage{Client: client}).Run(context.Background(), def, nil)
if err != nil || result.Output["ok"] != true || calls != 2 {
t.Fatalf("read-only malformed result was not tolerated: %v %v calls=%d", result, err, calls)
}
if len(client.repairs) != 1 || client.repairs[0].Accepted || client.repairs[0].Method != "validation" || client.repairs[0].Diagnostic == "" {
t.Fatalf("format_repairs evidence missing or wrong: %+v", client.repairs)
}
})
for _, tools := range []string{"edit", ""} {
t.Run("tools="+tools, func(t *testing.T) {
def := modelDefinition()
if tools != "" {
def.Blocks["tools"] = tools
}
def.Configs["retries"] = "1"
calls := 0
client := &repairGenerator{generate: func(context.Context, ModelConfig, string) (string, error) {
calls++
return malformed, nil
}}
if _, err := (ModelStage{Client: client}).Run(context.Background(), def, nil); !errors.Is(err, ErrInvalidModelResult) || calls != 1 {
t.Fatalf("mutating/legacy stage replayed a malformed result: calls=%d err=%v", calls, err)
}
})
}
}
// TestAgentTimeoutRetainsReadbackEvidenceAndSettlesUsage proves A3: a timed-out
// unbudgeted V2 agent retains one bounded redacted readback and records knowable
// usage, while a failing readback never changes the returned error.
func TestAgentTimeoutRetainsReadbackEvidenceAndSettlesUsage(t *testing.T) {
def := modelDefinition() // retries=0
def.Configs["timeout"] = "30ms"
readback := `{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"partial\":true}"}]}]}`
served := func(readbackErr bool) func(context.Context, string, ...string) ([]byte, error) {
return func(ctx context.Context, _ string, args ...string) ([]byte, error) {
path := args[2]
switch {
case path == "/api/session":
return []byte(`{"data":{"id":"ses_timeout"}}`), nil
case strings.HasSuffix(path, "/prompt"):
<-ctx.Done()
return nil, ctx.Err()
case strings.HasSuffix(path, finalAssistantReadback):
if readbackErr {
return nil, errors.New("readback unavailable")
}
return []byte(readback), nil
case path == "/api/session/ses_timeout":
return usageResponse("ses_timeout", .3, 11), nil
case strings.HasSuffix(path, "/interrupt"):
return []byte(`{}`), nil
}
return nil, fmt.Errorf("unexpected path %s", path)
}
}
t.Run("readback retained and usage settled", func(t *testing.T) {
agent := &AgentClient{OpenCodeClient: OpenCodeClient{Backend: "v2", sessionPollInterval: time.Millisecond, command: served(false)}}
budget := NewBudget(0, 0)
_, err := (ModelStage{Client: agent}).Run(WithBudget(context.Background(), budget), def, nil)
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("expected deadline failure, got %v", err)
}
if !json.Valid([]byte(agent.Transcript)) || !strings.Contains(agent.Transcript, "partial") {
t.Fatalf("timed-out readback evidence missing: %q", agent.Transcript)
}
snapshot := budget.Snapshot()
if !snapshot.USDKnown || !snapshot.TokensKnown || snapshot.UsedUSD != .3 || snapshot.UsedTokens != 11 {
t.Fatalf("timed-out usage not settled: %+v", snapshot)
}
})
t.Run("failing readback is non-fatal", func(t *testing.T) {
agent := &AgentClient{OpenCodeClient: OpenCodeClient{Backend: "v2", sessionPollInterval: time.Millisecond, command: served(true)}}
budget := NewBudget(0, 0)
_, err := (ModelStage{Client: agent}).Run(WithBudget(context.Background(), budget), def, nil)
if !errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport) {
t.Fatalf("failed readback changed the run error: %v", err)
}
if agent.Transcript != "" {
t.Fatalf("failed readback retained fabricated evidence: %q", agent.Transcript)
}
})
}
// TestWaitForSessionIdleRecordsOneTimeoutEvidence proves A3: a deadline observed
// during the inter-poll delay records exactly one sanitized evidence entry, and
// a clean settle records none.
func TestWaitForSessionIdleRecordsOneTimeoutEvidence(t *testing.T) {
log := &OperationLog{}
running := OpenCodeClient{sessionPollInterval: time.Hour, Operations: log, command: func(_ context.Context, _ string, _ ...string) ([]byte, error) {
return []byte(`{"data":{"ses_wait":{"type":"running"}}}`), nil
}}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
if err := running.waitForSessionIdle(ctx, "ses_wait"); !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("deadline not reported: %v", err)
}
if len(log.Failures) != 1 {
t.Fatalf("expected exactly one timeout evidence, got %d", len(log.Failures))
}
entry := log.Failures[0]
if entry.Operation != "wait for session idle" || !strings.Contains(entry.Diagnostic, "deadline reached") {
t.Fatalf("timeout evidence not sanitized as expected: %+v", entry)
}
clean := &OperationLog{}
settled := OpenCodeClient{sessionPollInterval: time.Hour, Operations: clean, command: func(_ context.Context, _ string, _ ...string) ([]byte, error) {
return []byte(`{"data":{}}`), nil
}}
if err := settled.waitForSessionIdle(context.Background(), "ses_wait"); err != nil {
t.Fatal(err)
}
if len(clean.Failures) != 0 {
t.Fatalf("clean settle recorded evidence: %+v", clean.Failures)
}
}
// TestTimedOutTranscriptEnvelopeRetainsSessionAndLimits proves the runner's
// failure-transcript composition (BudgetTranscript -> RetainSession ->
// BoundTranscript) yields a valid, bounded JSON object that keeps both the
// session pointer and the stage limits.
func TestTimedOutTranscriptEnvelopeRetainsSessionAndLimits(t *testing.T) {
evidence := `{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"partial\":true}"}]}]}`
composed := BoundTranscript(RetainSession(BudgetTranscript(evidence, NewBudget(0, 0), time.Hour, 2), "ses_timeout"), "ses_timeout")
var data map[string]json.RawMessage
if err := json.Unmarshal([]byte(composed), &data); err != nil {
t.Fatal(err)
}
for _, key := range []string{"retained_session", "stage_limits"} {
if _, ok := data[key]; !ok {
t.Fatalf("composed envelope missing %s: %s", key, composed)
}
}
if len(composed) > transcriptBound {
t.Fatalf("composed envelope exceeds bound: %d", len(composed))
}
if strings.Contains(composed, "Bearer") {
t.Fatal("composed envelope leaked a credential")
}
}
func TestRetainSessionAddsPointerToJSONObjectOnly(t *testing.T) {
if got := RetainSession("", "ses_1"); got != "" {
t.Fatalf("empty transcript was wrapped: %q", got)
}
if got := RetainSession("plain text", "ses_1"); got != "plain text" {
t.Fatalf("non-object transcript was wrapped: %q", got)
}
if got := RetainSession(`{"a":1}`, ""); got != `{"a":1}` {
t.Fatalf("empty session id changed the transcript: %q", got)
}
got := RetainSession(`{"stage_limits":{"attempts":1}}`, "ses_1")
var data map[string]any
if err := json.Unmarshal([]byte(got), &data); err != nil {
t.Fatal(err)
}
if data["retained_session"] != "ses_1" || data["stage_limits"] == nil {
t.Fatalf("retained session/limits lost: %v", data)
}
}
+18
View File
@@ -35,6 +35,24 @@ func ToolNames(def Definition) ([]string, bool, error) {
return tools, true, nil
}
// safeToReplay is the single definition of which post-admission work may be
// automatically retried. Only an explicit tools block that selects neither
// "edit" nor "shell" is side-effect free. An absent block is the legacy
// read/glob/grep/edit/shell bundle and is treated as mutating, matching the
// runner's useAgent/permissionRules semantics.
func safeToReplay(def Definition) bool {
tools, explicit, err := ToolNames(def)
if err != nil || !explicit {
return false
}
for _, tool := range tools {
if tool == "edit" || tool == "shell" {
return false
}
}
return true
}
func (c *AgentClient) permissionRules() []map[string]string {
rules := []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}}
tools := c.Tools