Make agent-stage retries safe and retain timeout failure evidence
Fix four stage failure-handling and observability defects (A2, A3, A6, A9).
- A2: add safeToReplay(def) as the single definition of retryable post-admission
work - only an explicit tools block that selects neither edit nor shell. The
agent/pi WorkStarted guards now break only when the stage is not safe to
replay, so read-only stages genuinely retry a post-admission timeout while
mutating and legacy-bundle stages still never replay admitted work.
- A6: re-emit a malformed final result on an explicit read-only stage up to the
retry budget, preserving format_repairs evidence; mutating stages keep the
immediate return and their session.
- A3: timed-out stages retain usable evidence. waitForSessionIdle records one
bounded sanitized OperationEvidence on deadline; the agent deferred cleanup
does a bounded best-effort latest-assistant readback and settles knowable V2
usage; RetainSession adds a retained_session pointer to failed step
transcripts so the session is visible beneath the BoundTranscript threshold.
- A9: a stage outcome routed to #failure now yields a non-empty run diagnostic
("stage <name> returned outcome <outcome>") while preferring an existing step
diagnostic.
Adds focused stage and server tests for retry safety, malformed-result
re-emission, timeout evidence retention and run-level diagnostics; updates the
packaged skill references and docs/cli.md, docs/model-results.md.
This commit is contained in:
+6
-3
@@ -396,9 +396,12 @@ Usage is polled during execution and read again afterward; an in-flight request
|
||||
may exceed a limit before usage arrives. These are not provider-side hard caps.
|
||||
|
||||
Automatic retries cover retryable transport failures and per-attempt timeouts.
|
||||
They do not replay admitted tool work, invalid results, or budget failures.
|
||||
Cleanup must be confirmed before another attempt. Existing explicit `retries=0`
|
||||
settings remain unchanged. The old 30-minute run deadline no longer truncates
|
||||
An explicit read-only stage (a tools block listing neither `edit` nor `shell`)
|
||||
may also retry a post-admission deadline or transport failure and re-emit a
|
||||
malformed final result up to `retries`; a stage whose tools include `edit`/`shell`
|
||||
or that has no tools block never replays admitted tool work or invalid results.
|
||||
Budget failures always stop. Cleanup must be confirmed before another attempt.
|
||||
Existing explicit `retries=0` settings remain unchanged. The old 30-minute run deadline no longer truncates
|
||||
stage timeouts; the 30-step and five-visits-per-stage limits still apply.
|
||||
Native ntfy stages also accept `timeout` (default `1h`), but never automatically
|
||||
retry potentially delivered notifications and do not accept model budgets.
|
||||
|
||||
@@ -47,7 +47,10 @@ Format recovery has a separate bounded budget, even with `retries=0`:
|
||||
If recovery fails, report the original precise error and repair failure; never
|
||||
rerun project edits, commands, or the entire stage to repair final formatting.
|
||||
Configured execution retries remain for retryable text-generation transport
|
||||
errors; audited tool-enabled execution is never automatically replayed.
|
||||
errors; an explicit read-only stage may additionally re-emit a malformed final
|
||||
result and retry a post-admission deadline or transport failure up to `retries`.
|
||||
A stage whose tools include `edit`/`shell`, or that has no tools block (the legacy
|
||||
read/glob/grep/edit/shell bundle), never has admitted tool work replayed.
|
||||
Repair records include the original/candidate response, parsing diagnostic,
|
||||
method and acceptance flag in the bounded transcript preview. `pipelines
|
||||
inspect <run-id> --json` and the TUI evidence section expose these records.
|
||||
@@ -86,7 +89,9 @@ failure, and vice versa. Neither label includes a provider response body, prompt
|
||||
credential or arbitrary model content. Duplicate nested fields are rejected by
|
||||
strict parsing rather than merged or last-wins, and only a lossless insertion of
|
||||
missing commas/colons or trailing closers is repaired, so an unknown evidence or
|
||||
verdict value can never become `success` and admitted tool work is never replayed.
|
||||
verdict value can never become `success`, and admitted tool work on a mutating or
|
||||
legacy-bundle stage is never replayed (only an explicit read-only stage may
|
||||
re-emit a malformed result within `retries`).
|
||||
|
||||
Confirmed ordinary-endpoint failures (2026-10-11): an all-history readback returned
|
||||
exactly 262144 bytes of incomplete JSON (`invalid agent messages`); a reviewer
|
||||
|
||||
@@ -56,9 +56,11 @@ authentication and inference require `pipelines provider-check` on the execution
|
||||
computer/container. Optional `retries` defaults to 2 (three total attempts),
|
||||
and accepts integers 0–100; explicit 0 disables execution retries. Optional
|
||||
`timeout` defaults to `1h` per attempt and accepts positive durations such as
|
||||
`30m` or `90s`. Automatic retries only cover retryable transport failures and
|
||||
timeouts; admitted tool work is never replayed automatically. Optional
|
||||
`budget_usd=1.50` and `budget_tokens=50000` independently cap reported usage.
|
||||
`30m` or `90s`. Automatic retries cover retryable transport failures and
|
||||
timeouts; an explicit read-only stage may additionally retry a post-admission
|
||||
deadline or transport failure, while a stage with `edit`/`shell` (or no tools
|
||||
block) never replays admitted work. Optional `budget_usd=1.50` and
|
||||
`budget_tokens=50000` independently cap reported usage.
|
||||
They accumulate across attempts, format repair sessions, and repeated visits
|
||||
to this stage in the same run. Omit either budget to leave that resource unlimited.
|
||||
Both budgets must be positive; tokens include input, output, reasoning and cache
|
||||
@@ -104,7 +106,10 @@ permission policy. Keep ordinary workflows language-neutral and unrestricted
|
||||
within their run directory unless the user asks for a narrower scope.
|
||||
|
||||
Malformed final JSON has bounded, syntax-only recovery even with `retries=0`;
|
||||
it never replays stage tools or changes verdict/evidence. Compatible native stable
|
||||
it never changes verdict/evidence. On an explicit read-only stage a rejected
|
||||
malformed result may be re-emitted up to `retries` with its `format_repairs`
|
||||
evidence retained; a stage with `edit`/`shell` or no tools block never replays
|
||||
stage tools or re-emits the result. Compatible native stable
|
||||
backends request schema output with schema retries disabled; a retained deny-all,
|
||||
no-inference compatibility probe identifies known format incompatibility. Incompatible
|
||||
native backends and V2 require strict local envelope validation. Do not promise
|
||||
|
||||
@@ -196,5 +196,10 @@ VALIDATION` names the exact duplicate-field/syntax detail and byte offset for a
|
||||
strictly invalid final result; `HARNESS TRANSPORT` covers request, active-session
|
||||
poll and final-readback failures. A validation failure never re-prompts the
|
||||
admitted tool-using stage: completed edits and tool side effects are retained and
|
||||
not undone, so a full retry repeats work. Neither category exposes provider/model
|
||||
bodies or credentials.
|
||||
not undone, so a full retry repeats work. An explicit read-only stage is the one
|
||||
exception: a malformed result there mutates nothing, so it may be re-emitted up
|
||||
to `retries` with its `format_repairs` evidence retained, and a post-admission
|
||||
deadline or transport failure may be retried. A stage whose tools include
|
||||
`edit`/`shell`, or that has no tools block (the legacy read/glob/grep/edit/shell
|
||||
bundle), is never replayed. Neither category exposes provider/model bodies or
|
||||
credentials.
|
||||
|
||||
@@ -94,6 +94,20 @@ field is sent. A review-only fresh execution is a future recovery idea, **not
|
||||
implemented**. Onboarding UI visual approval remains unresolved and must not be
|
||||
reported complete.
|
||||
|
||||
## Failure-handling source changes: 2026-10-15
|
||||
|
||||
Source-level, mocked `internal/stage` and `internal/server` regression coverage
|
||||
(not live-provider evidence): an explicit read-only tools stage now actually
|
||||
retries a post-admission stage deadline or harness transport failure, creating a
|
||||
fresh bounded session per attempt, while a stage with `edit`/`shell` or no tools
|
||||
block still never replays admitted work; a malformed final result is re-emitted
|
||||
up to `retries` only on such a read-only stage with its `format_repairs` evidence
|
||||
retained. A timed-out agent stage settles knowable V2 usage and retains one
|
||||
bounded, redacted latest-assistant readback plus a `retained_session` pointer in
|
||||
its stored transcript. A declared outcome routed to `#failure` with no step
|
||||
diagnostic now yields a run diagnostic like `stage <key> returned outcome
|
||||
"blocked"`; an existing step diagnostic is still used verbatim.
|
||||
|
||||
## Dated source-level coverage: 2026-10-11
|
||||
|
||||
Source-level/mocked regression coverage (not live-provider evidence):
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/pipeline"
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/stage"
|
||||
)
|
||||
|
||||
// TestFailureRouteRunDiagnostic proves A9: a declared outcome routed to #failure
|
||||
// with an empty step diagnostic still yields a meaningful run diagnostic, while
|
||||
// an existing step diagnostic is preferred verbatim.
|
||||
func TestFailureRouteRunDiagnostic(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "runs.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
source := strings.Replace(pipeline.Template([]string{"worker"}, true), "success -> #success", "success -> #success\nblocked -> #failure", 1)
|
||||
if _, err := store.Submit(ctx, "diagnostic", source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := store.List(ctx)
|
||||
r, err := newRunner(ctx, store)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
|
||||
return stage.Result{Outcome: "blocked", Output: stage.Values{}}, nil
|
||||
}
|
||||
run, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "blocked", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.wg.Wait()
|
||||
detail, err := store.GetRun(ctx, run.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if detail.Status != "failed" || detail.Outcome != "error" || !strings.Contains(detail.Diagnostic, `stage worker returned outcome "blocked"`) {
|
||||
t.Fatalf("empty run diagnostic not replaced: %+v", detail)
|
||||
}
|
||||
|
||||
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
|
||||
return stage.Result{}, errors.New("fixture execution failed")
|
||||
}
|
||||
run, err = r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "execution", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.wg.Wait()
|
||||
detail, err = store.GetRun(ctx, run.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if detail.Status != "failed" || detail.Diagnostic != "fixture execution failed" {
|
||||
t.Fatalf("existing step diagnostic not preferred: %+v", detail)
|
||||
}
|
||||
}
|
||||
@@ -473,6 +473,11 @@ func (r *runner) run(ctx context.Context, id int64, d *protocol.PipelineDetail,
|
||||
step.Outcome = "error"
|
||||
step.Output = stage.Values{"error": err.Error(), "stage": key, "input": input}
|
||||
}
|
||||
// A failed or timed-out step carries the session pointer even when the
|
||||
// bounded transcript would otherwise be only its stage_limits envelope.
|
||||
if err != nil && stageSession != "" {
|
||||
step.Transcript = stage.RetainSession(step.Transcript, stageSession)
|
||||
}
|
||||
step.Target = def.Edges[step.Outcome]
|
||||
step.Control = &protocol.ControlEvidence{Visit: visits[key], MaxVisits: maxVisits, Memory: reads}
|
||||
if err == nil && selected.Limits != nil && visits[key] >= maxVisits {
|
||||
@@ -526,7 +531,11 @@ func (r *runner) run(ctx context.Context, id int64, d *protocol.PipelineDetail,
|
||||
return
|
||||
}
|
||||
if step.Target == protocol.FailureEnd {
|
||||
finish("failed", "error", step.Diagnostic)
|
||||
diagnostic := step.Diagnostic
|
||||
if diagnostic == "" {
|
||||
diagnostic = fmt.Sprintf("stage %s returned outcome %q", step.Stage, step.Outcome)
|
||||
}
|
||||
finish("failed", "error", diagnostic)
|
||||
return
|
||||
}
|
||||
if step.Target == "" {
|
||||
|
||||
@@ -93,6 +93,7 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
|
||||
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 15*time.Second)
|
||||
defer cancel()
|
||||
if err != nil {
|
||||
c.captureTimedOutEvidence(ctx, err, config, watch)
|
||||
if stopErr := c.Interrupt(cleanup, c.SessionID); stopErr != nil {
|
||||
err = fmt.Errorf("agent interrupt failed; retained session %s needs inspection: %w", c.SessionID, errors.Join(err, stopErr, ErrStageCleanup))
|
||||
return
|
||||
@@ -185,6 +186,27 @@ func (c OpenCodeClient) settleV2Usage(ctx context.Context, budget *Budget, id st
|
||||
_ = budget.readUsage(id, data, false)
|
||||
}
|
||||
|
||||
// captureTimedOutEvidence retains one bounded, redacted best-effort readback of
|
||||
// the latest assistant message after a post-admission deadline or transport
|
||||
// failure, and settles knowable V2 usage for an unbudgeted stage. A failed
|
||||
// readback is ignored and never changes the returned error.
|
||||
func (c *AgentClient) captureTimedOutEvidence(ctx context.Context, err error, config ModelConfig, watch *budgetWatch) {
|
||||
if !c.WorkStarted || c.SessionID == "" || !(errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport)) {
|
||||
return
|
||||
}
|
||||
readCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
|
||||
data, readErr := c.call(readCtx, "get", "/api/session/"+c.SessionID+finalAssistantReadback, nil)
|
||||
cancel()
|
||||
if readErr == nil && len(data) > 0 {
|
||||
c.Transcript = RedactEvidence(string(data))
|
||||
}
|
||||
// A budgeted stage already settles through watch.finish; only an unbudgeted
|
||||
// V2 stage needs this bounded aggregate read.
|
||||
if watch == nil && c.Backend == "v2" {
|
||||
c.settleV2Usage(ctx, config.Budget, c.SessionID)
|
||||
}
|
||||
}
|
||||
|
||||
// finalAssistantReadback is the single bounded request used to select a V2
|
||||
// final result: only the latest assistant message. Fetching the full assistant
|
||||
// history also pulls large prior tool-read contents and can exceed the response
|
||||
|
||||
@@ -234,6 +234,30 @@ func BoundTranscript(transcript, sessionID string) string {
|
||||
return string(encoded)
|
||||
}
|
||||
|
||||
// RetainSession adds a retained_session pointer to an already-JSON-object
|
||||
// transcript so a failed or timed-out step carries the session ID even below the
|
||||
// 32 KiB BoundTranscript threshold. Non-object and empty transcripts are
|
||||
// returned unchanged and the value is JSON-encoded, so the result stays valid.
|
||||
func RetainSession(transcript, sessionID string) string {
|
||||
if sessionID == "" {
|
||||
return transcript
|
||||
}
|
||||
var data map[string]json.RawMessage
|
||||
if json.Unmarshal([]byte(transcript), &data) != nil || data == nil {
|
||||
return transcript
|
||||
}
|
||||
encoded, err := json.Marshal(sessionID)
|
||||
if err != nil {
|
||||
return transcript
|
||||
}
|
||||
data["retained_session"] = encoded
|
||||
merged, err := json.Marshal(data)
|
||||
if err != nil {
|
||||
return transcript
|
||||
}
|
||||
return string(merged)
|
||||
}
|
||||
|
||||
func (l *OperationLog) record(e OperationEvidence, started time.Time) {
|
||||
if l == nil {
|
||||
return
|
||||
|
||||
+59
-23
@@ -59,6 +59,26 @@ func wrapModelStageFailure(key string, err error) error {
|
||||
}
|
||||
}
|
||||
|
||||
// replayablePostAdmission reports whether a post-admission error can be retried
|
||||
// from the start without risking a repeated tool mutation: a stage context
|
||||
// deadline or a harness transport failure. A structured-output validation
|
||||
// failure is classified separately as ErrInvalidModelResult.
|
||||
func replayablePostAdmission(err error) bool {
|
||||
return errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport)
|
||||
}
|
||||
|
||||
// toleratedModelResultRetry reports whether a malformed final result may be
|
||||
// re-emitted. It requires an explicit read-only stage, remaining attempts,
|
||||
// live work, available budget and a plain validation error; fail-closed
|
||||
// accounting and cleanup errors still stop immediately.
|
||||
func toleratedModelResultRetry(ctx context.Context, def Definition, config ModelConfig, attempt int, resultErr error) bool {
|
||||
return errors.Is(resultErr, ErrInvalidModelResult) &&
|
||||
!errors.Is(resultErr, ErrBudgetExceeded) &&
|
||||
!errors.Is(resultErr, ErrBudgetAccounting) &&
|
||||
!errors.Is(resultErr, ErrStageCleanup) &&
|
||||
safeToReplay(def) && attempt < config.Retries && ctx.Err() == nil && config.Budget != nil && config.Budget.Check() == nil
|
||||
}
|
||||
|
||||
var providerID = regexp.MustCompile(`^[A-Za-z0-9_.-]+$`)
|
||||
var modelID = regexp.MustCompile(`^[A-Za-z0-9_./:-]+$`)
|
||||
|
||||
@@ -208,8 +228,9 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
|
||||
return Result{}, fmt.Errorf("%w: generator cannot enforce usage limits", ErrBudgetAccounting)
|
||||
}
|
||||
}
|
||||
// Replaying admitted tool work can repeat mutations. Only pre-admission
|
||||
// transport errors may automatically retry an audited agent.
|
||||
// Replaying admitted tool work can repeat mutations. Only a stage whose
|
||||
// explicit tools cannot mutate the run directory (safeToReplay) may
|
||||
// automatically retry work that started after admission.
|
||||
if agent, ok := client.(*AgentClient); ok {
|
||||
if tools, explicit, _ := ToolNames(def); explicit {
|
||||
agent.Tools = tools
|
||||
@@ -232,6 +253,7 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
|
||||
if generateErr == nil && attemptCtx.Err() != nil {
|
||||
generateErr = attemptCtx.Err()
|
||||
}
|
||||
replayPostAdmission := false
|
||||
if generateErr == nil {
|
||||
result, resultErr := decodeModelResult(stageResultText(text), def.Edges)
|
||||
if resultErr != nil {
|
||||
@@ -241,27 +263,41 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
|
||||
resultErr = errors.Join(resultErr, attemptCtx.Err())
|
||||
}
|
||||
cancel()
|
||||
return result, wrapModelStageFailure(def.Key, resultErr)
|
||||
}
|
||||
err = generateErr
|
||||
if attemptCtx.Err() != nil && !errors.Is(err, ErrBudgetExceeded) && !errors.Is(err, ErrBudgetAccounting) {
|
||||
err = errors.Join(err, attemptCtx.Err())
|
||||
}
|
||||
cancel()
|
||||
if ctx.Err() != nil {
|
||||
return Result{}, ctx.Err()
|
||||
}
|
||||
if errors.Is(err, ErrBudgetExceeded) || errors.Is(err, ErrBudgetAccounting) || errors.Is(err, ErrStageCleanup) {
|
||||
break
|
||||
}
|
||||
if agent, ok := client.(*AgentClient); ok && agent.WorkStarted {
|
||||
break
|
||||
}
|
||||
if pi, ok := client.(*PiClient); ok && pi.WorkStarted {
|
||||
break
|
||||
}
|
||||
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded)) {
|
||||
break
|
||||
if resultErr == nil {
|
||||
return result, nil
|
||||
}
|
||||
// A malformed final result mutates nothing on an explicit read-only
|
||||
// stage, so it may be re-emitted within the retry budget. A mutating
|
||||
// or legacy-bundle stage retains its admitted work and returns now.
|
||||
if toleratedModelResultRetry(ctx, def, spec.Config, attempt, resultErr) {
|
||||
err = resultErr
|
||||
} else {
|
||||
return result, wrapModelStageFailure(def.Key, resultErr)
|
||||
}
|
||||
} else {
|
||||
err = generateErr
|
||||
if attemptCtx.Err() != nil && !errors.Is(err, ErrBudgetExceeded) && !errors.Is(err, ErrBudgetAccounting) {
|
||||
err = errors.Join(err, attemptCtx.Err())
|
||||
}
|
||||
cancel()
|
||||
if ctx.Err() != nil {
|
||||
return Result{}, ctx.Err()
|
||||
}
|
||||
if errors.Is(err, ErrBudgetExceeded) || errors.Is(err, ErrBudgetAccounting) || errors.Is(err, ErrStageCleanup) {
|
||||
break
|
||||
}
|
||||
replayPostAdmission = safeToReplay(def) && replayablePostAdmission(err)
|
||||
// Never blindly replay admitted tool work: only a read-only stage may
|
||||
// retry a post-admission deadline or transport failure.
|
||||
if agent, ok := client.(*AgentClient); ok && agent.WorkStarted && !replayPostAdmission {
|
||||
break
|
||||
}
|
||||
if pi, ok := client.(*PiClient); ok && pi.WorkStarted && !replayPostAdmission {
|
||||
break
|
||||
}
|
||||
if attempt == spec.Config.Retries || (!retryableModelError(err) && !errors.Is(err, context.DeadlineExceeded) && !replayPostAdmission) {
|
||||
break
|
||||
}
|
||||
}
|
||||
delay := time.Duration(1<<min(attempt, 4)) * 100 * time.Millisecond
|
||||
timer := time.NewTimer(delay)
|
||||
|
||||
@@ -196,6 +196,7 @@ func (c OpenCodeClient) waitForSessionIdle(ctx context.Context, sessionID string
|
||||
if interval <= 0 {
|
||||
interval = defaultSessionPollInterval
|
||||
}
|
||||
started := time.Now()
|
||||
for {
|
||||
active, err := c.sessionActive(ctx, sessionID)
|
||||
if err != nil {
|
||||
@@ -208,6 +209,16 @@ func (c OpenCodeClient) waitForSessionIdle(ctx context.Context, sessionID string
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
timer.Stop()
|
||||
// A settled session is the only success signal; retain one bounded
|
||||
// sanitized record so a timed-out wait is not an empty transcript.
|
||||
c.Operations.record(OperationEvidence{
|
||||
Operation: "wait for session idle",
|
||||
Backend: c.Backend,
|
||||
Executable: opencode.Executable(),
|
||||
Directory: c.Directory,
|
||||
ExitCode: -1,
|
||||
Diagnostic: "stage deadline reached while waiting for the agent session to become idle",
|
||||
}, started)
|
||||
return ctx.Err()
|
||||
case <-timer.C:
|
||||
}
|
||||
|
||||
@@ -0,0 +1,311 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// repairGenerator is a generatorFunc that also records format repairs, so a
|
||||
// toleration test can assert the retained format_repairs evidence.
|
||||
type repairGenerator struct {
|
||||
generate func(context.Context, ModelConfig, string) (string, error)
|
||||
repairs []FormatRepair
|
||||
}
|
||||
|
||||
func (g *repairGenerator) Generate(ctx context.Context, config ModelConfig, prompt string) (string, error) {
|
||||
return g.generate(ctx, config, prompt)
|
||||
}
|
||||
|
||||
func (g *repairGenerator) RecordFormatRepair(repair FormatRepair) {
|
||||
g.repairs = append(g.repairs, repair)
|
||||
}
|
||||
|
||||
func TestSafeToReplayRequiresExplicitReadOnlyTools(t *testing.T) {
|
||||
legacy := modelDefinition()
|
||||
if safeToReplay(legacy) {
|
||||
t.Fatal("absent tools block (legacy bundle) must not be replayable")
|
||||
}
|
||||
mutating := modelDefinition()
|
||||
mutating.Blocks["tools"] = "read\nedit"
|
||||
if safeToReplay(mutating) {
|
||||
t.Fatal("edit-capable stage must not be replayable")
|
||||
}
|
||||
shell := modelDefinition()
|
||||
shell.Blocks["tools"] = "shell"
|
||||
if safeToReplay(shell) {
|
||||
t.Fatal("shell-capable stage must not be replayable")
|
||||
}
|
||||
readonly := modelDefinition()
|
||||
readonly.Blocks["tools"] = "read\nglob\ngrep\nwebsearch\nwebfetch"
|
||||
if !safeToReplay(readonly) {
|
||||
t.Fatal("explicit read-only stage should be replayable")
|
||||
}
|
||||
empty := modelDefinition()
|
||||
empty.Blocks["tools"] = ""
|
||||
if !safeToReplay(empty) {
|
||||
t.Fatal("explicit empty tools block is side-effect free")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadOnlyAgentRetriesPostAdmissionDeadline proves A2: an explicit
|
||||
// read-only agent stage retries a post-admission deadline on attempt 2 with a
|
||||
// fresh bounded session, and the shared budget records both attempts.
|
||||
func TestReadOnlyAgentRetriesPostAdmissionDeadline(t *testing.T) {
|
||||
def := modelDefinition()
|
||||
def.Blocks["tools"] = "read"
|
||||
def.Configs["retries"] = "1"
|
||||
def.Configs["timeout"] = "30ms"
|
||||
var sessions, prompts atomic.Int64
|
||||
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(ctx context.Context, _ string, args ...string) ([]byte, error) {
|
||||
path := args[2]
|
||||
switch {
|
||||
case path == "/api/session":
|
||||
n := sessions.Add(1)
|
||||
return []byte(fmt.Sprintf(`{"data":{"id":"ses_read_%d"}}`, n)), nil
|
||||
case strings.HasSuffix(path, "/prompt"):
|
||||
if prompts.Add(1) == 1 {
|
||||
<-ctx.Done()
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
return []byte(`{}`), nil
|
||||
case path == "/api/session/active":
|
||||
return []byte(`{"data":{}}`), nil
|
||||
case strings.HasSuffix(path, finalAssistantReadback):
|
||||
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"outcome\":\"success\",\"output\":{\"summary\":\"done\"}}"}]}]}`), nil
|
||||
case strings.HasSuffix(path, "/interrupt"):
|
||||
return []byte(`{}`), nil
|
||||
}
|
||||
return nil, fmt.Errorf("unexpected path %s", path)
|
||||
}}}
|
||||
budget := NewBudget(0, 0)
|
||||
result, err := (ModelStage{Client: agent}).Run(WithBudget(context.Background(), budget), def, nil)
|
||||
if err != nil || result.Output["summary"] != "done" {
|
||||
t.Fatalf("read-only deadline was not retried to success: %v %v", result, err)
|
||||
}
|
||||
if sessions.Load() != 2 || prompts.Load() != 2 || budget.Snapshot().Attempts != 2 {
|
||||
t.Fatalf("expected two fresh attempts, got sessions=%d prompts=%d attempts=%d", sessions.Load(), prompts.Load(), budget.Snapshot().Attempts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMutatingAgentNeverRetriesPostAdmissionDeadline proves the narrowed
|
||||
// guard: a mutating stage retains its admitted session and never replays.
|
||||
func TestMutatingAgentNeverRetriesPostAdmissionDeadline(t *testing.T) {
|
||||
for _, tools := range []string{"edit", "read\nshell", ""} {
|
||||
t.Run("tools="+tools, func(t *testing.T) {
|
||||
def := modelDefinition()
|
||||
if tools != "" {
|
||||
def.Blocks["tools"] = tools
|
||||
}
|
||||
def.Configs["retries"] = "2"
|
||||
def.Configs["timeout"] = "30ms"
|
||||
var sessions, prompts atomic.Int64
|
||||
agent := &AgentClient{OpenCodeClient: OpenCodeClient{command: func(ctx context.Context, _ string, args ...string) ([]byte, error) {
|
||||
path := args[2]
|
||||
switch {
|
||||
case path == "/api/session":
|
||||
n := sessions.Add(1)
|
||||
return []byte(fmt.Sprintf(`{"data":{"id":"ses_mut_%d"}}`, n)), nil
|
||||
case strings.HasSuffix(path, "/prompt"):
|
||||
prompts.Add(1)
|
||||
<-ctx.Done()
|
||||
return nil, ctx.Err()
|
||||
case strings.HasSuffix(path, finalAssistantReadback):
|
||||
return []byte(`{"data":[{"finish":"stop","content":[{"type":"text","text":"{}"}]}]}`), nil
|
||||
case strings.HasSuffix(path, "/interrupt"):
|
||||
return []byte(`{}`), nil
|
||||
}
|
||||
return nil, fmt.Errorf("unexpected path %s", path)
|
||||
}}}
|
||||
_, err := (ModelStage{Client: agent}).Run(context.Background(), def, nil)
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("expected deadline failure, got %v", err)
|
||||
}
|
||||
if sessions.Load() != 1 || prompts.Load() != 1 || agent.SessionID != "ses_mut_1" {
|
||||
t.Fatalf("mutating stage replayed admitted work: sessions=%d prompts=%d session=%q", sessions.Load(), prompts.Load(), agent.SessionID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestMalformedResultToleratedOnlyForReadOnlyStages proves A6: a malformed final
|
||||
// result is re-emitted on an explicit read-only stage (with format_repairs
|
||||
// retained) but aborts immediately on a mutating or legacy stage.
|
||||
func TestMalformedResultToleratedOnlyForReadOnlyStages(t *testing.T) {
|
||||
malformed := `{"outcome":"success","output":{},"extra":true}`
|
||||
t.Run("read-only", func(t *testing.T) {
|
||||
def := modelDefinition()
|
||||
def.Blocks["tools"] = "read"
|
||||
def.Configs["retries"] = "1"
|
||||
calls := 0
|
||||
client := &repairGenerator{generate: func(context.Context, ModelConfig, string) (string, error) {
|
||||
calls++
|
||||
if calls == 1 {
|
||||
return malformed, nil
|
||||
}
|
||||
return `{"outcome":"success","output":{"ok":true}}`, nil
|
||||
}}
|
||||
result, err := (ModelStage{Client: client}).Run(context.Background(), def, nil)
|
||||
if err != nil || result.Output["ok"] != true || calls != 2 {
|
||||
t.Fatalf("read-only malformed result was not tolerated: %v %v calls=%d", result, err, calls)
|
||||
}
|
||||
if len(client.repairs) != 1 || client.repairs[0].Accepted || client.repairs[0].Method != "validation" || client.repairs[0].Diagnostic == "" {
|
||||
t.Fatalf("format_repairs evidence missing or wrong: %+v", client.repairs)
|
||||
}
|
||||
})
|
||||
for _, tools := range []string{"edit", ""} {
|
||||
t.Run("tools="+tools, func(t *testing.T) {
|
||||
def := modelDefinition()
|
||||
if tools != "" {
|
||||
def.Blocks["tools"] = tools
|
||||
}
|
||||
def.Configs["retries"] = "1"
|
||||
calls := 0
|
||||
client := &repairGenerator{generate: func(context.Context, ModelConfig, string) (string, error) {
|
||||
calls++
|
||||
return malformed, nil
|
||||
}}
|
||||
if _, err := (ModelStage{Client: client}).Run(context.Background(), def, nil); !errors.Is(err, ErrInvalidModelResult) || calls != 1 {
|
||||
t.Fatalf("mutating/legacy stage replayed a malformed result: calls=%d err=%v", calls, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestAgentTimeoutRetainsReadbackEvidenceAndSettlesUsage proves A3: a timed-out
|
||||
// unbudgeted V2 agent retains one bounded redacted readback and records knowable
|
||||
// usage, while a failing readback never changes the returned error.
|
||||
func TestAgentTimeoutRetainsReadbackEvidenceAndSettlesUsage(t *testing.T) {
|
||||
def := modelDefinition() // retries=0
|
||||
def.Configs["timeout"] = "30ms"
|
||||
readback := `{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"partial\":true}"}]}]}`
|
||||
served := func(readbackErr bool) func(context.Context, string, ...string) ([]byte, error) {
|
||||
return func(ctx context.Context, _ string, args ...string) ([]byte, error) {
|
||||
path := args[2]
|
||||
switch {
|
||||
case path == "/api/session":
|
||||
return []byte(`{"data":{"id":"ses_timeout"}}`), nil
|
||||
case strings.HasSuffix(path, "/prompt"):
|
||||
<-ctx.Done()
|
||||
return nil, ctx.Err()
|
||||
case strings.HasSuffix(path, finalAssistantReadback):
|
||||
if readbackErr {
|
||||
return nil, errors.New("readback unavailable")
|
||||
}
|
||||
return []byte(readback), nil
|
||||
case path == "/api/session/ses_timeout":
|
||||
return usageResponse("ses_timeout", .3, 11), nil
|
||||
case strings.HasSuffix(path, "/interrupt"):
|
||||
return []byte(`{}`), nil
|
||||
}
|
||||
return nil, fmt.Errorf("unexpected path %s", path)
|
||||
}
|
||||
}
|
||||
t.Run("readback retained and usage settled", func(t *testing.T) {
|
||||
agent := &AgentClient{OpenCodeClient: OpenCodeClient{Backend: "v2", sessionPollInterval: time.Millisecond, command: served(false)}}
|
||||
budget := NewBudget(0, 0)
|
||||
_, err := (ModelStage{Client: agent}).Run(WithBudget(context.Background(), budget), def, nil)
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("expected deadline failure, got %v", err)
|
||||
}
|
||||
if !json.Valid([]byte(agent.Transcript)) || !strings.Contains(agent.Transcript, "partial") {
|
||||
t.Fatalf("timed-out readback evidence missing: %q", agent.Transcript)
|
||||
}
|
||||
snapshot := budget.Snapshot()
|
||||
if !snapshot.USDKnown || !snapshot.TokensKnown || snapshot.UsedUSD != .3 || snapshot.UsedTokens != 11 {
|
||||
t.Fatalf("timed-out usage not settled: %+v", snapshot)
|
||||
}
|
||||
})
|
||||
t.Run("failing readback is non-fatal", func(t *testing.T) {
|
||||
agent := &AgentClient{OpenCodeClient: OpenCodeClient{Backend: "v2", sessionPollInterval: time.Millisecond, command: served(true)}}
|
||||
budget := NewBudget(0, 0)
|
||||
_, err := (ModelStage{Client: agent}).Run(WithBudget(context.Background(), budget), def, nil)
|
||||
if !errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrHarnessTransport) {
|
||||
t.Fatalf("failed readback changed the run error: %v", err)
|
||||
}
|
||||
if agent.Transcript != "" {
|
||||
t.Fatalf("failed readback retained fabricated evidence: %q", agent.Transcript)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestWaitForSessionIdleRecordsOneTimeoutEvidence proves A3: a deadline observed
|
||||
// during the inter-poll delay records exactly one sanitized evidence entry, and
|
||||
// a clean settle records none.
|
||||
func TestWaitForSessionIdleRecordsOneTimeoutEvidence(t *testing.T) {
|
||||
log := &OperationLog{}
|
||||
running := OpenCodeClient{sessionPollInterval: time.Hour, Operations: log, command: func(_ context.Context, _ string, _ ...string) ([]byte, error) {
|
||||
return []byte(`{"data":{"ses_wait":{"type":"running"}}}`), nil
|
||||
}}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := running.waitForSessionIdle(ctx, "ses_wait"); !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("deadline not reported: %v", err)
|
||||
}
|
||||
if len(log.Failures) != 1 {
|
||||
t.Fatalf("expected exactly one timeout evidence, got %d", len(log.Failures))
|
||||
}
|
||||
entry := log.Failures[0]
|
||||
if entry.Operation != "wait for session idle" || !strings.Contains(entry.Diagnostic, "deadline reached") {
|
||||
t.Fatalf("timeout evidence not sanitized as expected: %+v", entry)
|
||||
}
|
||||
clean := &OperationLog{}
|
||||
settled := OpenCodeClient{sessionPollInterval: time.Hour, Operations: clean, command: func(_ context.Context, _ string, _ ...string) ([]byte, error) {
|
||||
return []byte(`{"data":{}}`), nil
|
||||
}}
|
||||
if err := settled.waitForSessionIdle(context.Background(), "ses_wait"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(clean.Failures) != 0 {
|
||||
t.Fatalf("clean settle recorded evidence: %+v", clean.Failures)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTimedOutTranscriptEnvelopeRetainsSessionAndLimits proves the runner's
|
||||
// failure-transcript composition (BudgetTranscript -> RetainSession ->
|
||||
// BoundTranscript) yields a valid, bounded JSON object that keeps both the
|
||||
// session pointer and the stage limits.
|
||||
func TestTimedOutTranscriptEnvelopeRetainsSessionAndLimits(t *testing.T) {
|
||||
evidence := `{"data":[{"finish":"stop","content":[{"type":"text","text":"{\"partial\":true}"}]}]}`
|
||||
composed := BoundTranscript(RetainSession(BudgetTranscript(evidence, NewBudget(0, 0), time.Hour, 2), "ses_timeout"), "ses_timeout")
|
||||
var data map[string]json.RawMessage
|
||||
if err := json.Unmarshal([]byte(composed), &data); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, key := range []string{"retained_session", "stage_limits"} {
|
||||
if _, ok := data[key]; !ok {
|
||||
t.Fatalf("composed envelope missing %s: %s", key, composed)
|
||||
}
|
||||
}
|
||||
if len(composed) > transcriptBound {
|
||||
t.Fatalf("composed envelope exceeds bound: %d", len(composed))
|
||||
}
|
||||
if strings.Contains(composed, "Bearer") {
|
||||
t.Fatal("composed envelope leaked a credential")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetainSessionAddsPointerToJSONObjectOnly(t *testing.T) {
|
||||
if got := RetainSession("", "ses_1"); got != "" {
|
||||
t.Fatalf("empty transcript was wrapped: %q", got)
|
||||
}
|
||||
if got := RetainSession("plain text", "ses_1"); got != "plain text" {
|
||||
t.Fatalf("non-object transcript was wrapped: %q", got)
|
||||
}
|
||||
if got := RetainSession(`{"a":1}`, ""); got != `{"a":1}` {
|
||||
t.Fatalf("empty session id changed the transcript: %q", got)
|
||||
}
|
||||
got := RetainSession(`{"stage_limits":{"attempts":1}}`, "ses_1")
|
||||
var data map[string]any
|
||||
if err := json.Unmarshal([]byte(got), &data); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if data["retained_session"] != "ses_1" || data["stage_limits"] == nil {
|
||||
t.Fatalf("retained session/limits lost: %v", data)
|
||||
}
|
||||
}
|
||||
@@ -35,6 +35,24 @@ func ToolNames(def Definition) ([]string, bool, error) {
|
||||
return tools, true, nil
|
||||
}
|
||||
|
||||
// safeToReplay is the single definition of which post-admission work may be
|
||||
// automatically retried. Only an explicit tools block that selects neither
|
||||
// "edit" nor "shell" is side-effect free. An absent block is the legacy
|
||||
// read/glob/grep/edit/shell bundle and is treated as mutating, matching the
|
||||
// runner's useAgent/permissionRules semantics.
|
||||
func safeToReplay(def Definition) bool {
|
||||
tools, explicit, err := ToolNames(def)
|
||||
if err != nil || !explicit {
|
||||
return false
|
||||
}
|
||||
for _, tool := range tools {
|
||||
if tool == "edit" || tool == "shell" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (c *AgentClient) permissionRules() []map[string]string {
|
||||
rules := []map[string]string{{"action": "*", "resource": "*", "effect": "deny"}}
|
||||
tools := c.Tools
|
||||
|
||||
Reference in New Issue
Block a user