control: make advisory review findings non-blocking and bound retryable blocks

Cut the review loop's repeat cost and make a blocked stage actionable, at the
definition level only, with the runner's existing visit counting and caps.

The published maintenance, quick and focused pipelines gain a reviewer
`advisory -> next stage` edge for non-blocking notes, so an advisory-only review
continues the run instead of buying another implementer visit; `rejected ->
implementer` and its unchanged {"max_visits":3,"on":["rejected"]} limit keep a
must-fix finding looping and still terminate. The implementer gains
blocked_retryable -> implementer with {"max_visits":5,"on":["blocked_retryable"]}
for a transient environment/tooling/device problem a fresh attempt could
survive, while blocked -> #failure stays terminal. The cap is at least the
reviewer's rejection budget so the implementer's own pre-execution cap cannot
preempt the reviewer's rejected control.

No production Go code changed. New tests in internal/server/review_contract_test.go
cover advisory-only never revisiting the implementer, must-fix still looping to
the reviewer limit, the implementer cap not preempting the reviewer, retryable
blocks bounded then failing closed, a retry that recovers, and a terminal block
failing immediately; concurrency/control/human tests share the new submitRunner
and startRun seams.

Documentation checked: internal/author/skill/references/control.md and
docs/harnesses-and-control.md carry the new edges and the visit-cap rule;
no packaged template defines these pipelines (they are published immutable
versions), and docs/model-results.md needs no change because the {outcome,output}
result contract is unchanged. README and other shared docs deliberately untouched.

Verified: go build ./... and go test -count=1 ./... pass.

Residual limitation: the must-fix-vs-advisory split is carried by the prompt
contract, not a machine-readable marker, because the recorded findings have none;
a model that still returns rejected for advisory-only notes would still buy a
visit. All 18 recorded implementer blocked runs classify terminal, so the
blocked_retryable edge recovers 0 of them.
This commit is contained in:
2026-10-17 17:06:51 +01:00
parent 2ffe87cbba
commit e70c94e8dd
6 changed files with 291 additions and 34 deletions
+18
View File
@@ -75,6 +75,24 @@ For bounded review loops, add a fenced `limits` JSON object to the reviewer:
On the third rejection the server terminates at failure regardless of model
counters. Approval on that visit can still succeed. Limits never enter prompts.
A reviewer that must separate a blocking defect from advisory notes declares both
edges: `rejected -> implement` (kept, with the `limits` above) is reserved for a
must-fix finding that names a concrete location, evidence and remedy, and
`advisory -> <next stage>` carries non-blocking duplication, performance or style
notes. An advisory outcome continues the run and never buys another implementer
visit. An implementer that can tell a transient environment, tooling or device
problem from an acceptance criterion that genuinely cannot be met declares a
bounded self-edge: `blocked_retryable -> implementer`, while `blocked -> #failure`
stays terminal. `max_visits` caps every visit of that stage, including re-entries
from the reviewer's `rejected` edge, so an implementer that is also the rejection
target needs a cap at least as large as the reviewer's budget; the published
maintenance, quick and focused pipelines use
`{"max_visits":5,"on":["blocked_retryable"]}`. A smaller implementer cap stops the
loop on the implementer's pre-execution limit before the reviewer's own cap can
fire, which silently makes the reviewer's `rejected` control unreachable. The runner
already counts the visits and stops at the cap, so neither loop needs new server
code and no model counts visits.
Add a fenced `memory` map such as
`{"task":"input.prompt","checks":"planner.checks"}` to select reviewer
input without previous verdicts. Implementers can select
@@ -16,6 +16,7 @@ does not guarantee independent review: explicitly select its input fields.
```edges
approved -> #success
advisory -> #success
rejected -> implement
error -> #failure
```
@@ -24,6 +25,20 @@ error -> #failure
These blocks accompany a normal review stage's prompt/output/tools/configs.
Pipelines counts actual visits, ignoring model-generated counters. On visit three,
`approved` can succeed; `rejected` retains its verdict but routes to `#failure`.
`rejected` is for a must-fix finding with a concrete location, evidence and remedy;
`advisory` carries non-blocking notes and continues the run without another
implementer visit, so an advisory-only review never buys a full re-implementation.
An implementer may likewise declare `blocked_retryable -> implementer` for a
transient environment, tooling or device problem a fresh attempt could survive,
keeping `blocked -> #failure` for an acceptance criterion that genuinely cannot be
met. `max_visits` caps every visit of that stage, including re-entries from the
reviewer's `rejected` edge, so an implementer that is also the rejection target must
use a cap at least as large as the reviewer's budget; the published pipelines use
`{"max_visits":5,"on":["blocked_retryable"]}`. A lower implementer cap stops the loop
before the reviewer's own cap fires, making the reviewer's `rejected` control
unreachable. Both edges need only the existing visit counting and cap; neither the
reviewer nor the implementer counts visits, and no stage invents edits or checks to
appear busy.
Visit count, limit decision and original target are retained in `inspect --json`
and run metadata. `max_visits` accepts 1–5. Every `on` outcome must declare a
nonterminal edge. Without `on`, entering beyond the cap fails before execution.
+42 -23
View File
@@ -20,6 +20,46 @@ func concurrentStore(t *testing.T) (context.Context, *runner, int64) {
return newConcurrentRunner(t, pipeline.Template([]string{"one"}, true))
}
// submitRunner opens a fresh store, submits source through the parser and returns
// the runner, its store and the pipeline id. Callers own the returned store's
// lifetime and cleanup so tests can inspect persisted runs; newConcurrentRunner and
// the review-contract tests share this setup instead of copying it.
func submitRunner(t *testing.T, ctx context.Context, source string) (*runner, *pipeline.Store, int64) {
t.Helper()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatal(err)
}
if _, err := store.Submit(ctx, "test", source); err != nil {
store.Close()
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
store.Close()
t.Fatal(err)
}
r, err := newRunner(ctx, store)
if err != nil {
store.Close()
t.Fatal(err)
}
return r, store, items[0].ID
}
// startRun launches one run of a submitted pipeline and waits for it to stop,
// returning the started run detail. It is the single start helper for tests that
// only need a completed run.
func startRun(t *testing.T, r *runner, pipelineID int64, key string) *protocol.RunDetail {
t.Helper()
run, err := r.start(protocol.Message{PipelineID: pipelineID, Version: 1, RequestKey: key, Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
return run
}
// newConcurrentRunner builds the shared test store/runner around a submitted
// pipeline source. Tests that need an agent stage (useAgent true) must pass a
// source with a non-empty tools block: the default template's empty tools block
@@ -28,34 +68,13 @@ func concurrentStore(t *testing.T) (context.Context, *runner, int64) {
func newConcurrentRunner(t *testing.T, source string) (context.Context, *runner, int64) {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
store, err := pipeline.Open(filepath.Join(t.TempDir(), "concurrent.db"))
if err != nil {
cancel()
t.Fatal(err)
}
if _, err := store.Submit(ctx, "test", source); err != nil {
cancel()
store.Close()
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
cancel()
store.Close()
t.Fatal(err)
}
r, err := newRunner(ctx, store)
if err != nil {
cancel()
store.Close()
t.Fatal(err)
}
r, store, id := submitRunner(t, ctx, source)
t.Cleanup(func() {
cancel()
r.wg.Wait()
store.Close()
})
return ctx, r, items[0].ID
return ctx, r, id
}
// agentStore returns a runner whose pipeline has an agent stage (a non-empty,
+1 -5
View File
@@ -59,11 +59,7 @@ func TestRuntimeLimitAndSelectedMemory(t *testing.T) {
return stage.Result{Outcome: "rejected", Output: stage.Values{"findings": "fix layout", "ui_revision": 0}}, nil
}
}
run, err := runner.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "test", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}})
if err != nil {
t.Fatal(err)
}
runner.wg.Wait()
run := startRun(t, runner, items[0].ID, "test")
retained, err := store.GetRun(ctx, run.ID)
if err != nil {
t.Fatal(err)
+1 -6
View File
@@ -48,12 +48,7 @@ func startHumanRun(t *testing.T, ctx context.Context, r *runner, store *pipeline
}
}
}
run, err := r.start(protocol.Message{PipelineID: pipelineID, Version: 1, RequestKey: key, Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
return run
return startRun(t, r, pipelineID, key)
}
func TestHumanStageWaitsAndDecisionResumes(t *testing.T) {
+214
View File
@@ -0,0 +1,214 @@
package server
import (
"context"
"strconv"
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
)
// reviewLoopRunner submits a definition through the parser and opens a runner
// over it, returning the runner, its store, the pipeline id and a per-stage call
// counter. It reuses the shared submitRunner seam so every assertion below runs
// the real runLoop, not a copy of its setup or routing.
func reviewLoopRunner(t *testing.T, ctx context.Context, source string) (*runner, *pipeline.Store, int64, map[string]int) {
t.Helper()
r, store, id := submitRunner(t, ctx, source)
t.Cleanup(func() { store.Close() })
return r, store, id, map[string]int{}
}
func getRun(t *testing.T, store *pipeline.Store, ctx context.Context, id int64) *protocol.RunDetail {
t.Helper()
detail, err := store.GetRun(ctx, id)
if err != nil {
t.Fatal(err)
}
return detail
}
// TestReviewerAdvisoryDoesNotRevisitImplementer proves an advisory-only review
// outcome moves the run forward and never buys another worker visit, while the
// rejected edge and its limits stay declared. The advisory route is the single
// definition-level mechanism that keeps a non-blocking rejection from looping.
func TestReviewerAdvisoryDoesNotRevisitImplementer(t *testing.T) {
ctx := context.Background()
r, store, id, calls := reviewLoopRunner(t, ctx, reviewerLoopSource())
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
calls[def.Key]++
if def.Key == "review" {
return stage.Result{Outcome: "advisory", Output: stage.Values{"findings": "non-blocking nits only"}}, nil
}
return stage.Result{Outcome: "success", Output: stage.Values{"summary": def.Key}}, nil
}
run := startRun(t, r, id, "advisory")
detail := getRun(t, store, ctx, run.ID)
if calls["worker"] != 1 {
t.Fatalf("advisory review bought %d worker visits, want 1", calls["worker"])
}
if detail.Status != "succeeded" || len(detail.Steps) != 3 {
t.Fatalf("advisory route ended %s with %d steps, want succeeded with 3", detail.Status, len(detail.Steps))
}
if detail.Steps[1].Outcome != "advisory" || detail.Steps[1].Target != "commit" {
t.Fatalf("advisory step = %+v", detail.Steps[1])
}
}
// TestReviewerMustFixRejectionLoopsAndStopsAtLimit proves a rejection that names a
// real defect still routes back to the implementer and still terminates at the
// reviewer's limits cap with the retained diagnostic, so the rejection path is
// preserved. It runs the exact published combination - the implementer's own
// blocked_retryable limits block plus the reviewer's rejected -> implementer edge -
// rather than the reviewer limits alone.
func TestReviewerMustFixRejectionLoopsAndStopsAtLimit(t *testing.T) {
ctx := context.Background()
r, store, id, calls := reviewLoopRunner(t, ctx, publishedReviewLoopSource(5))
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
calls[def.Key]++
if def.Key == "reviewer" {
return stage.Result{Outcome: "rejected", Output: stage.Values{"findings": "unmet acceptance check at worker.go:12"}}, nil
}
return stage.Result{Outcome: "success", Output: stage.Values{"summary": def.Key}}, nil
}
run := startRun(t, r, id, "mustfix")
detail := getRun(t, store, ctx, run.ID)
if calls["reviewer"] != 3 || calls["implementer"] != 3 || len(detail.Steps) != 6 {
t.Fatalf("must-fix loop = calls %v, %d steps", calls, len(detail.Steps))
}
last := detail.Steps[5]
if detail.Status != "failed" || last.Outcome != "rejected" || last.Control == nil || !last.Control.LimitReached || last.Target != protocol.FailureEnd {
t.Fatalf("limit did not terminate the must-fix loop: %+v", detail)
}
if last.Diagnostic != "Pipelines stopped reviewer after 3 visits on outcome rejected" {
t.Fatalf("limit diagnostic = %q", last.Diagnostic)
}
}
// TestImplementerVisitCapMustNotPreemptReviewerLimit pins the regression the
// published shape had: an implementer limits cap below the reviewer's rejection
// budget stops the loop on the implementer's pre-execution visit cap, so the
// reviewer's third visit and its "Pipelines stopped reviewer..." diagnostic are
// unreachable. The cap must be at least the reviewer's budget (the published
// pipelines use 5) or the reviewer's rejected control is dead.
func TestImplementerVisitCapMustNotPreemptReviewerLimit(t *testing.T) {
ctx := context.Background()
r, store, id, calls := reviewLoopRunner(t, ctx, publishedReviewLoopSource(2))
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
calls[def.Key]++
if def.Key == "reviewer" {
return stage.Result{Outcome: "rejected", Output: stage.Values{"findings": "unmet acceptance check"}}, nil
}
return stage.Result{Outcome: "success", Output: stage.Values{"summary": def.Key}}, nil
}
run := startRun(t, r, id, "preempted")
detail := getRun(t, store, ctx, run.ID)
if calls["reviewer"] != 2 || calls["implementer"] != 2 || len(detail.Steps) != 4 {
t.Fatalf("implementer cap did not preempt the reviewer: calls %v, %d steps", calls, len(detail.Steps))
}
if detail.Status != "failed" || detail.Diagnostic != "stage implementer visit limit reached (2)" {
t.Fatalf("preempted limit diagnostic = %+v", detail)
}
}
// TestBlockedRetryableRetriesWithinBudgetThenStops proves a transient block is
// retried only inside its declared visit bound and then fails closed, so a
// retryable implementer block cannot become a retry storm.
func TestBlockedRetryableRetriesWithinBudgetThenStops(t *testing.T) {
ctx := context.Background()
r, store, id, calls := reviewLoopRunner(t, ctx, retryableBlockSource(2))
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
calls[def.Key]++
return stage.Result{Outcome: "blocked_retryable", Output: stage.Values{"blocked_reason": "device lock held"}}, nil
}
run := startRun(t, r, id, "retryable")
detail := getRun(t, store, ctx, run.ID)
if calls["worker"] != 2 || len(detail.Steps) != 2 {
t.Fatalf("retry bound = calls %v, %d steps, want 2", calls, len(detail.Steps))
}
last := detail.Steps[1]
if detail.Status != "failed" || last.Outcome != "blocked_retryable" || last.Control == nil || !last.Control.LimitReached || last.Target != protocol.FailureEnd {
t.Fatalf("retryable block not bounded: %+v", last)
}
if last.Diagnostic != "Pipelines stopped worker after 2 visits on outcome blocked_retryable" {
t.Fatalf("retry bound diagnostic = %q", last.Diagnostic)
}
}
// TestBlockedRetryableRetryCanRecover proves the bounded retry is useful: a stage
// that blocks once on a transient cause and succeeds on the retry continues to
// its normal next stage instead of failing the run.
func TestBlockedRetryableRetryCanRecover(t *testing.T) {
ctx := context.Background()
r, store, id, calls := reviewLoopRunner(t, ctx, retryableBlockSource(3))
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
calls[def.Key]++
if def.Key == "worker" && calls["worker"] == 1 {
return stage.Result{Outcome: "blocked_retryable", Output: stage.Values{"blocked_reason": "held device lock"}}, nil
}
return stage.Result{Outcome: "success", Output: stage.Values{"summary": def.Key}}, nil
}
run := startRun(t, r, id, "recovered")
detail := getRun(t, store, ctx, run.ID)
if calls["worker"] != 2 || calls["review"] != 1 || detail.Status != "succeeded" {
t.Fatalf("retry did not recover: calls %v, status %s", calls, detail.Status)
}
}
// TestTerminalBlockedFailsImmediately proves a terminal block (an acceptance
// criterion another attempt cannot change) stops on the first visit with no
// retry and a meaningful run diagnostic.
func TestTerminalBlockedFailsImmediately(t *testing.T) {
ctx := context.Background()
source := strings.Replace(pipeline.Template([]string{"worker", "review"}, true), "success -> review", "success -> review\nblocked -> #failure", 1)
r, store, id, calls := reviewLoopRunner(t, ctx, source)
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
calls[def.Key]++
return stage.Result{Outcome: "blocked", Output: stage.Values{"blocked_reason": "absent capability"}}, nil
}
run := startRun(t, r, id, "terminal")
detail := getRun(t, store, ctx, run.ID)
if calls["worker"] != 1 || len(detail.Steps) != 1 {
t.Fatalf("terminal block was retried: calls %v, %d steps", calls, len(detail.Steps))
}
if detail.Status != "failed" || detail.Diagnostic != `stage worker returned outcome "blocked"` {
t.Fatalf("terminal block diagnostic = %+v", detail)
}
if detail.Steps[0].Target != protocol.FailureEnd {
t.Fatalf("terminal block target = %q", detail.Steps[0].Target)
}
}
// reviewerLoopSource is the worker/review/commit shape the maintenance pipelines
// use: the reviewer keeps its bounded rejection edge and gains an advisory edge
// that continues the run without another worker visit.
func reviewerLoopSource() string {
source := pipeline.Template([]string{"worker", "review", "commit"}, true)
source = strings.Replace(source, "# stage review", "# stage review\n\n```limits\n{\"max_visits\":3,\"on\":[\"rejected\"]}\n```", 1)
return strings.Replace(source, "success -> commit", "success -> commit\nadvisory -> commit\nrejected -> worker", 1)
}
// publishedReviewLoopSource reconstructs the published maintenance/quick/focused
// control shape with the real stage keys: the implementer carries its own
// blocked_retryable limits block and self-edge, and the reviewer's rejected edge
// returns to that same implementer. implementerVisits is the implementer's
// max_visits, which also caps its re-entries from the reviewer, so it must be at
// least the reviewer's rejection budget.
func publishedReviewLoopSource(implementerVisits int) string {
source := pipeline.Template([]string{"implementer", "reviewer", "commit"}, true)
source = strings.Replace(source, "# stage implementer", "# stage implementer\n\n```limits\n{\"max_visits\":"+strconv.Itoa(implementerVisits)+",\"on\":[\"blocked_retryable\"]}\n```", 1)
source = strings.Replace(source, "success -> reviewer", "success -> reviewer\nblocked_retryable -> implementer\nblocked -> #failure", 1)
source = strings.Replace(source, "# stage reviewer", "# stage reviewer\n\n```limits\n{\"max_visits\":3,\"on\":[\"rejected\"]}\n```", 1)
return strings.Replace(source, "success -> commit", "success -> commit\nadvisory -> commit\nrejected -> implementer", 1)
}
// retryableBlockSource is a worker/review shape whose worker declares a bounded
// blocked_retryable self-edge, mirroring the published definitions.
func retryableBlockSource(maxVisits int) string {
source := pipeline.Template([]string{"worker", "review"}, true)
source = strings.Replace(source, "# stage worker", "# stage worker\n\n```limits\n{\"max_visits\":"+strconv.Itoa(maxVisits)+",\"on\":[\"blocked_retryable\"]}\n```", 1)
return strings.Replace(source, "success -> review", "success -> review\nblocked_retryable -> worker", 1)
}