Merge: no UI-review auto-skip without complete change capture; capture lists over-budget changes
# Conflicts: # internal/protocol/runs.go
This commit is contained in:
@@ -665,6 +665,13 @@ total as input tokens. The
|
||||
CLI is the deliverable; a TUI picker for choosing overrides is a documented
|
||||
follow-up, not half-built.
|
||||
|
||||
The run detail (`inspect <id> --json`, `get_run`) also carries an additive
|
||||
top-level `warnings` string list, derived at read time from the steps; `inspect`
|
||||
repeats each entry on stderr as `warning: ...`. Today it names every visit whose
|
||||
retained file-change capture was `partial` or `unavailable`, because such a
|
||||
capture is not a complete workspace diff and disables the server skip fallback
|
||||
(see [stage-file-diffs.md](stage-file-diffs.md)). Older peers ignore the field.
|
||||
|
||||
|
||||
On either the runs list or run inspector, press `P` (or use `:pipeline`) to open
|
||||
the selected run's exact pipeline version. Navigation does not cancel the run.
|
||||
|
||||
@@ -26,10 +26,21 @@ includes the same evidence under each step's `file_changes` field.
|
||||
`secrets/`, Git internals and `node_modules/` are excluded. This is a path-based
|
||||
precaution, **not secret-content scanning**; ordinary source patches may contain
|
||||
sensitive text and become part of retained history.
|
||||
- Limits: 10,000 inventoried entries, 1 MiB per file, 32 MiB per snapshot,
|
||||
five seconds per snapshot, 5,000 combined lines per textual comparison,
|
||||
64 KiB per patch and 128 KiB serialized file evidence per visit. Oversized,
|
||||
unreadable, omitted or truncated evidence is explicitly flagged partial.
|
||||
- Limits: 10,000 inventoried entries, 1 MiB per file and 32 MiB per snapshot of
|
||||
retained text, 256 MiB per hashed file, five seconds per snapshot, 5,000
|
||||
combined lines per textual comparison, 64 KiB per patch and 128 KiB serialized
|
||||
file evidence per visit. Oversized, unreadable, omitted or truncated evidence
|
||||
is explicitly flagged partial.
|
||||
- Over-budget content degrades to a listing, not silence: a text file over 1 MiB,
|
||||
or beyond the 32 MiB snapshot budget, is still compared by SHA-256 and a change
|
||||
is listed with `no_patch: true` (and `truncated`) but no patch. In a Git working
|
||||
tree, a path the snapshot could not compare at all (above 256 MiB, unreadable,
|
||||
or past the time/file limit) is listed the same way when its
|
||||
`git status --porcelain` entry changed during the visit; a path dirty with the
|
||||
same status before and after cannot be proven changed and stays in `skipped`.
|
||||
Git is only read (`--no-optional-locks`); protected paths are never listed.
|
||||
- `pipelines inspect RUN_ID --json` carries a top-level `warnings` list (also
|
||||
printed to stderr) naming every visit whose capture was partial or unavailable.
|
||||
- Failed/canceled stages still retain changes when execution returns. Unconfirmed
|
||||
agent cleanup is flagged partial. A server/process crash cannot guarantee an
|
||||
after-snapshot; recovered steps without capture say “not recorded.”
|
||||
@@ -41,3 +52,24 @@ includes the same evidence under each step's `file_changes` field.
|
||||
|
||||
Snapshots are transient server memory. The database retains patches/metadata,
|
||||
not full copies of every workspace file. No new model calls are used for capture.
|
||||
|
||||
## Server skip fallback
|
||||
|
||||
A non-human stage that declares a `skipped` edge (typically a UI reviewer) is
|
||||
completed by the server without a model call, and routed along `skipped`, only
|
||||
when every earlier visit of the run recorded a **complete** capture with **no**
|
||||
changed files and no image artifact. The step's verdict reason begins
|
||||
"Server fallback skip (no model run):" and has `reason_source: fallback`.
|
||||
|
||||
Everything else runs the stage's model, which may still choose `skipped` itself:
|
||||
|
||||
- any recorded file change, whatever its path or extension (no repository layout
|
||||
or "non-UI" extension list is assumed: Go TUIs, Flutter `.dart`, Kotlin, Swift,
|
||||
templates, strings and config files can all be rendered surface);
|
||||
- any partial or unavailable capture, or a visit with no capture (a crash, a
|
||||
historical row, a human decision during which the workspace was unobserved, or
|
||||
a checkpoint-retry prefix);
|
||||
- `auto_skip=false` in the stage's `configs`.
|
||||
|
||||
A wrong run costs one model call; a wrong skip would ship an unreviewed change, so
|
||||
absent or partial evidence is never read as "nothing changed".
|
||||
|
||||
@@ -82,6 +82,14 @@ and accepts integers 0–100; explicit 0 disables execution retries. Optional
|
||||
`timeout` defaults to `1h` per attempt and accepts positive durations such as
|
||||
`30m` or `90s`.
|
||||
|
||||
A stage that declares a `skipped` edge (for example a UI reviewer) is eligible
|
||||
for the server skip fallback: when every earlier visit's file-change capture was
|
||||
complete and recorded no changed files or screenshots, the server completes the
|
||||
stage as `skipped` without a model call (`reason_source: fallback`). Any recorded
|
||||
change or incomplete capture runs the model. Optional `auto_skip=false` (values
|
||||
`true`/`false`, default `true`) always runs the model. Read
|
||||
`references/operations.md` ("Server skip fallback") for the exact rule.
|
||||
|
||||
Optional `reasoning_effort=high` selects a model's reasoning/thinking level for
|
||||
this stage. Values are `default`, `none` (alias `off`), `minimal`, `low`, `medium`,
|
||||
`high`, `xhigh`, or `max`. Omission or `default` preserves the harness/model
|
||||
|
||||
@@ -191,9 +191,14 @@ The stage's File changes section highlights added/removed lines and hunk headers
|
||||
node_modules and common environment/key/credential/database paths are excluded.
|
||||
This is path-based protection, **not secret-content scanning**. Patches may still
|
||||
contain sensitive text; never publish them without approval.
|
||||
- Limits: 10,000 entries, 1 MiB/file, 32 MiB/snapshot, five-second snapshot context,
|
||||
5,000 combined text lines, 64 KiB/patch, 128 KiB serialized file evidence/visit.
|
||||
Missing/skipped/truncated captures must be described as partial/unavailable.
|
||||
- Limits: 10,000 entries, 1 MiB/file and 32 MiB/snapshot of retained text,
|
||||
256 MiB/hashed file, five-second snapshot context, 5,000 combined text lines,
|
||||
64 KiB/patch, 128 KiB serialized file evidence/visit. Text over the 1 MiB or
|
||||
32 MiB budget is still compared by hash, and in Git a path the snapshot could
|
||||
not compare is listed when its `git status` entry changes; such entries carry
|
||||
`no_patch: true` (and `truncated`) with no patch, and the visit is `partial`.
|
||||
Missing/skipped/truncated captures must be described as partial/unavailable;
|
||||
`inspect` adds a run-level `warnings` entry naming every partial/unavailable visit.
|
||||
- Failed/canceled stages retain changes when execution returns; unconfirmed agent
|
||||
cleanup is partial. Crashes may have no after-capture. Historical visits say
|
||||
not recorded—**never infer no edits from absent evidence**.
|
||||
@@ -201,6 +206,19 @@ The stage's File changes section highlights added/removed lines and hunk headers
|
||||
changes outside scope, ignored new files and changes reverted before capture can
|
||||
be absent or mixed in. Capture failure does not change the declared run verdict.
|
||||
|
||||
### Server skip fallback (`skipped` edge)
|
||||
|
||||
A non-human stage that declares a `skipped` edge is completed by the server, with
|
||||
no model call, and routed along `skipped` only when **every** earlier visit of the
|
||||
run recorded `file_changes` with status `complete`, none recorded any changed file,
|
||||
and none captured an image artifact. The step's verdict reason starts
|
||||
"Server fallback skip (no model run):" with `reason_source: fallback`. Any changed
|
||||
file (no path or extension is treated as "not UI"), any partial/unavailable or
|
||||
missing capture (crash, historical row, human decision, checkpoint-retry prefix)
|
||||
runs the model, which may still answer `skipped` itself. `auto_skip=false` in the
|
||||
stage's `configs` disables the fallback for that stage. Never describe a fallback
|
||||
skip as a model judgement, and never treat it as evidence that nothing changed.
|
||||
|
||||
Do not add model output fields, new stages or extra model calls merely to obtain
|
||||
file diffs: current capture is server-owned. Do not promise full files/rollback.
|
||||
|
||||
|
||||
@@ -1001,6 +1001,13 @@ func (a App) Run(ctx context.Context, args []string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if command == "inspect" && r.RunDetail != nil {
|
||||
// Warnings are also in the JSON; stderr keeps them visible to a reader
|
||||
// of the human stream without corrupting machine-readable stdout.
|
||||
for _, warning := range r.RunDetail.Warnings {
|
||||
a.notice("warning: " + warning)
|
||||
}
|
||||
}
|
||||
return a.output(r.RunDetail)
|
||||
case "decide":
|
||||
if len(pos) < 1 || len(pos) > 2 {
|
||||
|
||||
+191
-37
@@ -14,6 +14,7 @@ import (
|
||||
"os/exec"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"unicode/utf8"
|
||||
|
||||
@@ -28,13 +29,37 @@ const (
|
||||
maxPatchBytes = 128 << 10
|
||||
maxFilePatchBytes = 64 << 10
|
||||
maxLines = 5000
|
||||
binarySniffBytes = 8 << 10
|
||||
)
|
||||
|
||||
// maxHashBytes bounds one hash-only file; larger files are skipped (and, in a
|
||||
// Git working tree, listed only when their Git status changes). It is a
|
||||
// variable so package tests can exercise that path without huge files.
|
||||
var maxHashBytes int64 = 256 << 20
|
||||
|
||||
// snapshotBudget is the retained text bytes per snapshot. Files beyond it (or
|
||||
// beyond the per-file limit) are still compared by content hash and listed
|
||||
// without a patch, so an over-budget workspace never hides which files changed.
|
||||
var snapshotBudget atomic.Int64
|
||||
|
||||
func init() { snapshotBudget.Store(maxSnapshotBytes) }
|
||||
|
||||
// SetSnapshotBudgetForTesting lowers the retained-text budget so tests can
|
||||
// exercise an over-budget workspace without writing tens of megabytes. It
|
||||
// returns a function restoring the previous budget. Production never calls it.
|
||||
func SetSnapshotBudgetForTesting(n int64) (restore func()) {
|
||||
previous := snapshotBudget.Swap(n)
|
||||
return func() { snapshotBudget.Store(previous) }
|
||||
}
|
||||
|
||||
type file struct {
|
||||
data []byte
|
||||
hash [32]byte
|
||||
mode fs.FileMode
|
||||
binary bool
|
||||
// hashOnly marks a text file compared by content hash only: it exceeded the
|
||||
// per-file or per-snapshot text budget, so no patch can be produced.
|
||||
hashOnly bool
|
||||
}
|
||||
|
||||
type Snapshot struct {
|
||||
@@ -44,6 +69,63 @@ type Snapshot struct {
|
||||
available bool
|
||||
diagnostic string
|
||||
scope string
|
||||
// gitStatus maps run-relative paths to their porcelain XY status code when
|
||||
// the run directory is a Git working tree and the status read succeeded
|
||||
// (gitOK). Compare uses it to list changed paths the content snapshot could
|
||||
// not compare (time/file limit, unreadable); it never reads HEAD/index content.
|
||||
gitStatus map[string]string
|
||||
gitOK bool
|
||||
}
|
||||
|
||||
// readGitStatus reads `git status --porcelain=v1 -z` for the run directory and
|
||||
// returns its paths relative to that directory. It only reads: optional locks
|
||||
// are disabled so the index is never refreshed or rewritten.
|
||||
func readGitStatus(ctx context.Context, directory string) (map[string]string, bool) {
|
||||
prefixOut, err := exec.CommandContext(ctx, "git", "-c", "core.fsmonitor=false", "-C", directory, "rev-parse", "--show-prefix").Output()
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
prefix := strings.TrimSpace(string(prefixOut))
|
||||
cmd := exec.CommandContext(ctx, "git", "--no-optional-locks", "-c", "core.fsmonitor=false", "-C", directory, "status", "--porcelain=v1", "-z", "--no-renames", "--untracked-files=all", "--", ".")
|
||||
out := &boundedOutput{limit: 4 << 20}
|
||||
cmd.Stdout = out
|
||||
if err := cmd.Run(); err != nil {
|
||||
return nil, false
|
||||
}
|
||||
status := map[string]string{}
|
||||
for _, entry := range strings.Split(out.String(), "\x00") {
|
||||
if len(entry) < 4 || entry[2] != ' ' {
|
||||
continue
|
||||
}
|
||||
name, ok := strings.CutPrefix(entry[3:], prefix)
|
||||
if !ok || name == "" {
|
||||
continue // outside the run directory
|
||||
}
|
||||
status[name] = entry[:2]
|
||||
}
|
||||
return status, true
|
||||
}
|
||||
|
||||
// hashFile streams f from its start into a content hash without retaining it.
|
||||
// It returns ok=false when the file exceeds maxHashBytes or cannot be read.
|
||||
func hashFile(f *os.File) (hash [32]byte, binary bool, ok bool) {
|
||||
if _, err := f.Seek(0, io.SeekStart); err != nil {
|
||||
return hash, false, false
|
||||
}
|
||||
h := sha256.New()
|
||||
head := make([]byte, binarySniffBytes)
|
||||
n, err := io.ReadFull(f, head)
|
||||
if err != nil && !errors.Is(err, io.ErrUnexpectedEOF) && !errors.Is(err, io.EOF) {
|
||||
return hash, false, false
|
||||
}
|
||||
head = head[:n]
|
||||
h.Write(head)
|
||||
copied, err := io.Copy(h, io.LimitReader(f, maxHashBytes+1-int64(n)))
|
||||
if err != nil || int64(n)+copied > maxHashBytes {
|
||||
return hash, false, false
|
||||
}
|
||||
copy(hash[:], h.Sum(nil))
|
||||
return hash, bytes.IndexByte(head, 0) >= 0, true
|
||||
}
|
||||
|
||||
func protected(name string) bool {
|
||||
@@ -130,6 +212,9 @@ func Capture(ctx context.Context, directory string, previous ...*Snapshot) *Snap
|
||||
s.partial = true
|
||||
s.diagnostic = "Workspace inventory incomplete."
|
||||
}
|
||||
if git {
|
||||
s.gitStatus, s.gitOK = readGitStatus(ctx, directory)
|
||||
}
|
||||
// Keep pre-stage paths even if a stage changes .gitignore/the index.
|
||||
known := map[string]bool{}
|
||||
for _, name := range names {
|
||||
@@ -179,7 +264,7 @@ func Capture(ctx context.Context, directory string, previous ...*Snapshot) *Snap
|
||||
continue
|
||||
}
|
||||
info, err := root.Lstat(name)
|
||||
if err != nil || !info.Mode().IsRegular() || info.Size() > maxFileBytes {
|
||||
if err != nil || !info.Mode().IsRegular() || info.Size() > maxHashBytes {
|
||||
s.skipped[name] = true
|
||||
continue
|
||||
}
|
||||
@@ -194,28 +279,43 @@ func Capture(ctx context.Context, directory string, previous ...*Snapshot) *Snap
|
||||
s.skipped[name] = true
|
||||
continue
|
||||
}
|
||||
data, readErr := io.ReadAll(io.LimitReader(f, maxFileBytes+1))
|
||||
if opened.Size() <= maxFileBytes {
|
||||
data, readErr := io.ReadAll(io.LimitReader(f, maxFileBytes+1))
|
||||
if readErr != nil {
|
||||
f.Close()
|
||||
s.skipped[name] = true
|
||||
continue
|
||||
}
|
||||
if len(data) <= maxFileBytes {
|
||||
f.Close()
|
||||
binary := bytes.IndexByte(data, 0) >= 0 || !utf8.Valid(data)
|
||||
hash := sha256.Sum256(data)
|
||||
hashOnly := false
|
||||
// Binary changes need only metadata and a hash, never retained patch bytes.
|
||||
// Counting screenshots against the retained-text budget can otherwise
|
||||
// exhaust it before any source files are reached in sorted inventory order.
|
||||
// Text beyond the budget keeps only its hash, so a change to it is still
|
||||
// listed (without a patch) instead of being skipped unseen.
|
||||
if binary {
|
||||
data = nil
|
||||
} else if int64(used+len(data)) > snapshotBudget.Load() {
|
||||
data = nil
|
||||
hashOnly = true
|
||||
} else {
|
||||
used += len(data)
|
||||
}
|
||||
s.files[name] = file{data: data, hash: hash, mode: opened.Mode().Perm(), binary: binary, hashOnly: hashOnly}
|
||||
continue
|
||||
}
|
||||
// The file grew past the per-file limit while being read: hash it.
|
||||
}
|
||||
hash, binary, ok := hashFile(f)
|
||||
f.Close()
|
||||
if readErr != nil || len(data) > maxFileBytes {
|
||||
if !ok {
|
||||
s.skipped[name] = true
|
||||
continue
|
||||
}
|
||||
binary := bytes.IndexByte(data, 0) >= 0 || !utf8.Valid(data)
|
||||
hash := sha256.Sum256(data)
|
||||
// Binary changes need only metadata and a hash, never retained patch bytes.
|
||||
// Counting screenshots against the retained-text budget can otherwise
|
||||
// exhaust it before any source files are reached in sorted inventory order.
|
||||
if binary {
|
||||
data = nil
|
||||
} else {
|
||||
if used+len(data) > maxSnapshotBytes {
|
||||
s.skipped[name] = true
|
||||
s.partial = true
|
||||
continue
|
||||
}
|
||||
used += len(data)
|
||||
}
|
||||
s.files[name] = file{data: data, hash: hash, mode: opened.Mode().Perm(), binary: binary}
|
||||
s.files[name] = file{hash: hash, mode: opened.Mode().Perm(), binary: binary, hashOnly: !binary}
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -258,6 +358,35 @@ func Compare(before, after *Snapshot) *protocol.StageFileChanges {
|
||||
sort.Strings(names)
|
||||
remaining := maxPatchBytes
|
||||
retained := 0
|
||||
add := func(change protocol.FileChange) {
|
||||
if change.Truncated {
|
||||
result.Status = "partial"
|
||||
if !change.NoPatch {
|
||||
result.Diagnostic = "Some patches were omitted/truncated by capture limits."
|
||||
}
|
||||
}
|
||||
encoded, _ := json.Marshal(change)
|
||||
if retained+len(encoded)+1 > maxPatchBytes {
|
||||
change.Patch = ""
|
||||
change.Truncated = true
|
||||
encoded, _ = json.Marshal(change)
|
||||
}
|
||||
if retained+len(encoded)+1 > maxPatchBytes {
|
||||
result.Omitted++
|
||||
result.Status = "partial"
|
||||
result.Diagnostic = "Additional file changes omitted by the retained evidence size limit."
|
||||
return
|
||||
}
|
||||
if change.Truncated {
|
||||
result.Status = "partial"
|
||||
}
|
||||
retained += len(encoded) + 1
|
||||
result.Files = append(result.Files, change)
|
||||
}
|
||||
listed := 0
|
||||
// decided records every path the content comparison settled (changed or
|
||||
// not), so the Git-status pass below only speaks for the rest.
|
||||
decided := map[string]bool{}
|
||||
for _, name := range names {
|
||||
if skipped[name] {
|
||||
continue
|
||||
@@ -268,6 +397,7 @@ func Compare(before, after *Snapshot) *protocol.StageFileChanges {
|
||||
if !aOK && before.partial || !bOK && after.partial {
|
||||
continue
|
||||
}
|
||||
decided[name] = true
|
||||
if aOK && bOK && a.hash == b.hash && a.mode == b.mode {
|
||||
continue
|
||||
}
|
||||
@@ -285,7 +415,12 @@ func Compare(before, after *Snapshot) *protocol.StageFileChanges {
|
||||
if bOK {
|
||||
change.AfterMode = fmt.Sprintf("%04o", b.mode)
|
||||
}
|
||||
if !change.Binary && a.hash != b.hash {
|
||||
if !change.Binary && a.hash != b.hash && (a.hashOnly || b.hashOnly) {
|
||||
// Over-budget content was compared by hash: list it, without a patch.
|
||||
change.Truncated = true
|
||||
change.NoPatch = true
|
||||
listed++
|
||||
} else if !change.Binary && a.hash != b.hash {
|
||||
if remaining == 0 || bytes.Count(a.data, []byte("\n"))+bytes.Count(b.data, []byte("\n")) > maxLines {
|
||||
change.Truncated = true
|
||||
} else {
|
||||
@@ -314,27 +449,46 @@ func Compare(before, after *Snapshot) *protocol.StageFileChanges {
|
||||
}
|
||||
}
|
||||
}
|
||||
if change.Truncated {
|
||||
result.Status = "partial"
|
||||
result.Diagnostic = "Some patches were omitted/truncated by capture limits."
|
||||
add(change)
|
||||
}
|
||||
// In a Git working tree, paths the content snapshot could not settle
|
||||
// (skipped as unreadable/oversized, or beyond the time/file limit) are still
|
||||
// listed when their Git status changed during the visit. A path dirty with
|
||||
// the same status before and after cannot be proven changed and stays
|
||||
// counted as skipped. Protected paths are never listed.
|
||||
if before.gitOK && after.gitOK {
|
||||
gitNames := map[string]bool{}
|
||||
for name := range before.gitStatus {
|
||||
gitNames[name] = true
|
||||
}
|
||||
encoded, _ := json.Marshal(change)
|
||||
if retained+len(encoded)+1 > maxPatchBytes {
|
||||
change.Patch = ""
|
||||
change.Truncated = true
|
||||
encoded, _ = json.Marshal(change)
|
||||
for name := range after.gitStatus {
|
||||
gitNames[name] = true
|
||||
}
|
||||
if retained+len(encoded)+1 > maxPatchBytes {
|
||||
result.Omitted++
|
||||
result.Status = "partial"
|
||||
result.Diagnostic = "Additional file changes omitted by the retained evidence size limit."
|
||||
continue
|
||||
ordered := make([]string, 0, len(gitNames))
|
||||
for name := range gitNames {
|
||||
if !decided[name] && fs.ValidPath(name) && !protected(name) {
|
||||
ordered = append(ordered, name)
|
||||
}
|
||||
}
|
||||
if change.Truncated {
|
||||
result.Status = "partial"
|
||||
sort.Strings(ordered)
|
||||
for _, name := range ordered {
|
||||
was, now := before.gitStatus[name], after.gitStatus[name]
|
||||
if was == now {
|
||||
continue
|
||||
}
|
||||
kind := "modified"
|
||||
switch {
|
||||
case now == "" && was == "??", strings.Contains(now, "D"):
|
||||
kind = "deleted"
|
||||
case was == "" && (now == "??" || strings.HasPrefix(now, "A")):
|
||||
kind = "added"
|
||||
}
|
||||
add(protocol.FileChange{Path: name, Kind: kind, Truncated: true, NoPatch: true})
|
||||
listed++
|
||||
}
|
||||
retained += len(encoded) + 1
|
||||
result.Files = append(result.Files, change)
|
||||
}
|
||||
if listed > 0 && result.Status == "partial" && result.Omitted == 0 {
|
||||
result.Diagnostic = "Some changed files exceeded capture limits and are listed without patches; this is not a complete workspace diff."
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
@@ -132,9 +132,14 @@ func TestProtectedSymlinkAndLimits(t *testing.T) {
|
||||
put(t, dir, "large", "small now")
|
||||
put(t, dir, "normal", "safe\n")
|
||||
changes := Compare(before, Capture(ctx, dir, before))
|
||||
if changes.Status != "partial" || len(changes.Files) != 1 || changes.Files[0].Path != "normal" {
|
||||
// The oversized file is compared by hash and listed without a patch; the
|
||||
// protected file and the symlink stay excluded.
|
||||
if changes.Status != "partial" || len(changes.Files) != 2 || changes.Files[0].Path != "large" || changes.Files[1].Path != "normal" {
|
||||
t.Fatal(changes)
|
||||
}
|
||||
if large := changes.Files[0]; !large.NoPatch || !large.Truncated || large.Patch != "" || large.Kind != "modified" {
|
||||
t.Fatalf("oversized file not listed without patch: %+v", large)
|
||||
}
|
||||
for _, f := range changes.Files {
|
||||
if strings.Contains(f.Patch, "secret") {
|
||||
t.Fatal("secret retained")
|
||||
@@ -189,3 +194,92 @@ func TestBinaryFilesDoNotStarveSourceDiffs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func gitRepo(t *testing.T) (string, func(...string)) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("git"); err != nil {
|
||||
t.Skip("git unavailable")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
run := func(args ...string) {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", append([]string{"-C", dir, "-c", "user.email=t@example.invalid", "-c", "user.name=t", "-c", "commit.gpgsign=false"}, args...)...)
|
||||
if data, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("%v: %v %s", args, err, data)
|
||||
}
|
||||
}
|
||||
run("init", "-q")
|
||||
return dir, run
|
||||
}
|
||||
|
||||
// TestOverBudgetChangeIsListedWithoutPatch is the run-236 regression: a
|
||||
// workspace whose text exceeds the snapshot budget must still list the one
|
||||
// changed file (sorted last, after the budget is exhausted) instead of
|
||||
// returning files: [] with every late path skipped.
|
||||
func TestOverBudgetChangeIsListedWithoutPatch(t *testing.T) {
|
||||
restore := SetSnapshotBudgetForTesting(64)
|
||||
defer restore()
|
||||
dir, git := gitRepo(t)
|
||||
if err := os.MkdirAll(filepath.Join(dir, "web/src"), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 20; i++ {
|
||||
put(t, dir, fmt.Sprintf("a-%02d.txt", i), strings.Repeat("filler\n", 4))
|
||||
}
|
||||
put(t, dir, "web/src/Inbox.test.tsx", "it('works')\n")
|
||||
git("add", ".")
|
||||
git("commit", "-qm", "init")
|
||||
ctx := context.Background()
|
||||
before := Capture(ctx, dir)
|
||||
put(t, dir, "web/src/Inbox.test.tsx", "it('works')\nit('more')\n")
|
||||
changes := Compare(before, Capture(ctx, dir, before))
|
||||
if len(changes.Files) != 1 || changes.Files[0].Path != "web/src/Inbox.test.tsx" {
|
||||
t.Fatalf("over-budget change not listed: %+v", changes)
|
||||
}
|
||||
if f := changes.Files[0]; f.Kind != "modified" || !f.NoPatch || f.Patch != "" {
|
||||
t.Fatalf("over-budget change should be listed without patch: %+v", f)
|
||||
}
|
||||
if changes.Status != "partial" || changes.Skipped != 0 {
|
||||
t.Fatalf("status/skipped = %q/%d, want partial/0", changes.Status, changes.Skipped)
|
||||
}
|
||||
// Unchanged over-budget files are compared by hash, so a visit that edits
|
||||
// nothing is a complete, empty capture rather than a partial one.
|
||||
again := Capture(ctx, dir)
|
||||
if empty := Compare(again, Capture(ctx, dir, again)); empty.Status != "complete" || len(empty.Files) != 0 {
|
||||
t.Fatalf("no-op over-budget visit = %+v, want complete and empty", empty)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGitStatusListsUncomparableChanges proves a file the content snapshot
|
||||
// skips (here: above the hash bound) is still listed when its Git status
|
||||
// changes during the visit, while a protected path never is.
|
||||
func TestGitStatusListsUncomparableChanges(t *testing.T) {
|
||||
previous := maxHashBytes
|
||||
maxHashBytes = 16
|
||||
defer func() { maxHashBytes = previous }()
|
||||
dir, git := gitRepo(t)
|
||||
put(t, dir, "big.bin", strings.Repeat("x", 64))
|
||||
put(t, dir, ".env", "A=1\n")
|
||||
put(t, dir, "small.txt", "one\n")
|
||||
git("add", "big.bin", "small.txt")
|
||||
git("commit", "-qm", "init")
|
||||
ctx := context.Background()
|
||||
before := Capture(ctx, dir)
|
||||
put(t, dir, "big.bin", strings.Repeat("y", 64))
|
||||
put(t, dir, "fresh.bin", strings.Repeat("z", 64))
|
||||
put(t, dir, ".env", "A=2\n")
|
||||
changes := Compare(before, Capture(ctx, dir, before))
|
||||
got := map[string]string{}
|
||||
for _, f := range changes.Files {
|
||||
if !f.NoPatch {
|
||||
t.Fatalf("git-listed change carries a patch: %+v", f)
|
||||
}
|
||||
got[f.Path] = f.Kind
|
||||
}
|
||||
if len(got) != 2 || got["big.bin"] != "modified" || got["fresh.bin"] != "added" {
|
||||
t.Fatalf("git status listing = %v (%+v)", got, changes)
|
||||
}
|
||||
if changes.Status != "partial" || !strings.Contains(changes.Diagnostic, "without patches") {
|
||||
t.Fatalf("status/diagnostic = %q/%q", changes.Status, changes.Diagnostic)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -303,6 +303,7 @@ strftime('%Y-%m-%dT%H:%M:%SZ',started_at),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ'
|
||||
// The run total is one pass over the steps already loaded above: it adds no
|
||||
// query and no transcript parse.
|
||||
r.Usage = protocol.TotalUsage(r.Steps)
|
||||
r.Warnings = protocol.RunWarnings(r.Steps)
|
||||
alerts, err := tx.QueryContext(ctx, `SELECT sequence,event,channel,status,attempts,diagnostic,message_id FROM run_notifications WHERE run_id=? ORDER BY sequence,event,channel`, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
package protocol
|
||||
|
||||
import "strings"
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
RunPipeline = "run_pipeline"
|
||||
@@ -363,6 +366,32 @@ type RunUsage struct {
|
||||
CacheKnownStages int `json:"cache_known_stages,omitempty"`
|
||||
}
|
||||
|
||||
// RunWarnings derives the run-level warnings from its steps: every visit whose
|
||||
// retained file-change capture was partial or unavailable is named, because
|
||||
// such evidence is not a complete workspace diff and must never be read as "no
|
||||
// edits". Steps with no capture (historical, human or server-skip visits) are
|
||||
// not flagged here; their File changes section already says "not recorded".
|
||||
func RunWarnings(steps []RunStep) []string {
|
||||
var visits []string
|
||||
for _, step := range steps {
|
||||
if step.FileChanges == nil || step.FileChanges.Status == "complete" {
|
||||
continue
|
||||
}
|
||||
visits = append(visits, fmt.Sprintf("%s #%d (%s)", step.Stage, step.Sequence, step.FileChanges.Status))
|
||||
}
|
||||
if len(visits) == 0 {
|
||||
return nil
|
||||
}
|
||||
const shown = 8
|
||||
list := visits
|
||||
extra := ""
|
||||
if len(list) > shown {
|
||||
extra = fmt.Sprintf(" and %d more", len(list)-shown)
|
||||
list = list[:shown]
|
||||
}
|
||||
return []string{fmt.Sprintf("File-change capture was incomplete for %d visit(s): %s%s. Recorded file changes are not a complete workspace diff, and no stage is auto-skipped after such a visit.", len(visits), strings.Join(list, ", "), extra)}
|
||||
}
|
||||
|
||||
// TotalUsage sums the per-stage usage of a run's steps into one run total. It is
|
||||
// the one definition of the run total, so per-run inspection cannot drift from
|
||||
// the history-wide aggregate over the same rows.
|
||||
@@ -432,11 +461,15 @@ type StageFileChanges struct {
|
||||
}
|
||||
|
||||
type FileChange struct {
|
||||
Path string `json:"path"`
|
||||
Kind string `json:"kind"` // added, modified, deleted
|
||||
Patch string `json:"patch,omitempty"`
|
||||
Binary bool `json:"binary,omitempty"`
|
||||
Truncated bool `json:"truncated,omitempty"`
|
||||
Path string `json:"path"`
|
||||
Kind string `json:"kind"` // added, modified, deleted
|
||||
Patch string `json:"patch,omitempty"`
|
||||
Binary bool `json:"binary,omitempty"`
|
||||
Truncated bool `json:"truncated,omitempty"`
|
||||
// NoPatch marks a change listed without any patch: the content exceeded the
|
||||
// capture budget (compared by hash) or the change was seen only in Git
|
||||
// status. Truncated is also set so older readers show "patch omitted".
|
||||
NoPatch bool `json:"no_patch,omitempty"`
|
||||
BeforeMode string `json:"before_mode,omitempty"`
|
||||
AfterMode string `json:"after_mode,omitempty"`
|
||||
}
|
||||
@@ -465,6 +498,10 @@ type RunDetail struct {
|
||||
Input map[string]any `json:"input"`
|
||||
Output map[string]any `json:"output"`
|
||||
Steps []RunStep `json:"steps"`
|
||||
// Warnings are run-level, read-time notes an operator should see before
|
||||
// trusting the evidence, e.g. that a visit's file-change capture was partial.
|
||||
// Derived from Steps by RunWarnings; absent when there is nothing to flag.
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
// TitleSource is "model" when Title came from the shared summariser or
|
||||
// "fallback" when it was derived from the prompt; TitleModel names the model
|
||||
// that produced a model title. They are retained so a diagnosis can tell the
|
||||
|
||||
+23
-14
@@ -951,11 +951,14 @@ type runState struct {
|
||||
input stage.Values
|
||||
// changedPaths retains a bounded sample of the run's recorded changed paths
|
||||
// and changedCount the full total, so the skip guard reads only in-memory
|
||||
// evidence. uiSurface is latched once any recorded change touched a UI
|
||||
// surface. Both are folded in by recordChanges (ui_surface.go).
|
||||
changedPaths []string
|
||||
changedCount int
|
||||
uiSurface bool
|
||||
// evidence. uiSurface is latched once any visit captured a rendered (image)
|
||||
// artifact. captureIncomplete counts visits whose file-change capture was
|
||||
// missing, partial or unavailable; any such visit forbids the server skip
|
||||
// fallback. All are folded in by recordChanges (ui_surface.go).
|
||||
changedPaths []string
|
||||
changedCount int
|
||||
uiSurface bool
|
||||
captureIncomplete int
|
||||
// capturedPaths is the full, deduplicated, first-seen-ordered set of the
|
||||
// run's server-captured changed paths, distinct from the bounded
|
||||
// changedPaths sample used only by the skip reason. capturedSeen is the
|
||||
@@ -1020,7 +1023,7 @@ func rebuildRunState(detail *protocol.RunDetail, decided protocol.RunStep) *runS
|
||||
}
|
||||
// Seed the UI-surface accumulator from the persisted evidence so a
|
||||
// resumed run reaches the same skip decision with no extra query.
|
||||
state.recordChanges(step.FileChanges, step.Artifacts)
|
||||
state.recordPersistedStep(step)
|
||||
// Only completed, recorded visits are eligible for a later stage's
|
||||
// review capability. The path is reconstructed by the runner below.
|
||||
if step.Status == "completed" {
|
||||
@@ -1062,7 +1065,7 @@ func rebuildRetryState(detail *protocol.RunDetail, failed protocol.RunStep, hand
|
||||
state.turns = append(state.turns, turn.MemoryEntry{Stage: step.Stage, Visit: state.visits[step.Stage], Output: turnOutput(step.Output), PublicFiles: added})
|
||||
state.lastOutput = turnOutput(step.Output)
|
||||
}
|
||||
state.recordChanges(step.FileChanges, step.Artifacts)
|
||||
state.recordPersistedStep(step)
|
||||
if step.Status == "completed" {
|
||||
state.artifactDirs = append(state.artifactDirs, filepath.Join("", strconv.Itoa(step.Sequence)))
|
||||
}
|
||||
@@ -1073,6 +1076,9 @@ func rebuildRetryState(detail *protocol.RunDetail, failed protocol.RunStep, hand
|
||||
// retry source's immediate steps (which is what made retry-of-retry fail
|
||||
// closed with "memory source ... has not completed").
|
||||
if prefix, ok := handoff["prefix"].(map[string]any); ok {
|
||||
// The checkpoint carries outputs, not file-change captures, so an inherited
|
||||
// prefix is unobserved evidence: the server skip fallback must not apply.
|
||||
state.captureIncomplete++
|
||||
for _, record := range retryPrefixRecords(handoff["prefix_provenance"]) {
|
||||
output, exists := prefix[record.stage].(map[string]any)
|
||||
if !exists {
|
||||
@@ -1595,7 +1601,7 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
}
|
||||
}
|
||||
_, declaresSkip := def.Edges["skipped"]
|
||||
skip := !isHuman && declaresSkip && !state.uiSurface
|
||||
skip := !isHuman && declaresSkip && autoSkipEligible(def.Configs, state)
|
||||
state.nextSequence = sequence + 1
|
||||
state.visits[key]++
|
||||
// Publish the run's server-captured changed paths into the input memory
|
||||
@@ -1687,13 +1693,15 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
r.deliverRunNotificationBatch(time.Now(), 1)
|
||||
return
|
||||
}
|
||||
// A stage that declares a `skipped` route is short-circuited when the run
|
||||
// has recorded no UI surface: the stage would answer exactly that, so skip
|
||||
// the model call and its capture work, record the step as completed and
|
||||
// route along the declared edge. Only a declared route is honoured, and no
|
||||
// other stage or outcome routing changes.
|
||||
// Server skip fallback: a stage that declares a `skipped` route is
|
||||
// short-circuited only when the run's evidence is complete and empty
|
||||
// (autoSkipEligible, ui_surface.go): every visit recorded a complete
|
||||
// capture with no file changes and no rendered artifact. Then skip the
|
||||
// model call and its capture work, record the step as completed with a
|
||||
// verdict that names the server fallback, and route along the declared
|
||||
// edge. Partial, unavailable or missing evidence always runs the model.
|
||||
if skip {
|
||||
reason := skipReasonLine(state)
|
||||
reason := skipReasonLine()
|
||||
step := protocol.RunStep{
|
||||
Sequence: sequence,
|
||||
Stage: key,
|
||||
@@ -1702,6 +1710,7 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
Output: stage.Values{"summary": reason},
|
||||
Target: def.Edges["skipped"],
|
||||
Control: &protocol.ControlEvidence{Visit: state.visits[key], MaxVisits: maxVisits, Memory: reads},
|
||||
Verdict: &protocol.StageVerdict{Reason: reason, ReasonSource: protocol.VerdictReasonFallback},
|
||||
}
|
||||
status, outcome, diagnostic, terminal, finishErr := r.finishStage(id, name, workspace, selected, state, step)
|
||||
if finishErr != nil {
|
||||
|
||||
+154
-72
@@ -2,12 +2,15 @@ package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/filediff"
|
||||
"git.a-wan.net/alan/pipelines/internal/pipeline"
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/stage"
|
||||
@@ -30,6 +33,12 @@ func uiReviewerSource(withSkip bool) string {
|
||||
// then returns the persisted detail, the visited keys and the number of verdict
|
||||
// (model) calls the runner made.
|
||||
func runUISkip(t *testing.T, source string, files map[string]string) (*protocol.RunDetail, []string, int) {
|
||||
t.Helper()
|
||||
return runUISkipIn(t, source, files, t.TempDir())
|
||||
}
|
||||
|
||||
// runUISkipIn is runUISkip against a prepared workspace directory.
|
||||
func runUISkipIn(t *testing.T, source string, files map[string]string, dir string) (*protocol.RunDetail, []string, int) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
@@ -50,7 +59,6 @@ func runUISkip(t *testing.T, source string, files map[string]string) (*protocol.
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer r.wg.Wait()
|
||||
dir := t.TempDir()
|
||||
var mu sync.Mutex
|
||||
var visited []string
|
||||
verdictCalls := 0
|
||||
@@ -106,37 +114,28 @@ func executed(visited []string, key string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// TestUISkipShortCircuitsWhenNoSurfaceChanged proves a non-UI change skips the
|
||||
// ui_reviewer stage with no model call and routes along its declared skip edge.
|
||||
func TestUISkipShortCircuitsWhenNoSurfaceChanged(t *testing.T) {
|
||||
detail, visited, verdictCalls := runUISkip(t, uiReviewerSource(true), map[string]string{"implement": "internal/server/fake.go"})
|
||||
if executed(visited, "ui_reviewer") {
|
||||
t.Fatalf("ui_reviewer executed a model call; visited=%v", visited)
|
||||
// assertModelRan fails unless ui_reviewer executed its model and routed through
|
||||
// its own outcome rather than the server fallback.
|
||||
func assertModelRan(t *testing.T, detail *protocol.RunDetail, visited []string, verdictCalls int) {
|
||||
t.Helper()
|
||||
if !executed(visited, "ui_reviewer") {
|
||||
t.Fatalf("ui_reviewer was auto-skipped; visited=%v steps=%+v", visited, detail.Steps)
|
||||
}
|
||||
if verdictCalls != 1 {
|
||||
t.Fatalf("verdict (model) calls = %d, want only implement's 1", verdictCalls)
|
||||
if verdictCalls != 2 {
|
||||
t.Fatalf("verdict (model) calls = %d, want 2 for both executed stages", verdictCalls)
|
||||
}
|
||||
if detail.Status != "succeeded" {
|
||||
t.Fatalf("run status = %q, want succeeded: %+v", detail.Status, detail.RunSummary)
|
||||
}
|
||||
if len(detail.Steps) != 2 {
|
||||
t.Fatalf("steps = %d, want 2: %+v", len(detail.Steps), detail.Steps)
|
||||
}
|
||||
step := detail.Steps[1]
|
||||
if step.Stage != "ui_reviewer" || step.Status != "completed" || step.Outcome != "skipped" || step.Target != protocol.SuccessEnd {
|
||||
t.Fatalf("skip step wrong: %+v", step)
|
||||
}
|
||||
if step.FileChanges != nil && len(step.FileChanges.Files) != 0 {
|
||||
t.Fatalf("skip step recorded file changes: %+v", step.FileChanges)
|
||||
}
|
||||
reason, _ := step.Output["summary"].(string)
|
||||
if !strings.Contains(reason, "internal/server/fake.go") {
|
||||
t.Fatalf("skip reason does not name the checked path: %q", reason)
|
||||
step := detail.Steps[len(detail.Steps)-1]
|
||||
if step.Stage != "ui_reviewer" || step.Outcome != "approved" || step.Target != protocol.SuccessEnd {
|
||||
t.Fatalf("executed step did not route through approved: %+v", step)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUISkipZeroChangesStillSkips proves the guard also short-circuits when the
|
||||
// run recorded no file changes at all, with a reason that says so.
|
||||
// TestUISkipZeroChangesStillSkips proves the fallback short-circuits when every
|
||||
// visit recorded a complete capture with no file changes, and that the verdict
|
||||
// says it was a server fallback with no model run.
|
||||
func TestUISkipZeroChangesStillSkips(t *testing.T) {
|
||||
detail, visited, verdictCalls := runUISkip(t, uiReviewerSource(true), nil)
|
||||
if executed(visited, "ui_reviewer") {
|
||||
@@ -148,48 +147,127 @@ func TestUISkipZeroChangesStillSkips(t *testing.T) {
|
||||
if detail.Status != "succeeded" || len(detail.Steps) != 2 {
|
||||
t.Fatalf("run = %q steps=%d, want succeeded/2", detail.Status, len(detail.Steps))
|
||||
}
|
||||
if first := detail.Steps[0].FileChanges; first == nil || first.Status != "complete" {
|
||||
t.Fatalf("implement capture = %+v, want complete", first)
|
||||
}
|
||||
step := detail.Steps[1]
|
||||
if step.Outcome != "skipped" {
|
||||
t.Fatalf("step outcome = %q, want skipped: %+v", step.Outcome, step)
|
||||
if step.Stage != "ui_reviewer" || step.Status != "completed" || step.Outcome != "skipped" || step.Target != protocol.SuccessEnd {
|
||||
t.Fatalf("skip step wrong: %+v", step)
|
||||
}
|
||||
if step.FileChanges != nil {
|
||||
t.Fatalf("skip step recorded a capture: %+v", step.FileChanges)
|
||||
}
|
||||
if step.Verdict == nil || step.Verdict.ReasonSource != protocol.VerdictReasonFallback || !strings.HasPrefix(step.Verdict.Reason, serverSkipReasonPrefix) {
|
||||
t.Fatalf("skip verdict does not name the server fallback: %+v", step.Verdict)
|
||||
}
|
||||
reason, _ := step.Output["summary"].(string)
|
||||
if !strings.Contains(reason, "no recorded file changes") {
|
||||
if !strings.Contains(reason, "recorded no file changes") {
|
||||
t.Fatalf("zero-change reason does not state that: %q", reason)
|
||||
}
|
||||
if len(detail.Warnings) != 0 {
|
||||
t.Fatalf("complete run carries warnings: %v", detail.Warnings)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUISkipRunsStageOnSurfaceChange proves a run that touched a UI surface
|
||||
// still executes the stage and routes through its normal outcome.
|
||||
func TestUISkipRunsStageOnSurfaceChange(t *testing.T) {
|
||||
// TestUISkipRunsModelOnAnyRecordedChange proves no path is classified as "not
|
||||
// UI": a backend file, a Flutter .dart file, a Kotlin file, a Go TUI file and a
|
||||
// web .tsx file all make the stage run its model. The fallback never encodes a
|
||||
// repository layout.
|
||||
func TestUISkipRunsModelOnAnyRecordedChange(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"android": "android/app/src/main/kotlin/Main.kt",
|
||||
"internal tui": "internal/tui/view.go",
|
||||
"presentation": "web/app.tsx",
|
||||
"go backend": "internal/server/fake.go",
|
||||
"flutter dart": "app/lib/features/home/home_page.dart",
|
||||
"kotlin": "android/app/src/main/kotlin/Main.kt",
|
||||
"go tui": "internal/tui/view.go",
|
||||
"web tsx": "web/app.tsx",
|
||||
"docs": "docs/readme.md",
|
||||
}
|
||||
for name, path := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
detail, visited, verdictCalls := runUISkip(t, uiReviewerSource(true), map[string]string{"implement": path})
|
||||
if !executed(visited, "ui_reviewer") {
|
||||
t.Fatalf("ui_reviewer did not execute on %s; visited=%v", path, visited)
|
||||
}
|
||||
if verdictCalls != 2 {
|
||||
t.Fatalf("verdict (model) calls = %d, want 2 for both executed stages", verdictCalls)
|
||||
}
|
||||
if detail.Status != "succeeded" {
|
||||
t.Fatalf("run status = %q, want succeeded: %+v", detail.Status, detail.RunSummary)
|
||||
}
|
||||
step := detail.Steps[len(detail.Steps)-1]
|
||||
if step.Stage != "ui_reviewer" || step.Outcome != "approved" || step.Target != protocol.SuccessEnd {
|
||||
t.Fatalf("executed step did not route through approved: %+v", step)
|
||||
}
|
||||
assertModelRan(t, detail, visited, verdictCalls)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestUISkipPartialCaptureNeverSkips proves a visit whose capture was partial
|
||||
// (here a protected .env file forces skipped>0) blocks the fallback even when
|
||||
// it recorded no file changes, and that inspect carries a run-level warning.
|
||||
func TestUISkipPartialCaptureNeverSkips(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, ".env"), []byte("A=1\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
detail, visited, verdictCalls := runUISkipIn(t, uiReviewerSource(true), nil, dir)
|
||||
if first := detail.Steps[0].FileChanges; first == nil || first.Status != "partial" || len(first.Files) != 0 {
|
||||
t.Fatalf("fixture capture = %+v, want partial with no files", first)
|
||||
}
|
||||
assertModelRan(t, detail, visited, verdictCalls)
|
||||
if len(detail.Warnings) != 1 || !strings.Contains(detail.Warnings[0], "implement #1 (partial)") {
|
||||
t.Fatalf("partial capture warning missing: %v", detail.Warnings)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUISkipOverBudgetGitWorkspaceRunsModel is the run-236 regression: in a Git
|
||||
// workspace whose text exceeds the snapshot budget, a stage edits one .tsx file
|
||||
// sorted after the budget is exhausted. The capture must list that file
|
||||
// (without a patch) and the next stage, which declares `skipped`, must run its
|
||||
// model rather than be auto-routed with "no recorded file changes".
|
||||
func TestUISkipOverBudgetGitWorkspaceRunsModel(t *testing.T) {
|
||||
if _, err := exec.LookPath("git"); err != nil {
|
||||
t.Skip("git unavailable")
|
||||
}
|
||||
restore := filediff.SetSnapshotBudgetForTesting(256)
|
||||
defer restore()
|
||||
dir := t.TempDir()
|
||||
git := func(args ...string) {
|
||||
t.Helper()
|
||||
cmd := exec.Command("git", append([]string{"-C", dir, "-c", "user.email=t@example.invalid", "-c", "user.name=t", "-c", "commit.gpgsign=false"}, args...)...)
|
||||
if data, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("%v: %v %s", args, err, data)
|
||||
}
|
||||
}
|
||||
git("init", "-q")
|
||||
target := "web/src/pages/inbox/PortfolioIntelligenceInbox.test.tsx"
|
||||
if err := os.MkdirAll(filepath.Join(dir, filepath.Dir(target)), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 40; i++ {
|
||||
if err := os.WriteFile(filepath.Join(dir, fmt.Sprintf("a-%02d.txt", i)), []byte(strings.Repeat("filler line\n", 4)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, filepath.FromSlash(target)), []byte(strings.Repeat("original test line\n", 8)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
git("add", ".")
|
||||
git("commit", "-qm", "fixture")
|
||||
detail, visited, verdictCalls := runUISkipIn(t, uiReviewerSource(true), map[string]string{"implement": target}, dir)
|
||||
capture := detail.Steps[0].FileChanges
|
||||
if capture == nil || len(capture.Files) != 1 || capture.Files[0].Path != target || !capture.Files[0].NoPatch {
|
||||
t.Fatalf("over-budget capture did not list the edited file: %+v", capture)
|
||||
}
|
||||
assertModelRan(t, detail, visited, verdictCalls)
|
||||
if len(detail.Warnings) == 0 {
|
||||
t.Fatal("partial capture produced no run-level warning")
|
||||
}
|
||||
}
|
||||
|
||||
// TestUISkipOptOut proves auto_skip=false makes a `skipped` stage run its model
|
||||
// even on complete, empty evidence.
|
||||
func TestUISkipOptOut(t *testing.T) {
|
||||
source := uiReviewerSource(true)
|
||||
at := strings.LastIndex(source, "timeout=1h\n")
|
||||
source = source[:at] + "timeout=1h\nauto_skip=false\n" + source[at+len("timeout=1h\n"):]
|
||||
detail, visited, verdictCalls := runUISkip(t, source, nil)
|
||||
assertModelRan(t, detail, visited, verdictCalls)
|
||||
}
|
||||
|
||||
// TestUISkipNeedsDeclaredEdge proves the guard never invents a skip for a stage
|
||||
// whose definition declares no `skipped` route: it always executes.
|
||||
// whose definition declares no `skipped` route: it always executes, even on
|
||||
// complete, empty evidence.
|
||||
func TestUISkipNeedsDeclaredEdge(t *testing.T) {
|
||||
detail, visited, verdictCalls := runUISkip(t, uiReviewerSource(false), map[string]string{"implement": "internal/server/fake.go"})
|
||||
detail, visited, verdictCalls := runUISkip(t, uiReviewerSource(false), nil)
|
||||
if !executed(visited, "ui_reviewer") {
|
||||
t.Fatalf("ui_reviewer without a skipped edge did not execute; visited=%v", visited)
|
||||
}
|
||||
@@ -205,33 +283,37 @@ func TestUISkipNeedsDeclaredEdge(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestUISurfacePathClassification(t *testing.T) {
|
||||
cases := []struct {
|
||||
path string
|
||||
want bool
|
||||
}{
|
||||
{"android/app/src/main/kotlin/Main.kt", true},
|
||||
{"android/README.md", true},
|
||||
{"internal/tui/view.go", true},
|
||||
{"internal/tui", true},
|
||||
{"web/index.html", true},
|
||||
{"web/app.css", true},
|
||||
{"web/app.js", true},
|
||||
{"web/app.ts", true},
|
||||
{"web/app.jsx", true},
|
||||
{"web/app.tsx", true},
|
||||
{"web/Index.HTML", true},
|
||||
{"internal/server/fake.go", false},
|
||||
{"androidx/foo.go", false},
|
||||
{"cmd/pipelines/main.go", false},
|
||||
{"docs/readme.md", false},
|
||||
{"README.md", false},
|
||||
// TestUISkipPersistedEvidence proves the resume/retry seed: a stored server
|
||||
// fallback step neither adds nor breaks evidence, while a stored visit with no
|
||||
// capture (a human decision, a crash, a historical row) or a partial one makes
|
||||
// the evidence incomplete.
|
||||
func TestUISkipPersistedEvidence(t *testing.T) {
|
||||
complete := &protocol.StageFileChanges{Status: "complete", Files: []protocol.FileChange{}}
|
||||
reason := skipReasonLine()
|
||||
serverSkip := protocol.RunStep{Stage: "ui", Outcome: "skipped", Verdict: &protocol.StageVerdict{Reason: reason, ReasonSource: protocol.VerdictReasonFallback}}
|
||||
s := &runState{}
|
||||
s.recordPersistedStep(protocol.RunStep{Stage: "implement", FileChanges: complete})
|
||||
s.recordPersistedStep(serverSkip)
|
||||
if !autoSkipEligible(nil, s) {
|
||||
t.Fatalf("complete evidence plus a server skip is not eligible: %+v", s)
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := uiSurfacePath(tc.path); got != tc.want {
|
||||
t.Errorf("uiSurfacePath(%q) = %v, want %v", tc.path, got, tc.want)
|
||||
legacy := serverSkip
|
||||
legacy.Verdict = &protocol.StageVerdict{Reason: "no UI surface changed (no recorded file changes)", ReasonSource: protocol.VerdictReasonFallback}
|
||||
for name, step := range map[string]protocol.RunStep{
|
||||
"no capture": {Stage: "gate", Outcome: "approve"},
|
||||
"partial": {Stage: "implement", FileChanges: &protocol.StageFileChanges{Status: "partial", Files: []protocol.FileChange{}}},
|
||||
"unavailable": {Stage: "implement", FileChanges: &protocol.StageFileChanges{Status: "unavailable"}},
|
||||
"legacy skip row": legacy,
|
||||
} {
|
||||
s := &runState{}
|
||||
s.recordPersistedStep(step)
|
||||
if autoSkipEligible(nil, s) {
|
||||
t.Errorf("%s: incomplete evidence is eligible for the fallback", name)
|
||||
}
|
||||
}
|
||||
if autoSkipEligible(map[string]string{"auto_skip": "false"}, &runState{}) {
|
||||
t.Error("auto_skip=false did not opt out")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUISurfaceArtifactClassification(t *testing.T) {
|
||||
|
||||
@@ -3,36 +3,35 @@ package server
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
// UI-surface classification is the single source of truth for the deterministic
|
||||
// pre-check that lets a stage declaring a `skipped` route short-circuit when the
|
||||
// run changed no rendered surface. The path/extension lists live only here; the
|
||||
// skip guard, its reason and the resumed-run seed all read them.
|
||||
// The server skip fallback lets a stage that declares a `skipped` route
|
||||
// short-circuit, without a model call, only when the run's evidence proves there
|
||||
// is nothing to review: every visit so far recorded a complete file-change
|
||||
// capture, none of them recorded any change, and no visit captured a rendered
|
||||
// (image) artifact. Anything else runs the stage's model, which may still choose
|
||||
// `skipped` itself.
|
||||
//
|
||||
// The rule deliberately classifies no path as "not UI". A path/extension list
|
||||
// encodes one repository's layout (Go TUIs, Flutter .dart, Kotlin, SwiftUI,
|
||||
// templates, strings files and config can all be rendered surface), and absent
|
||||
// or partial evidence must never be read as "no edits". A wrong run costs one
|
||||
// model call; a wrong skip ships an unreviewed UI change.
|
||||
|
||||
// maxRetainedChangedPaths bounds how many recorded changed paths a runState
|
||||
// retains for the skip reason. The full count is kept separately, so the reason
|
||||
// stays informative without unbounded memory.
|
||||
// retains as a sample. The full count is kept separately.
|
||||
const maxRetainedChangedPaths = 32
|
||||
|
||||
// uiSurfaceDirs are the workspace prefixes whose changes are a user-facing
|
||||
// surface. Matching is on a path boundary, so "androidx/" is not "android/".
|
||||
var uiSurfaceDirs = []string{"android", "internal/tui"}
|
||||
// serverSkipReasonPrefix starts every server fallback skip reason, so the
|
||||
// verdict says plainly that no model ran, and so a resumed run recognises its
|
||||
// own earlier fallback steps (which have no capture of their own).
|
||||
const serverSkipReasonPrefix = "Server fallback skip (no model run):"
|
||||
|
||||
// presentationExtensions are the rendered presentation file types a change can
|
||||
// touch without living under a surface directory.
|
||||
var presentationExtensions = map[string]bool{
|
||||
".html": true,
|
||||
".css": true,
|
||||
".js": true,
|
||||
".ts": true,
|
||||
".jsx": true,
|
||||
".tsx": true,
|
||||
}
|
||||
// autoSkipConfig is the stage config that opts a stage out of the fallback.
|
||||
const autoSkipConfig = "auto_skip"
|
||||
|
||||
// renderedArtifactExtensions are the image types a captured artifact may carry
|
||||
// when its content type was not recorded or sniffed as an image.
|
||||
@@ -44,22 +43,6 @@ var renderedArtifactExtensions = map[string]bool{
|
||||
".webp": true,
|
||||
}
|
||||
|
||||
// uiSurfacePath reports whether a run-relative, forward-slash workspace path
|
||||
// changed a user-facing surface: a file under android/ or internal/tui/, or a
|
||||
// file with a presentation extension. It is O(len(path)) with no I/O.
|
||||
func uiSurfacePath(p string) bool {
|
||||
clean := strings.TrimPrefix(path.Clean(filepath.ToSlash(p)), "/")
|
||||
if clean == "" || clean == "." {
|
||||
return false
|
||||
}
|
||||
for _, dir := range uiSurfaceDirs {
|
||||
if clean == dir || strings.HasPrefix(clean, dir+"/") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return presentationExtensions[strings.ToLower(path.Ext(clean))]
|
||||
}
|
||||
|
||||
// uiSurfaceArtifact reports whether a captured artifact is a rendered surface:
|
||||
// an image content type, or an image file extension when no type was recorded.
|
||||
func uiSurfaceArtifact(a protocol.StageArtifact) bool {
|
||||
@@ -69,19 +52,20 @@ func uiSurfaceArtifact(a protocol.StageArtifact) bool {
|
||||
return renderedArtifactExtensions[strings.ToLower(path.Ext(a.Name))]
|
||||
}
|
||||
|
||||
// recordChanges folds one completed step's observed changes into the run's
|
||||
// UI-surface decision. It retains a bounded sample of paths; the count is
|
||||
// unbounded so the reason can state the true total. It is O(changes) with no I/O.
|
||||
// recordChanges folds one executed visit's observed changes into the run's
|
||||
// skip evidence. A nil, partial or unavailable capture marks the run's
|
||||
// evidence incomplete for the rest of the run. It retains a bounded sample of
|
||||
// paths; the count is unbounded. It is O(changes) with no I/O.
|
||||
func (s *runState) recordChanges(changes *protocol.StageFileChanges, artifacts []protocol.StageArtifact) {
|
||||
if changes == nil || changes.Status != "complete" {
|
||||
s.captureIncomplete++
|
||||
}
|
||||
if changes != nil {
|
||||
for _, file := range changes.Files {
|
||||
s.changedCount++
|
||||
if len(s.changedPaths) < maxRetainedChangedPaths {
|
||||
s.changedPaths = append(s.changedPaths, file.Path)
|
||||
}
|
||||
if !s.uiSurface && uiSurfacePath(file.Path) {
|
||||
s.uiSurface = true
|
||||
}
|
||||
// The full captured set is a separate, unbounded dedupe of the actual
|
||||
// evidence, unlike the bounded changedPaths sample above.
|
||||
if file.Path == "" || s.capturedSeen[file.Path] {
|
||||
@@ -101,16 +85,38 @@ func (s *runState) recordChanges(changes *protocol.StageFileChanges, artifacts [
|
||||
}
|
||||
}
|
||||
|
||||
// skipReasonLine states, as one bounded line, why the guard skipped a stage: no
|
||||
// recorded change touched a UI surface. It names the retained checked paths.
|
||||
func skipReasonLine(state *runState) string {
|
||||
if state.changedCount == 0 {
|
||||
return "no UI surface changed (no recorded file changes)"
|
||||
// recordPersistedStep seeds the skip evidence from a stored step when a run is
|
||||
// resumed or retried. A server fallback skip executed nothing and carries no
|
||||
// capture, so it neither adds evidence nor makes it incomplete; every other
|
||||
// stored visit (including a human decision, during which the workspace was not
|
||||
// observed) is folded in like a live visit.
|
||||
func (s *runState) recordPersistedStep(step protocol.RunStep) {
|
||||
if serverSkipStep(step) {
|
||||
return
|
||||
}
|
||||
line := fmt.Sprintf("no UI surface changed in %d recorded file change(s): %s", state.changedCount, strings.Join(state.changedPaths, ", "))
|
||||
if extra := state.changedCount - len(state.changedPaths); extra > 0 {
|
||||
line = fmt.Sprintf("%s (+%d more)", line, extra)
|
||||
s.recordChanges(step.FileChanges, step.Artifacts)
|
||||
}
|
||||
|
||||
// serverSkipStep reports whether a stored step is a server fallback skip.
|
||||
func serverSkipStep(step protocol.RunStep) bool {
|
||||
return step.Outcome == "skipped" && step.FileChanges == nil && step.Verdict != nil &&
|
||||
step.Verdict.ReasonSource == protocol.VerdictReasonFallback &&
|
||||
strings.HasPrefix(step.Verdict.Reason, serverSkipReasonPrefix)
|
||||
}
|
||||
|
||||
// autoSkipEligible reports whether the server may skip a stage declaring a
|
||||
// `skipped` route: the stage has not opted out, every visit so far recorded a
|
||||
// complete capture, no change was recorded and no rendered artifact captured.
|
||||
func autoSkipEligible(configs map[string]string, state *runState) bool {
|
||||
if configs[autoSkipConfig] == "false" {
|
||||
return false
|
||||
}
|
||||
return state.captureIncomplete == 0 && state.changedCount == 0 && !state.uiSurface
|
||||
}
|
||||
|
||||
// skipReasonLine states, as one bounded line, why the server skipped a stage.
|
||||
func skipReasonLine() string {
|
||||
line := fmt.Sprintf("%s every earlier visit's file-change capture was complete and recorded no file changes or screenshots.", serverSkipReasonPrefix)
|
||||
runes := []rune(line)
|
||||
if len(runes) > protocol.VerdictReasonCap {
|
||||
return string(runes[:protocol.VerdictReasonCap])
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
package stage
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestAutoSkipConfig(t *testing.T) {
|
||||
for value, ok := range map[string]bool{"": true, "true": true, "false": true, "no": false, "FALSE": false, "0": false} {
|
||||
def := Definition{Type: ModelType, Blocks: map[string]string{"prompt": "task", "configs": "config"}, Configs: map[string]string{"provider": "fixture", "model": "model", "auto_skip": value}}
|
||||
if _, err := (ModelStage{}).Spec(def); (err == nil) != ok {
|
||||
t.Errorf("auto_skip=%q: err=%v, want ok=%v", value, err, ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -79,7 +79,7 @@ func (ModelStage) Spec(def Definition) (ModelSpec, error) {
|
||||
}
|
||||
for key := range def.Configs {
|
||||
switch key {
|
||||
case "provider", "model", "reasoning_effort", "retries", "timeout", "budget_usd", "budget_tokens", "harness", "delivery":
|
||||
case "provider", "model", "reasoning_effort", "retries", "timeout", "budget_usd", "budget_tokens", "harness", "delivery", "auto_skip":
|
||||
default:
|
||||
return ModelSpec{}, fmt.Errorf("unknown config %q", key)
|
||||
}
|
||||
@@ -101,6 +101,11 @@ func (ModelStage) Spec(def Definition) (ModelSpec, error) {
|
||||
if delivery := def.Configs["delivery"]; delivery != "" && delivery != "true" {
|
||||
return ModelSpec{}, fmt.Errorf("delivery must be true when set")
|
||||
}
|
||||
// auto_skip=false opts a stage with a `skipped` edge out of the server skip
|
||||
// fallback (the runner reads it); true is the default and may be explicit.
|
||||
if value := def.Configs["auto_skip"]; value != "" && value != "true" && value != "false" {
|
||||
return ModelSpec{}, fmt.Errorf("auto_skip must be true or false")
|
||||
}
|
||||
if def.Configs["delivery"] == "true" && backend != "opencode" {
|
||||
return ModelSpec{}, fmt.Errorf("delivery is supported only by the opencode harness")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user