Run every pipeline run in its own worktree

A fresh run started from a repository's main checkout gets its own
worktree and branch next to the repository, so runs never share a tree
or see the operator's uncommitted work, and several can run at once.
Retries and runs started in a linked worktree keep their directory. A
clean, landed run tree is removed after success; anything else is kept.

The delivery gate commits what the land stage left uncommitted in a run
worktree, and no longer requires the operator's main checkout to be
clean: Git's fast-forward refuses safely when a local change conflicts.
This commit is contained in:
2026-11-10 16:06:40 +00:00
parent d77b36f56f
commit 532ba48927
9 changed files with 537 additions and 29 deletions
+36
View File
@@ -0,0 +1,36 @@
# Every run in its own worktree (2026-11-10)
## Why
Code-change runs almost never shipped. Most failures were delivery mechanics,
not code: runs started from the shared checkout edited it directly, so
- only one run could use a repository at a time;
- the delivery gate refused because the operator's checkout had unrelated
modified or untracked files (runs 257, 260), or found no linked item
worktree at all (256);
- the land stage, a model committing from an incomplete changed-file list,
committed part of the work or stopped as blocked (268, 291, 298).
## What changed
- `internal/server/run_worktree.go`: a fresh run whose workspace is a
repository's main checkout gets `<repo>-runs/<slug>` on branch
`pipelines/run-<slug>` from `HEAD` (a subdirectory maps to the same
subdirectory). Linked worktrees and retries are used as they are. A refused
or duplicate admission discards the new tree. After a successful run, a clean
tree whose branch is on `main` is removed with its branch.
- `internal/server/delivery_gate.go`: in a run worktree the gate commits what
the land stage left uncommitted (`<title>\n\nPipelines run N.`), so a partial
file list can no longer drop work. The main checkout no longer has to be
clean; `git merge --ff-only` refuses on its own, changing nothing, when a
local change would be overwritten (the error says the item is checked and
ready on its branch).
- Tests: `run_worktree_test.go`; two diff-capture tests now start from a linked
worktree, since their fake stages edit the original directory.
## Not changed
- Untracked environment files (`.envrc`, `.tool-versions`) are not copied
into run trees; this repository has none.
- Merge conflicts with a moved main still fail the gate with the branch kept.
+17 -1
View File
@@ -161,10 +161,26 @@ A workflow that commits its change ends in `land`: a model stage with
3. No empty commit, push, amend or rebase. Report the hash.
With `delivery=true` and `checks` commands, after `land` succeeds the server:
requires the item and main checkouts to be clean, untracked files included; merges
commits whatever the run left uncommitted when the item is the run's own
worktree (see "Run worktrees" below); requires the item checkout to be clean,
untracked files included (the main checkout need not be: the final
fast-forward is Git's `merge --ff-only`, which refuses without touching
anything when a local change would be overwritten); merges
a moved `main` into the item (a conflict aborts with the workspace unchanged);
runs each check with `sh -c` from the root of a fresh detached worktree of the
commit, so include dependency bootstrap (`npm ci && npm test`); caps each check's
output at 64 KB; and fast-forwards `main`. A failing check blocks landing. Do not
check ancestry before the gate runs. Without a gate, the manager merges and
verifies ancestry; no separate merger pipeline is needed (`references/queue.md`).
## Run worktrees
A run started from a repository's main checkout runs in its own worktree:
`<repo>-runs/<date>-<title>-<hex>` next to the repository, on branch
`pipelines/run-<same>`, from the checkout's current `HEAD`. Runs never share a
tree, never see the operator's uncommitted work, and any number can run at
once. A run started inside a linked worktree (an item's tree) and every retry
use their directory as is. After a successful run whose tree is clean and
whose branch is on `main`, the server removes the tree and branch; anything
else (a failed run, unlanded commits, stray files) is kept for recovery.
+44 -18
View File
@@ -35,7 +35,14 @@ const deliveryCleanupTimeout = 10 * time.Second
// fast-forwards main. A conflicting merge is aborted, leaving the workspace
// exactly as it was. The returned note is non-empty when a catch-up merge
// happened, so the caller can record it.
func verifyDelivery(ctx context.Context, workspace string, access stage.LandingAccess, checks []string) (string, error) {
//
// Work the land stage left uncommitted in the item workspace is committed
// first (commitMessage): the workspace is the run's own tree, so everything in
// it is the run's work, and a partial file list must never drop part of it.
// The operator's main checkout need not be clean: the fast-forward is Git's
// own `merge --ff-only`, which refuses without touching anything when a local
// change or untracked file would be overwritten.
func verifyDelivery(ctx context.Context, workspace string, access stage.LandingAccess, checks []string, commitMessage string) (string, error) {
lock, err := acquireDeliveryLock(ctx, access.LockPath)
if err != nil {
return "", err
@@ -48,14 +55,15 @@ func verifyDelivery(ctx context.Context, workspace string, access stage.LandingA
if current != access {
return "", fmt.Errorf("delivery access changed during stage")
}
for _, directory := range []string{workspace, access.MainWorktree} {
status, err := gitOutput(directory, "status", "--porcelain=v1", "--untracked-files=all")
if err != nil {
return "", fmt.Errorf("delivery checkout status: %w", err)
}
if status != "" {
return "", fmt.Errorf("delivery checkout is not clean: %s", strings.Split(status, "\n")[0])
}
if err := commitDeliveryLeftovers(ctx, workspace, commitMessage); err != nil {
return "", err
}
status, err := gitOutput(workspace, "status", "--porcelain=v1", "--untracked-files=all")
if err != nil {
return "", fmt.Errorf("delivery checkout status: %w", err)
}
if status != "" {
return "", fmt.Errorf("delivery checkout is not clean: %s", strings.Split(status, "\n")[0])
}
itemHead, err := gitOutput(workspace, "rev-parse", "HEAD")
if err != nil {
@@ -84,16 +92,14 @@ func verifyDelivery(ctx context.Context, workspace string, access stage.LandingA
return note, fmt.Errorf("delivery item commit cannot fast-forward canonical main")
}
if output, err := exec.CommandContext(ctx, "git", "-C", access.MainWorktree, "merge", "--ff-only", itemHead).CombinedOutput(); err != nil {
return note, fmt.Errorf("delivery fast-forward canonical main: %w: %s", err, strings.TrimSpace(string(output)))
return note, fmt.Errorf("delivery fast-forward canonical main (the main checkout has local changes that conflict with the item; the item is checked and ready on its branch): %w: %s", err, strings.TrimSpace(string(output)))
}
for _, directory := range []string{workspace, access.MainWorktree} {
status, err := gitOutput(directory, "status", "--porcelain=v1", "--untracked-files=all")
if err != nil {
return note, fmt.Errorf("delivery settlement status: %w", err)
}
if status != "" {
return note, fmt.Errorf("delivery checkout is not clean after checks: %s", strings.Split(status, "\n")[0])
}
status, err = gitOutput(workspace, "status", "--porcelain=v1", "--untracked-files=all")
if err != nil {
return note, fmt.Errorf("delivery settlement status: %w", err)
}
if status != "" {
return note, fmt.Errorf("delivery checkout is not clean after checks: %s", strings.Split(status, "\n")[0])
}
// Re-derive at the settlement boundary. A success means exactly the checked
// item commit is now reachable from the same canonical main worktree.
@@ -111,6 +117,26 @@ func verifyDelivery(ctx context.Context, workspace string, access stage.LandingA
return note, nil
}
// commitDeliveryLeftovers commits everything still uncommitted in the item
// workspace (tracked and untracked, respecting .gitignore). An empty message
// or a clean tree commits nothing.
func commitDeliveryLeftovers(ctx context.Context, workspace, message string) error {
status, err := gitOutput(workspace, "status", "--porcelain=v1", "--untracked-files=all")
if err != nil {
return fmt.Errorf("delivery checkout status: %w", err)
}
if status == "" || strings.TrimSpace(message) == "" {
return nil
}
if output, err := boundedCommandOutput(ctx, workspace, "add", "--all"); err != nil {
return fmt.Errorf("delivery commit of remaining changes: %w: %s", err, strings.TrimSpace(string(output)))
}
if output, err := boundedCommandOutput(ctx, workspace, "commit", "--no-verify", "-m", message); err != nil {
return fmt.Errorf("delivery commit of remaining changes: %w: %s", err, strings.TrimSpace(string(output)))
}
return nil
}
// catchUpDeliveryItem merges canonical main into the clean item workspace with a
// normal merge commit and returns the new item HEAD. It never rewrites history:
// a failed merge is aborted and the workspace is verified to be back at
+5
View File
@@ -196,6 +196,11 @@ func TestRetainedStageDiffsSurviveCommittedMergeAndNoopVisit(t *testing.T) {
}
runGit(t, workspace, "add", "source.txt")
runGit(t, workspace, "commit", "-qm", "base")
// Run from a linked worktree: a run started from a main checkout would get
// its own worktree, away from the files this fake stage edits.
item := filepath.Join(t.TempDir(), "item")
runGit(t, workspace, "worktree", "add", "-q", "-b", "item", item)
workspace, path = item, filepath.Join(item, "source.txt")
branch := runGitOutput(t, workspace, "branch", "--show-current")
r, err := newRunner(ctx, store)
+8 -8
View File
@@ -83,19 +83,19 @@ func TestVerifyDeliveryRequiresCleanLandedMainAndPassingChecks(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if _, err := verifyDelivery(context.Background(), item, access, []string{"test -f delivered"}); err != nil {
if _, err := verifyDelivery(context.Background(), item, access, []string{"test -f delivered"}, ""); err != nil {
t.Fatalf("complete landed delivery rejected: %v", err)
}
if err := os.WriteFile(filepath.Join(item, "missing-helper"), []byte("dirty\n"), 0600); err != nil {
t.Fatal(err)
}
if _, err := verifyDelivery(context.Background(), item, access, []string{"true"}); err == nil {
if _, err := verifyDelivery(context.Background(), item, access, []string{"true"}, ""); err == nil {
t.Fatal("dirty task-owned helper was accepted")
}
if err := os.Remove(filepath.Join(item, "missing-helper")); err != nil {
t.Fatal(err)
}
if _, err := verifyDelivery(context.Background(), item, access, []string{"false"}); err == nil {
if _, err := verifyDelivery(context.Background(), item, access, []string{"false"}, ""); err == nil {
t.Fatal("failed required check was accepted")
}
if err := os.WriteFile(filepath.Join(item, "unlanded"), []byte("no\n"), 0600); err != nil {
@@ -104,7 +104,7 @@ func TestVerifyDeliveryRequiresCleanLandedMainAndPassingChecks(t *testing.T) {
runGit(t, item, "add", "unlanded")
runGit(t, item, "commit", "-m", "unlanded")
committed := runGitOutput(t, item, "rev-parse", "HEAD")
note, err := verifyDelivery(context.Background(), item, access, []string{"test -f unlanded"})
note, err := verifyDelivery(context.Background(), item, access, []string{"test -f unlanded"}, "")
if err != nil {
t.Fatalf("complete unlanded item was not landed after candidate verification: %v", err)
}
@@ -153,7 +153,7 @@ func TestVerifyDeliveryCatchesUpWithUnrelatedMainChange(t *testing.T) {
t.Fatal(err)
}
// The check only passes on the merged tree, proving it ran after the merge.
note, err := verifyDelivery(context.Background(), item, access, []string{"test -f work && test -f other"})
note, err := verifyDelivery(context.Background(), item, access, []string{"test -f work && test -f other"}, "")
if err != nil {
t.Fatalf("cleanly mergeable item was rejected: %v", err)
}
@@ -183,7 +183,7 @@ func TestVerifyDeliveryAbortsConflictingCatchUp(t *testing.T) {
if err != nil {
t.Fatal(err)
}
_, err = verifyDelivery(context.Background(), item, access, []string{"true"})
_, err = verifyDelivery(context.Background(), item, access, []string{"true"}, "")
if err == nil || !strings.Contains(err.Error(), "main moved and the item no longer merges cleanly: README") {
t.Fatalf("conflicting catch-up did not fail clearly: %v", err)
}
@@ -211,7 +211,7 @@ func TestVerifyDeliveryFailingChecksAfterCatchUpLeaveMainUnchanged(t *testing.T)
t.Fatal(err)
}
// Passes on the item tree alone, fails on the merged tree.
_, err = verifyDelivery(context.Background(), item, access, []string{"test -f work", "test ! -f other"})
_, err = verifyDelivery(context.Background(), item, access, []string{"test -f work", "test ! -f other"}, "")
if err == nil || !strings.Contains(err.Error(), "merged") || !strings.Contains(err.Error(), "test ! -f other") {
t.Fatalf("failing merged-tree check did not fail with catch-up context: %v", err)
}
@@ -246,7 +246,7 @@ func TestVerifyDeliveryFailingCandidateLeavesMainUnchanged(t *testing.T) {
t.Fatal(err)
}
before := runGitOutput(t, main, "rev-parse", "HEAD")
if _, err := verifyDelivery(context.Background(), item, access, []string{"test -f candidate", "false"}); err == nil {
if _, err := verifyDelivery(context.Background(), item, access, []string{"test -f candidate", "false"}, ""); err == nil {
t.Fatal("failing candidate check landed main")
}
if after := runGitOutput(t, main, "rev-parse", "HEAD"); after != before {
+197
View File
@@ -0,0 +1,197 @@
package server
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
)
// runWorktreeBranchPrefix names the branch of every worktree the server creates
// for a run, so cleanup can tell its own trees from an operator's.
const runWorktreeBranchPrefix = "pipelines/run-"
// runWorktreesDirSuffix is appended to a repository's directory name for the
// sibling directory that holds its run worktrees: /repos/app -> /repos/app-runs.
const runWorktreesDirSuffix = "-runs"
// isolateRunWorkspace gives a new run its own Git worktree when it was started
// from a repository's main checkout, so concurrent runs never share a tree and a
// run never sees (or commits) the operator's uncommitted work. A workspace that
// is not in a Git repository, or is already a linked worktree (an item's tree,
// or a retry reusing its run's tree), is returned unchanged. A subdirectory of
// the main checkout maps to the same subdirectory of the new worktree.
//
// The worktree starts at the checkout's current HEAD on a new branch
// pipelines/run-<slug> and lives in <repo>-runs/<slug> next to the repository.
func isolateRunWorkspace(ctx context.Context, workspace, title string) (string, error) {
top, err := gitOutput(workspace, "rev-parse", "--show-toplevel")
if err != nil {
return workspace, nil // Not a Git checkout: nothing to isolate.
}
gitDir, err := gitOutput(workspace, "rev-parse", "--path-format=absolute", "--git-dir")
if err != nil {
return "", fmt.Errorf("run worktree: %w", err)
}
commonDir, err := gitOutput(workspace, "rev-parse", "--path-format=absolute", "--git-common-dir")
if err != nil {
return "", fmt.Errorf("run worktree: %w", err)
}
if filepath.Clean(gitDir) != filepath.Clean(commonDir) {
return workspace, nil // Already a linked worktree.
}
if _, err := gitOutput(workspace, "rev-parse", "--verify", "HEAD"); err != nil {
return workspace, nil // No commit to branch from yet.
}
top, err = filepath.EvalSymlinks(top)
if err != nil {
return "", fmt.Errorf("run worktree: %w", err)
}
rel, err := filepath.Rel(top, workspace)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(os.PathSeparator)) {
return "", fmt.Errorf("run worktree: workspace %s is outside its repository %s", workspace, top)
}
slug := runWorktreeSlug(time.Now(), title)
root := filepath.Join(filepath.Dir(top), filepath.Base(top)+runWorktreesDirSuffix)
if err := os.MkdirAll(root, 0o755); err != nil {
return "", fmt.Errorf("run worktree: %w", err)
}
tree := filepath.Join(root, slug)
branch := runWorktreeBranchPrefix + slug
cmd := exec.CommandContext(ctx, "git", "-C", top, "worktree", "add", "-b", branch, tree, "HEAD")
if output, err := cmd.CombinedOutput(); err != nil {
return "", fmt.Errorf("run worktree: git worktree add %s: %w: %s", tree, err, strings.TrimSpace(string(output)))
}
resolved, err := filepath.EvalSymlinks(tree)
if err != nil {
return "", fmt.Errorf("run worktree: %w", err)
}
return filepath.Join(resolved, rel), nil
}
var runSlugUnsafe = regexp.MustCompile(`[^a-z0-9]+`)
// runWorktreeSlug is a sortable, readable and unique directory/branch name:
// 20261110-1530-short-title-ab12.
func runWorktreeSlug(now time.Time, title string) string {
words := strings.Trim(runSlugUnsafe.ReplaceAllString(strings.ToLower(title), "-"), "-")
if len(words) > 40 {
words = strings.TrimRight(words[:40], "-")
}
random := make([]byte, 2)
_, _ = rand.Read(random)
parts := []string{now.UTC().Format("20060102-1504")}
if words != "" {
parts = append(parts, words)
}
parts = append(parts, hex.EncodeToString(random))
return strings.Join(parts, "-")
}
// removeLandedRunWorktree deletes a run worktree the server created once its
// work is safe: the tree is clean and its branch is already in the main
// main branch (landed, or no commits at all). Anything else — an
// operator's tree, uncommitted files, unlanded commits — is kept, so a failed
// or unfinished run never loses work. It reports whether the tree was removed.
func removeLandedRunWorktree(workspace string) bool {
if !isRunWorktree(workspace) {
return false
}
top, err := gitOutput(workspace, "rev-parse", "--show-toplevel")
if err != nil {
return false
}
branch, err := gitOutput(top, "symbolic-ref", "--quiet", "--short", "HEAD")
if err != nil {
return false
}
if status, err := gitOutput(top, "status", "--porcelain=v1", "--untracked-files=all"); err != nil || status != "" {
return false
}
main, err := firstWorktree(top)
if err != nil {
return false
}
// Landed means reachable from the repository's main branch.
if exec.Command("git", "-C", top, "merge-base", "--is-ancestor", "HEAD", "refs/heads/main").Run() != nil {
return false
}
if exec.Command("git", "-C", main, "worktree", "remove", top).Run() != nil {
return false
}
_ = exec.Command("git", "-C", main, "branch", "-d", branch).Run()
return true
}
// firstWorktree is the repository's main (first-listed) worktree, the one
// linked worktrees are managed from.
func firstWorktree(directory string) (string, error) {
list, err := gitOutput(directory, "worktree", "list", "--porcelain")
if err != nil {
return "", err
}
first, _, _ := strings.Cut(list, "\n")
if !strings.HasPrefix(first, "worktree ") {
return "", fmt.Errorf("no main worktree")
}
return strings.TrimPrefix(first, "worktree "), nil
}
// discardRunWorktree removes a run worktree that was created for a run that
// never started (refused or a duplicate request), with its branch. It only
// touches trees the server created.
func discardRunWorktree(workspace string) {
if !isRunWorktree(workspace) {
return
}
top, err := gitOutput(workspace, "rev-parse", "--show-toplevel")
if err != nil {
return
}
branch, err := gitOutput(top, "symbolic-ref", "--quiet", "--short", "HEAD")
if err != nil {
return
}
main, err := firstWorktree(top)
if err != nil {
return
}
_ = exec.Command("git", "-C", main, "worktree", "remove", "--force", top).Run()
_ = exec.Command("git", "-C", main, "branch", "-D", branch).Run()
}
// deliveryCommitMessage is the message for the work a land stage left
// uncommitted: the run's title and id. Only a run in a server-created run
// worktree gets one; elsewhere the tree may hold someone else's work, so the
// gate keeps refusing an unclean checkout there.
func (r *runner) deliveryCommitMessage(id int64) string {
c, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
detail, err := r.store.GetRun(c, id)
if err != nil || detail == nil || !isRunWorktree(detail.Workspace) {
return ""
}
title := strings.TrimSpace(detail.Title)
if title == "" {
title = detail.Name
}
return fmt.Sprintf("%s\n\nPipelines run %d.", title, id)
}
// isRunWorktree reports whether directory is inside a worktree the server
// created for a run.
func isRunWorktree(directory string) bool {
top, err := gitOutput(directory, "rev-parse", "--show-toplevel")
if err != nil {
return false
}
branch, err := gitOutput(top, "symbolic-ref", "--quiet", "--short", "HEAD")
return err == nil && strings.HasPrefix(branch, runWorktreeBranchPrefix) &&
strings.HasSuffix(filepath.Base(filepath.Dir(top)), runWorktreesDirSuffix)
}
+193
View File
@@ -0,0 +1,193 @@
package server
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func newRunWorktreeRepo(t *testing.T) (root, main string) {
t.Helper()
root, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
main = filepath.Join(root, "app")
runGit(t, root, "init", "-b", "main", main)
runGit(t, main, "config", "user.email", "test@example.invalid")
runGit(t, main, "config", "user.name", "test")
if err := os.MkdirAll(filepath.Join(main, "sub"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(main, "sub", "README"), []byte("base\n"), 0o600); err != nil {
t.Fatal(err)
}
runGit(t, main, "add", ".")
runGit(t, main, "commit", "-m", "base")
return root, main
}
func TestIsolateRunWorkspaceGivesAMainCheckoutRunItsOwnWorktree(t *testing.T) {
root, main := newRunWorktreeRepo(t)
// The operator's uncommitted work stays behind.
if err := os.WriteFile(filepath.Join(main, "draft"), []byte("mine\n"), 0o600); err != nil {
t.Fatal(err)
}
tree, err := isolateRunWorkspace(context.Background(), main, "Fix the thing!")
if err != nil {
t.Fatal(err)
}
if filepath.Dir(tree) != filepath.Join(root, "app-runs") {
t.Fatalf("tree %s is not under app-runs", tree)
}
if !strings.Contains(filepath.Base(tree), "fix-the-thing") {
t.Fatalf("tree name %s lacks the title", tree)
}
if _, err := os.Stat(filepath.Join(tree, "draft")); !os.IsNotExist(err) {
t.Fatal("the operator's uncommitted file reached the run worktree")
}
branch, err := gitOutput(tree, "symbolic-ref", "--short", "HEAD")
if err != nil || !strings.HasPrefix(branch, runWorktreeBranchPrefix) {
t.Fatalf("branch %q, %v", branch, err)
}
if !isRunWorktree(tree) {
t.Fatal("the new tree is not recognized as a run worktree")
}
// A subdirectory maps to the same subdirectory; a second run gets another tree.
sub, err := isolateRunWorkspace(context.Background(), filepath.Join(main, "sub"), "")
if err != nil {
t.Fatal(err)
}
if filepath.Base(sub) != "sub" || filepath.Dir(filepath.Dir(sub)) != filepath.Join(root, "app-runs") || filepath.Dir(sub) == tree {
t.Fatalf("subdirectory run went to %s", sub)
}
// A linked worktree (a retry, or an item's own tree) and a plain directory
// are used as they are.
if same, err := isolateRunWorkspace(context.Background(), tree, "x"); err != nil || same != tree {
t.Fatalf("linked worktree became %s, %v", same, err)
}
plain := t.TempDir()
if same, err := isolateRunWorkspace(context.Background(), plain, "x"); err != nil || same != plain {
t.Fatalf("plain directory became %s, %v", same, err)
}
if isRunWorktree(main) || isRunWorktree(plain) {
t.Fatal("a non-run tree was recognized as a run worktree")
}
}
func TestRemoveLandedRunWorktreeKeepsUnlandedWork(t *testing.T) {
_, main := newRunWorktreeRepo(t)
tree, err := isolateRunWorkspace(context.Background(), main, "work")
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(tree, "change"), []byte("x\n"), 0o600); err != nil {
t.Fatal(err)
}
if removeLandedRunWorktree(tree) {
t.Fatal("removed a tree with uncommitted work")
}
runGit(t, tree, "add", "change")
runGit(t, tree, "commit", "-m", "change")
if removeLandedRunWorktree(tree) {
t.Fatal("removed a tree whose commit is not on main")
}
branch, _ := gitOutput(tree, "symbolic-ref", "--short", "HEAD")
runGit(t, main, "merge", "--ff-only", branch)
if !removeLandedRunWorktree(tree) {
t.Fatal("kept a clean, landed run tree")
}
if _, err := os.Stat(tree); !os.IsNotExist(err) {
t.Fatal("tree still on disk")
}
if _, err := gitOutput(main, "rev-parse", "--verify", "refs/heads/"+branch); err == nil {
t.Fatal("landed branch was not deleted")
}
// An operator's own linked worktree is never removed.
own := filepath.Join(filepath.Dir(main), "own")
runGit(t, main, "worktree", "add", "-b", "own", own)
if removeLandedRunWorktree(own) {
t.Fatal("removed an operator worktree")
}
}
func TestDeliveryCommitsLeftoversAndIgnoresAnUncleanMainCheckout(t *testing.T) {
_, main := newRunWorktreeRepo(t)
tree, err := isolateRunWorkspace(context.Background(), main, "deliver")
if err != nil {
t.Fatal(err)
}
access, err := landingAccess(tree)
if err != nil {
t.Fatal(err)
}
// The land stage committed one file and left another; the operator has
// unrelated untracked and modified files in the main checkout.
if err := os.WriteFile(filepath.Join(tree, "committed"), []byte("a\n"), 0o600); err != nil {
t.Fatal(err)
}
runGit(t, tree, "add", "committed")
runGit(t, tree, "commit", "-m", "part")
if err := os.WriteFile(filepath.Join(tree, "left"), []byte("b\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(main, "scratch.md"), []byte("notes\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(main, "sub", "README"), []byte("edited\n"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := verifyDelivery(context.Background(), tree, access, []string{"test -f committed && test -f left"}, "Deliver\n\nPipelines run 1."); err != nil {
t.Fatal(err)
}
for _, name := range []string{"committed", "left"} {
if _, err := os.Stat(filepath.Join(main, name)); err != nil {
t.Fatalf("%s did not land in the main checkout: %v", name, err)
}
}
if got, _ := os.ReadFile(filepath.Join(main, "sub", "README")); string(got) != "edited\n" {
t.Fatal("the operator's edit was overwritten")
}
if subject, _ := gitOutput(main, "log", "-1", "--format=%s"); subject != "Deliver" {
t.Fatalf("leftover commit subject %q", subject)
}
// A conflicting local change in the main checkout refuses the
// fast-forward and changes nothing there.
if err := os.WriteFile(filepath.Join(tree, "sub", "README"), []byte("item\n"), 0o600); err != nil {
t.Fatal(err)
}
before, _ := gitOutput(main, "rev-parse", "HEAD")
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()
if _, err := verifyDelivery(ctx, tree, access, []string{"true"}, "Conflict"); err == nil || !strings.Contains(err.Error(), "fast-forward") {
t.Fatalf("conflicting fast-forward: %v", err)
}
if after, _ := gitOutput(main, "rev-parse", "HEAD"); after != before {
t.Fatal("main moved despite the refused fast-forward")
}
if got, _ := os.ReadFile(filepath.Join(main, "sub", "README")); string(got) != "edited\n" {
t.Fatal("the operator's edit was overwritten")
}
// Outside a run worktree there is no message, so nothing is committed
// for the stage and an unclean item tree is still refused.
other := filepath.Join(filepath.Dir(main), "other")
runGit(t, main, "worktree", "add", "-b", "other", other)
if err := os.WriteFile(filepath.Join(other, "stray"), []byte("x\n"), 0o600); err != nil {
t.Fatal(err)
}
otherAccess, err := landingAccess(other)
if err != nil {
t.Fatal(err)
}
if _, err := verifyDelivery(context.Background(), other, otherAccess, []string{"true"}, ""); err == nil || !strings.Contains(err.Error(), "not clean") {
t.Fatalf("unclean operator tree: %v", err)
}
}
+33 -1
View File
@@ -319,7 +319,29 @@ func (r *runner) startWithSchedule(request protocol.Message, origin *pipeline.Sc
if info, err := os.Stat(workspace); err != nil || !info.IsDir() {
return nil, fmt.Errorf("run workspace is no longer available: %s", workspace)
}
// A fresh run started from a repository's main checkout gets its own
// worktree and branch, so runs never share a tree with each other or with
// the operator's uncommitted work. A retry keeps its run's tree.
isolated := ""
if existing == 0 && request.RetrySource == 0 {
isolateCtx, isolateCancel := context.WithTimeout(r.ctx, time.Minute)
tree, isolateErr := isolateRunWorkspace(isolateCtx, workspace, runTitle.Text)
isolateCancel()
if isolateErr != nil {
return nil, isolateErr
}
if tree != workspace {
isolated = tree
}
workspace = tree
request.Input["pwd"] = workspace
}
id, created, err := r.store.CreateRunTitled(r.ctx, d, request.RequestKey, workspace, request.AllowTools, request.WritePaths, request.Input, pipeline.RunOverrides{Schedule: origin, Model: request.Model, StageModels: request.StageModels, NotifyMode: request.NotifyMode, OutputFormat: request.OutputFormat}, runTitle, retryOf...)
if isolated != "" && (err != nil || !created) {
// The run was refused or an identical request won the race: the fresh
// tree holds nothing yet.
discardRunWorktree(isolated)
}
if err != nil {
return nil, err
}
@@ -405,6 +427,16 @@ func (r *runner) launch(id int64, workspace string, fn func(context.Context)) {
if detail.FinishedAt != "" || detail.Status == protocol.RunStatusWaiting {
delete(r.active, id)
}
// A landed (or change-free) successful run's own worktree is no
// longer needed; anything unlanded or uncommitted is kept.
if detail.Status == "succeeded" && detail.FinishedAt != "" {
workspace := detail.Workspace
go func() {
if removeLandedRunWorktree(workspace) {
log.Printf("run %d: removed its landed worktree %s", id, workspace)
}
}()
}
}()
fn(ctx)
}()
@@ -2053,7 +2085,7 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
} else {
gateCtx, gateCancel := context.WithTimeout(stageCtx, modelConfig.Timeout)
var catchUp string
catchUp, err = verifyDelivery(gateCtx, workspace, landing, checks)
catchUp, err = verifyDelivery(gateCtx, workspace, landing, checks, r.deliveryCommitMessage(id))
gateCancel()
// A failed gate already carries the note in its diagnostic. A step
// diagnostic on success renders as a stop, so success is logged.
+4 -1
View File
@@ -242,7 +242,10 @@ func TestUISkipOverBudgetGitWorkspaceRunsModel(t *testing.T) {
}
git("add", ".")
git("commit", "-qm", "fixture")
detail, visited, verdictCalls := runUISkipIn(t, uiReviewerSource(true), map[string]string{"implement": target}, dir)
// A linked worktree runs in place (a main checkout would get its own tree).
item := filepath.Join(t.TempDir(), "item")
git("worktree", "add", "-q", "-b", "item", item)
detail, visited, verdictCalls := runUISkipIn(t, uiReviewerSource(true), map[string]string{"implement": target}, item)
capture := detail.Steps[0].FileChanges
// Clean tracked files take their baseline from Git, so the budget no longer
// runs out and the edited file keeps a full patch.