Merge branch 'main' into web-shell-polish

This commit is contained in:
2026-10-19 22:31:40 +01:00
45 changed files with 282 additions and 3954 deletions
-15
View File
@@ -16,21 +16,6 @@ service manager directly. Legacy Python-managed installations require an explici
For source development, use `go run ./cmd/pipelines help`; start the native server
separately. Set `PIPELINES_URL` or pass `--url` to select another server.
## Local resource pools
`pipelines resource-pool-inspect NAME --json` reads a declared pool's members,
enabled/quarantined status and non-secret metadata. It does not report lease tokens.
`pipelines resource-pool-configure pool.json --yes --json` replaces the full pool
declaration; use `-` instead of a file to read stdin. Input is limited to 256 KiB.
Without `--yes`, interactive confirmation is required. Both commands require the
loopback operator transport; ordinary remote/stage clients are refused.
Declarations contain `name` and `members`, each with `member`, `enabled`, optional
`quarantined` and bounded string `metadata`. Existing validation limits the member
count to 32 and metadata to 16 fields per member. Configuration does not release
active owners, discard waiters or prove a device is ready; inspect ownership and
verify readiness before activating a target. Never put credentials in metadata.
## Dashboard and status
`pipelines --url ws://HOST:PORT/connect ping --json` returns `pong` with the
-37
View File
@@ -1,37 +0,0 @@
# Android device pool — 2026-10-19
The Android UI reviewer used to request one hard-coded device
(`{"exclusive":"phone-47241FDAS00DLK"}`), so a second UI run had to wait behind
the single Pixel even when another device was idle. Android UI stages now request
the `android-devices` pool instead; the runner grants one free member and passes
it to the stage as `resource_target`, and the stage takes its `adb -s` target
from `resource_target.metadata.adb_serial`. A stage still never chooses a device.
## Pool members
| member | adb serial | metadata |
| --- | --- | --- |
| `phone-47241FDAS00DLK` | `47241FDAS00DLK` | `kind=phone`, `model=Pixel_9_Pro_XL` |
| `emulator-Pipelines_Fold_A` | `emulator-5554` | `kind=emulator`, `avd=Pipelines_Fold_A` |
The emulator is a single managed **foldable** AVD (`Pipelines_Fold_A`, unfolded
built-in display 2076x2152, cover display 1080x2424), started by the operator. It
is the only emulator in scope: ad-hoc emulators are still not started per task.
The operator's **Pixel 10 Pro Fold** (`58281FDCG000JF`) is delivery-only and is
deliberately not a pool member — never a test target.
## Rule reconciliation
`android/AGENTS.md` previously said "never start or use an emulator" and pinned
the Pixel as the only device. That rule is replaced by the pool above so the map,
the runner resource declaration and the live device set agree; the safety rules
(UI-reviewer-only device work, lease-scoped fixtures, settings/build restoration,
fail-closed cleanup) are unchanged.
## Scope
This note, the map update and the pool declaration are configuration and
documentation only. No device probing, health check or autostart is added: the
emulator must already be running and the pool declaration is a static list, so an
offline member is simply not grantable until it is running again.
@@ -1,25 +0,0 @@
# Android target pool allocation — 2026-10-18, phase one not activated
This item adds a reusable store contract only. Pool declarations are not
activated in live jobs, runner handoff, server configuration, Android maps,
shared documentation, or deployment.
`Store.ConfigureResourcePool(ctx, protocol.ResourcePool)` replaces a bounded
pool declaration. Each member has a unique 1–128 byte resource key, an enabled
flag, quarantine state, and at most 16 bounded ASCII-keyed metadata values
(each value is at most 128 bytes). A pool has 1–32 explicit members. Invalid,
empty, duplicate, unknown, or malformed declarations are rejected. Updating a
pool never transfers a live owner; queued requests use the new declaration on
the next eligible release.
`Store.RequestResourcePool(ctx, pool, visit)` atomically returns a lease for one
free eligible concrete member, or a FIFO wait channel. The lease's `Target`
contains only the concrete member and bounded non-secret metadata. Capability
tokens remain private to the store and are never protocol/model metadata.
Direct concrete-resource leases and pool leases share the owner table, so they
cannot overlap. Disabled or quarantined members are skipped; if none are
eligible, the request queues without inference. Release, cancellation, stale
credentials, recovery, and restart retain the existing no-expiry semantics.
This is scheduler coordination only: no device probing, emulator creation,
credentials, activation, or live job changes are included.
-19
View File
@@ -1,19 +0,0 @@
# Inspected resource recovery
The operator recovery contract now keeps quarantined resource ownership and its
audit transactionally coupled to FIFO handoff. Inspection is redacted and
reconciliation is accepted only over a loopback operator connection with the
exact owner visit plus positive cleanup/restoration evidence. The persisted
capability remains private to the authoritative live store; missing waiter
channels fail closed without changing owner or audit state.
Use the configured `PIPELINES_URL` endpoint, not a disposable URL. First run
the redacted resource inspection, then reconcile only after the parked visit
has been inspected and quiesced, supplying its exact run/stage/sequence and
explicit evidence (including `no mutating action occurred` where applicable).
The old deployed server does not support this action; deploy a compatible
server before attempting recovery. This is an operator trust boundary, not an
OS sandbox or unrestricted authorization grant.
The shared README and packaged OpenCode skill references should be updated by
the manual `docs-fold` process; they are intentionally not edited here.
-158
View File
@@ -1,158 +0,0 @@
# Stage-exclusive resources — 2026-10-18, not deployed
This is source work for run 355, **not a deployment, installation or publication**.
No physical phone, real operator session or deployed service was used or changed.
The Android role split does not retrospectively gate pinned versions or existing runs.
## Contract and scope
An executable non-human stage can declare one runner-owned control block:
```resources
{"exclusive":"phone-47241FDAS00DLK"}
```
The key is 1–128 ASCII identifier bytes (letter first, then letters, digits, `_`, `-`).
Duplicate/unknown fields, multiple resources and human-stage resources are rejected.
One capacity-one resource per stage visit, one server/store authority; no expressions,
nested acquisition, distributed scheduling, capacity settings or multi-resource requests.
Resource-free definitions retain their execution path. Old servers reject this new block;
do not publish to an incompatible server.
Acquisition follows selection and automatic UI-skip evaluation, before BeginStep,
visits/sequences, artifacts, filediff, stage timeout, budget setup or executor calls.
Queued runs retain workspace reservation, use legacy `queued` (never human `waiting`),
and carry bounded `resource_wait` identity/stage/sequence/queued-since metadata.
Events use `resource_wait_set=true` with absent `resource_wait` to clear caches.
CLI lists and TUI/Android details show “Waiting for resource …”; human decisions and
attention notifications remain separate. Older clients still see an unfinished run.
SQLite owners and FIFO waiters are indexed by resource/order and run identity.
Private random capabilities attribute ownership to resource/run/stage/sequence and
never enter wire data, model input, transcripts or artifacts. Only the full capability
and positive cleanup evidence authorize release; mismatches and repeated release cannot
release a subsequent owner. One buffered channel per live queued run, no resource
polling, waiter tickers, model-based acquisition, queue driver or new daemon.
The gate holds through tool cleanup. OpenCode interruption now confirms session idle/
V1 settlement before clearing session ownership. Pi recovery requires both its receipt
and process/process-group exit, not merely abort acknowledgement. Executor panic,
ErrStageCleanup, retained active sessions and failed cleanup queries quarantine the owner.
An explicit `output.resource_cleanup_confirmed=false` (or malformed value) also vetoes
release for failed restoration; true never substitutes for harness cleanup evidence.
The generic runner does not inspect Android settings or magically verify restoration.
The Android UI reviewer must keep all device work foreground/awaited, restore original
settings/connection and main's build, and report uncertainty with the release veto.
## Restart and retention
Automatic FIFO progress is promised only within a live server process. Startup interrupts
persisted queued resource visits, removes their waiter rows, and never replays an executor.
Durably cleanup-confirmed owners can be reclaimed through the token-checked delete helper.
Unconfirmed owners remain quarantined, including crashes before BeginStep, before session
registration, or after cleanup but before its confirmation is persisted. Session interruption
alone cannot prove resource-specific restoration. Age never authorizes reclamation.
Quarantined ownership/session evidence survives recovery and protects even terminal runs
from retention. Reconciliation requires inspected cleanup and restoration, not lock deletion.
There is intentionally no force-unlock CLI or automatic uncertain-owner reclamation.
## Android draft, provenance and activation
Read-only retrieval: `pipelines show Android --version 2 --json` returned pipeline 20,
v2, hash `8c93aeb9b76f1e6e7696415d6308d786ec08d805a1a346fbabe4d252d5d65032`.
`python3` recomputed SHA-256 of the returned source and asserted equality.
The exact source is retained as `internal/server/testdata/android-v2-published.md`;
the proposed source is `internal/server/testdata/android-resource-draft.md`.
The provenance test compares graph fields/outcomes unchanged, apart from the approved
three stage prompts and the UI review resource; every stage keeps its published v3 model: Terra for implementer/committer and Luna for reviewer/UI reviewer. No unrelated published definition was edited.
Artifact handoff recovery: the validated v3 draft was copied to this run's retained
artifact directory as `android-resource-draft.md`. The similarly named artifact from
run 355 remains the old v2 baseline and must not be published. The proposed source in
this workspace and this run's artifact have matching SHA-256 digests
`a10d8b9fa3dc6a1875b10469061ed25fb4a7a404d463d1a2d304a309d55aa5d9`.
Activation, requiring separate explicit operator authorization:
1. Review/land the source change and independently rerun the required checks.
2. Arrange a compatible deployment separately; nothing here deploys or restarts it.
3. Reconcile every existing phone user, pinned/live run and any quarantined owner.
4. Transfer the **workspace proposed draft**, provenance and this note into the retained
stage artifacts after resolving the artifact edit permission.
5. Validate that retained draft against the configured authoritative compatible endpoint;
only then explicitly publish it. Do not use a disposable endpoint as a project default.
6. Opt in other device users explicitly before relying on mutual exclusion system-wide.
Read-only published inventory (`pipelines show <name> --json`, v2 for each):
- UI sweep: `survey` and `ui_reviewer` request visual/screenshots work; phone work there
needs explicit resource opt-in and restoration/fixture policy.
- Generic: `reviewer` and `ui_reviewer` have visual/device verification wording; when
targeting a phone, separate code review from leased device review first.
- Plan and land: `implementer`, `reviewer`, `ui_reviewer` contain rendered-surface work;
they need an explicit technology-specific role split/lease before phone use.
- Probe: one harness smoke stage; no default phone declaration. Any device-specific
probe needs explicit opt-in, never an inferred resource or Android stage-name special case.
Server/TUI/other definitions were not changed. Inventory is not activation evidence.
## Verification evidence and remaining limits
Executed focused checks:
- Run 358 recovery revalidation: `go build ./... && go vet ./... && go test ./...`
passed; `go test -race ./internal/pipeline ./internal/server ./internal/stage -run
'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1` passed; and the
Android map invocation with `JAVA_HOME`/`ANDROID_HOME`, installed Gradle,
`--offline --no-daemon --rerun-tasks :app:assembleDebug :app:testDebugUnitTest`
executed all 41 tasks and reported `BUILD SUCCESSFUL`. These remain build/unit and
fake-executor scheduler evidence,
not physical-phone verification or screenshots.
- Recovery after integrating main `88595eb`: `go build ./...`, `go vet ./...`,
`go test ./internal/server -run Resource -count=1`,
`go test ./internal/pipeline ./internal/protocol ./internal/tui -run Resource -count=1`,
`go test ./internal/stage -count=1`,
`go test -race ./internal/pipeline ./internal/server ./internal/stage -run 'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1`, and
`go test ./...` all passed. `gofmt -l .` and `git diff --check` produced no output.
- Recovery Android build/unit check with the map's offline, no-daemon Gradle invocation:
`:app:assembleDebug :app:testDebugUnitTest` passed. This remains build/unit evidence,
not phone verification or screenshots.
- `go test ./internal/server ./internal/pipeline ./internal/protocol ./internal/tui -run Resource -count=1`:
passed. Includes fake-executor queue >30ms timeout, independent-resource progress,
FIFO/mismatch/duplicate/grant-cancel cycles, blocked cleanup, panic/quarantine,
persistence/BeginStep/query failure, no-acquire skip, restart/retention, wire defaults,
narrow/wide/plain TUI frames and real CLI/WebSocket scratch exercise.
- `go test ./internal/stage -count=1`: passed; the later full suite also included
`TestResourcePiReceiptRequiresProcessExit` (a test-owned sleep process, not a device).
- Android map invocation with `JAVA_HOME`/`ANDROID_HOME` and installed Gradle,
`--offline --no-daemon :app:assembleDebug :app:testDebugUnitTest`: BUILD SUCCESSFUL.
These are build/unit checks, not device verification or screenshots.
- After the Go tree settled: `go build ./...`, `go vet ./...`, `go test ./...`
all exited zero (including author-package tests).
- `go test -race ./internal/pipeline ./internal/server ./internal/stage -run 'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1`:
all three packages passed, including concurrent grant/cancel cycles.
- `gofmt -l .` and `git diff --check`: exited zero with no output.
The scratch CLI test uses the real CLI and WebSocket handler with temporary SQLite and
an explicitly **FAKE executor**. It validates the proposed Android draft on compatible
scratch code, observes B as queued with zero steps while A holds fake-phone, inspects
subscription metadata, releases A, runs `wait`, and observes B succeeded automatically.
It does not call a model or use a physical phone. Test log output is evidence, not a
live-deployment claim; no retained CLI transcript was written to the denied artifact path.
Live OpenCode/Pi shell/ADB termination and Android visual/device restoration are **not
tested** here. The checks prove scheduler behavior with controlled fakes, plus test-owned
process-exit checks, not device readiness on the current deployment.
Commit/merge/landing belong to the later committer stage; no commit or push here.
## Stale shared documentation for docs-fold (do not edit in this item)
- `docs/cli.md`: resource queue presentation, inspection metadata, unchanged wait/decide semantics.
- `internal/author/skill/references/format.md`: optional resources block and validation/key bounds.
- `internal/author/skill/references/control.md`: runner-owned declaration, timeout/accounting boundary,
release veto and automatic skip placement.
- `internal/author/skill/references/queue.md`: FIFO resource waits vs workspace exclusion and human waits.
- `internal/author/skill/references/operations.md`: authority, restart interruption/quarantine,
inspected recovery, retention and explicit compatible-deployment/publication prerequisites.
- `internal/author/skill/references/verification.md`: fake-vs-device evidence, fixtures and leased restoration.
- `internal/author/skill/references/orchestration.md`: no queue drivers/polling/extra agents; explicit
opt-in of every device user, no retroactive gating or distributed guarantees.
No packaged skill/reference, shared README/topic documentation, pipeline registry,
installation, deployment, provider setting or `internal/stage/model_failure.go` was edited.
@@ -1,7 +0,0 @@
# Resource-pool operator configuration — 2026-10-18
`resource-pool-configure <pool.json|-> --yes` replaces one bounded pool declaration
through the loopback-only operator transport. `resource-pool-inspect <pool>` reads
the declared members, enabled/quarantine status, and non-secret metadata. Existing
owners and waiters remain under the store's normal ownership, quarantine, and FIFO
rules; configuration does not release active leases.
@@ -1,13 +0,0 @@
# Resource-pool runner handoff — 2026-10-18
Stage `resources` declarations now accept exactly one of `exclusive` or `pool`.
Pool stages wait on the named pool, then receive the selected concrete member and
bounded metadata through explicit runtime context. Model execution serializes that
server-granted context as a separate `resource_target` runtime field for both
native and plain harnesses; the task input map remains unchanged and cannot select
the target. Capability tokens remain private to the store. Retained control
evidence records the concrete target; queued run state names the pool while an
active owner names the selected member.
This is parser/runner handoff only. It does not add Android definitions, pool
configuration rollout, device setup, probing, or deployment.
+1 -1
View File
@@ -21,7 +21,7 @@ Use `pipelines wait` for runs you started, without a polling orchestration loop.
Inspect diffs, run proportionate checks, resolve conflicts and finish landing.
Never discard user work or delete branches. Verify ancestry after the actual
merge or server gate. Keep device leases and do not disrupt active shared work.
merge or server gate. Do not disrupt active shared work.
The user's request authorizes routine steps needed to complete it. Ask only about
real ambiguity, destructive actions or changes outside that scope. Report actual
+2 -2
View File
@@ -20,8 +20,8 @@ does not freeze unrelated work. Continue at the necessary stage or recover direc
## Definitions
Edit a retained draft, not immutable history. Give each stage one simple job and
only the input it needs. Preserve review gates, runtime limits, permissions,
resource declarations and the user's constraints unless asked to change them.
only the input it needs. Preserve review gates, runtime limits, permissions and
the user's constraints unless asked to change them.
Use `pipelines validate DRAFT --json` before publication. Publish or execute when
the user requests it; do not ask again for routine steps already authorized.
Keep version guards. `--yes` is not a conflict policy; never silently force an
@@ -20,9 +20,8 @@ delivery gate succeeds with a clean checked candidate; the server lands afterwar
Read-only, nonvisual work does not require device screenshots. Visual acceptance
requires a ready fixture, verified connection and actual captured interactions.
Preserve exclusive target ownership; do not force-unlock uncertain cleanup.
Inspect tool/process termination before reconciling an inactive resource. Stop and
await exact owned fixture processes, not just a go-run parent. Do not disrupt a
different active lease or restart a shared service over running work.
Preserve exclusive fixture ownership; inspect tool/process termination before
reuse. Stop and await exact owned fixture processes, not just a go-run parent. Do
not disrupt another active fixture or restart a shared service over running work.
Report the exact blocker and next action briefly. Keep independent work moving.
+1 -1
View File
@@ -23,7 +23,7 @@ Wait for shared work to quiesce before restarting its service. Back up durable
state, preserve endpoint/configuration, verify the new runtime identity and keep
a rollback path. Do not wipe data or remove credentials to repair deployment.
Respect target leases and the user's device boundaries. A test-device authorization
Respect the user's device boundaries. A test-device authorization
does not authorize use of the personal phone. Installation on the personal Fold
requires the user's delivery request; preserve app data and verify the exact APK.
Use `android/scripts/capture_png.sh -s SERIAL PATH` for app captures, not a raw
+3 -146
View File
@@ -7,7 +7,6 @@ import (
"flag"
"fmt"
"io"
"net/url"
"os"
"os/exec"
"path/filepath"
@@ -47,9 +46,7 @@ type options struct {
keepSet bool
all bool
json, source, noOpen, yes, allow, createOnly, force, wait bool
version, expected, limit, recoverySequence, retryVisit int
recoveryRun int64
recoveryStage, resource, poolFile string
version, expected, limit, retryVisit int
timeout, waitTimeout time.Duration
stageModels stageModelFlags
}
@@ -186,18 +183,6 @@ func (a App) flags(command string) (*flag.FlagSet, *options) {
case "decide":
f.StringVar(&o.option, "option", "", "Option key to choose among the waiting stage's edges (required)")
f.StringVar(&o.feedback, "feedback", "", "Optional free-text feedback recorded with the decision")
case "resource-inspect":
f.StringVar(&o.resource, "resource", "", "Exclusive resource key")
case "resource-reconcile":
f.StringVar(&o.evidence, "evidence", "", "Bounded positive cleanup/restoration evidence (required)")
f.StringVar(&o.actor, "actor", "operator", "Bounded operator identity")
f.BoolVar(&o.allow, "no-mutating-action", false, "Evidence explicitly records that no mutating action occurred")
f.BoolVar(&o.yes, "yes", false, "Confirm reconciliation without prompting")
case "resource-pool-inspect":
f.StringVar(&o.resource, "pool", "", "Resource pool name")
case "resource-pool-configure":
f.StringVar(&o.poolFile, "file", "", "JSON pool declaration, or - for stdin")
f.BoolVar(&o.yes, "yes", false, "Confirm this mutation without prompting")
case "debug":
f.StringVar(&o.harness, "harness", "", "Debugging harness: configured opencode/pi, or nvim")
f.BoolVar(&o.yes, "yes", false, "Approve opening a debugging session without prompting")
@@ -484,13 +469,6 @@ func (a App) Run(ctx context.Context, args []string) error {
requestTimeout = 30 * time.Second
}
connectURL := o.url
if command == "resource-inspect" || command == "resource-reconcile" || command == "resource-pool-inspect" || command == "resource-pool-configure" {
var transportErr error
connectURL, transportErr = operatorTransportURL(o.url)
if transportErr != nil {
return transportErr
}
}
dialCtx, cancel := context.WithTimeout(ctx, requestTimeout)
c, err := client.Open(dialCtx, connectURL)
cancel()
@@ -803,7 +781,7 @@ func (a App) Run(ctx context.Context, args []string) error {
}
case "runs":
for _, run := range r.Runs {
fmt.Fprintf(a.Out, "%d\t%s\t%s\tv%d\t%s\n", run.ID, run.Label(), run.Name, run.Version, run.StatusLabel())
fmt.Fprintf(a.Out, "%d\t%s\t%s\tv%d\t%s\n", run.ID, run.Label(), run.Name, run.Version, run.Status)
}
case "ping":
fmt.Fprintln(a.Out, "Connected ·", r.Time)
@@ -926,117 +904,6 @@ func (a App) Run(ctx context.Context, args []string) error {
}
}
return fmt.Errorf("template %q not found", key)
case "resource-inspect":
if len(pos) > 1 {
return fmt.Errorf("resource-inspect takes one resource key")
}
if o.resource == "" && len(pos) == 1 {
o.resource = pos[0]
}
if strings.TrimSpace(o.resource) == "" {
return fmt.Errorf("resource-inspect requires a resource key")
}
r, err := call(protocol.Message{Type: protocol.InspectResource, Resource: o.resource})
if err != nil {
return err
}
if r.ResourceInspection == nil {
return fmt.Errorf("server did not return resource inspection")
}
return a.output(r.ResourceInspection)
case "resource-reconcile":
if len(pos) != 4 {
return fmt.Errorf("resource-reconcile requires resource, run-id, stage and sequence")
}
runID, err := strconv.ParseInt(pos[1], 10, 64)
if err != nil || runID <= 0 {
return fmt.Errorf("run ID must be positive")
}
sequence, err := strconv.Atoi(pos[3])
if err != nil || sequence <= 0 {
return fmt.Errorf("sequence must be positive")
}
if strings.TrimSpace(o.evidence) == "" {
return fmt.Errorf("--evidence is required")
}
if err := a.confirmAction(fmt.Sprintf("Reconcile quarantined resource %s for run #%d?", pos[0], runID), o.yes); err != nil {
return err
}
r, err := call(protocol.Message{Type: protocol.ReconcileResource, Resource: pos[0], ResourceRecovery: &protocol.ResourceRecoveryRequest{Resource: pos[0], RunID: runID, Stage: pos[2], Sequence: sequence, Actor: o.actor, Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, NoMutatingAction: o.allow, Summary: o.evidence}}})
if err != nil {
return err
}
if o.json {
return a.output(r)
}
_, err = fmt.Fprintf(a.Out, "Reconciled resource %s for run #%d\n", pos[0], runID)
return err
case "resource-pool-inspect":
if len(pos) > 1 {
return fmt.Errorf("resource-pool-inspect takes one pool name")
}
if o.resource == "" && len(pos) == 1 {
o.resource = pos[0]
}
if strings.TrimSpace(o.resource) == "" {
return fmt.Errorf("resource-pool-inspect requires a pool name")
}
r, err := call(protocol.Message{Type: protocol.InspectResourcePool, Resource: o.resource})
if err != nil {
return err
}
if r.ResourcePool == nil {
return fmt.Errorf("server did not return resource pool")
}
return a.output(r.ResourcePool)
case "resource-pool-configure":
if len(pos) > 1 {
return fmt.Errorf("resource-pool-configure takes one JSON file")
}
file := o.poolFile
if file == "" && len(pos) == 1 {
file = pos[0]
}
if file == "" {
return fmt.Errorf("resource-pool-configure requires a JSON file")
}
var data []byte
if file == "-" {
input := a.In
if input == nil {
input = os.Stdin
}
data, err = io.ReadAll(io.LimitReader(input, 256*1024+1))
} else {
var input *os.File
input, err = os.Open(file)
if err == nil {
data, err = io.ReadAll(io.LimitReader(input, 256*1024+1))
_ = input.Close()
}
}
if err != nil {
return err
}
if len(data) > 256*1024 {
return fmt.Errorf("resource pool input exceeds 262144 bytes")
}
var pool protocol.ResourcePool
if err := json.Unmarshal(data, &pool); err != nil {
return fmt.Errorf("invalid resource pool JSON: %w", err)
}
if err := a.confirmAction(fmt.Sprintf("Configure resource pool %s?", pool.Name), o.yes); err != nil {
return err
}
r, err := call(protocol.Message{Type: protocol.ConfigureResourcePool, ResourcePool: &pool})
if err != nil {
return err
}
if o.json {
return a.output(r)
}
_, err = fmt.Fprintf(a.Out, "Configured resource pool %s\n", pool.Name)
return err
case "inspect", "cancel", "retry":
value, err := one()
if err != nil {
@@ -1217,16 +1084,6 @@ func (a App) Run(ctx context.Context, args []string) error {
return nil
}
func operatorTransportURL(raw string) (string, error) {
u, err := url.Parse(raw)
if err != nil || u.Scheme == "" || u.Host == "" {
return "", fmt.Errorf("invalid configured pipeline URL")
}
u.Path = "/operator-connect"
u.RawPath = ""
return u.String(), nil
}
func (a App) output(value any) error {
encoder := json.NewEncoder(a.Out)
encoder.SetIndent("", " ")
@@ -1672,7 +1529,7 @@ func (e *waitLimitError) Runs() string {
// waitLine is the one-line terminal report `pipelines wait` prints: the run id,
// its status and outcome, its stage and how long it took.
func waitLine(d *protocol.RunDetail) string {
line := fmt.Sprintf("run %d · %s", d.ID, d.StatusLabel())
line := fmt.Sprintf("run %d · %s", d.ID, d.Status)
if outcome := strings.TrimSpace(d.Outcome); outcome != "" {
line += " (" + outcome + ")"
}
+53 -58
View File
@@ -2,67 +2,62 @@ package cli
// Help is also the public command inventory for agents and parity checks.
var usage = map[string]string{
"provider-check": "provider-check --model PROVIDER/MODEL [--yes] [--json]",
"new": "new [name] [prompt] [--template KEY] [--no-open | --json]",
"edit": "edit [name|id] [prompt] [--version N] [--no-open | --json]",
"drafts": "drafts [--json]",
"draft-delete": "draft-delete <draft.md> [--json]",
"validate": "validate <draft.md> [--json]",
"submit": "submit <draft.md> [--name NAME] [--create-only] [--yes] [--json]",
"list": "list [--json]",
"search": "search QUERY [--type KIND] [--limit N] [--json]",
"show": "show <name|id> [--version N] [--json]",
"versions": "versions <name|id> [--json]",
"export": "export [name|id ...] [--output FILE.zip] [--all] [--version N]",
"import": "import [FILE.zip] [--all] [--on-conflict ask|keep|both|import] [--yes] [--json]",
"templates": "templates [--json]",
"template": "template <key> [--source | --json]",
"run": "run [name|id] [prompt] [--title TITLE] [--model PROVIDER/MODEL] [--stage-model STAGE=PROVIDER/MODEL ...] [--json [--wait [--wait-timeout DURATION]]]",
"runs": "runs [--pipeline NAME] [--json]",
"inspect": "inspect <run-id> [--json]",
"wait": "wait <run-id> [run-id ...] [--timeout DURATION] [--json]",
"cancel": "cancel <run-id> [--json]",
"decide": "decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]",
"retry": "retry <run-id> [--mode resume|restart] [--stage KEY [--visit N]] [--version N] [--input-file FILE|-] [--json [--wait [--wait-timeout DURATION]]]",
"title": "title <run-id> \"<title>\" [--json]",
"debug": "debug <run-id> [--json | --no-open]",
"chat-adopt": "chat-adopt <session-id> [--json]",
"connection": "connection [--json]",
"ping": "ping [--json]",
"version": "version [--json]",
"status": "status [--json]",
"stats": "stats [--json]",
"resource-inspect": "resource-inspect <resource> [--json]",
"resource-reconcile": "resource-reconcile <resource> <run-id> <stage> <sequence> --evidence TEXT [--no-mutating-action] [--yes] [--json]",
"resource-pool-inspect": "resource-pool-inspect <pool> [--json]",
"resource-pool-configure": "resource-pool-configure <pool.json|-> --yes [--json]",
"rpc": "rpc <request.json|->",
"retention": "retention report|apply [--older-than DURATION] [--keep N] [--db FILE] [--artifacts DIR] [--json] [--yes]",
"tui": "tui",
"provider-check": "provider-check --model PROVIDER/MODEL [--yes] [--json]",
"new": "new [name] [prompt] [--template KEY] [--no-open | --json]",
"edit": "edit [name|id] [prompt] [--version N] [--no-open | --json]",
"drafts": "drafts [--json]",
"draft-delete": "draft-delete <draft.md> [--json]",
"validate": "validate <draft.md> [--json]",
"submit": "submit <draft.md> [--name NAME] [--create-only] [--yes] [--json]",
"list": "list [--json]",
"search": "search QUERY [--type KIND] [--limit N] [--json]",
"show": "show <name|id> [--version N] [--json]",
"versions": "versions <name|id> [--json]",
"export": "export [name|id ...] [--output FILE.zip] [--all] [--version N]",
"import": "import [FILE.zip] [--all] [--on-conflict ask|keep|both|import] [--yes] [--json]",
"templates": "templates [--json]",
"template": "template <key> [--source | --json]",
"run": "run [name|id] [prompt] [--title TITLE] [--model PROVIDER/MODEL] [--stage-model STAGE=PROVIDER/MODEL ...] [--json [--wait [--wait-timeout DURATION]]]",
"runs": "runs [--pipeline NAME] [--json]",
"inspect": "inspect <run-id> [--json]",
"wait": "wait <run-id> [run-id ...] [--timeout DURATION] [--json]",
"cancel": "cancel <run-id> [--json]",
"decide": "decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]",
"retry": "retry <run-id> [--mode resume|restart] [--stage KEY [--visit N]] [--version N] [--input-file FILE|-] [--json [--wait [--wait-timeout DURATION]]]",
"title": "title <run-id> \"<title>\" [--json]",
"debug": "debug <run-id> [--json | --no-open]",
"chat-adopt": "chat-adopt <session-id> [--json]",
"connection": "connection [--json]",
"ping": "ping [--json]",
"version": "version [--json]",
"status": "status [--json]",
"stats": "stats [--json]",
"rpc": "rpc <request.json|->",
"retention": "retention report|apply [--older-than DURATION] [--keep N] [--db FILE] [--artifacts DIR] [--json] [--yes]",
"tui": "tui",
}
var notes = map[string]string{
"version": "Identify the local CLI binary without connecting to a server. --version is an alias. Use ping --json to identify the running server separately.",
"ping": "Pong includes the running server's compiled build.version, build.commit and build.dirty (null when unknown). This identifies the server, not the client or runtime workspace. Older servers omit build. The client compares its own build/protocol against the server: a same-protocol build mismatch prints one warning to stderr, a server speaking a newer protocol fails the command, and a response carrying a field this client does not model prints a one-time unrecognized-field notice instead of dropping it silently.",
"connection": "Prints the server endpoint and phone notification endpoints. Like ping, it reports the client/server build relationship and warns or fails rather than silently proceeding against a mismatched server.",
"status": "Reports server, host and ntfy publishing health. The client compares its build/protocol against the server and warns once (or fails on a newer protocol) rather than proceeding silently.",
"export": "Without a name, select pipelines with Space and save a ZIP in the current directory. --output writes a portable definition bundle; --all skips selection. One name without --output prints Markdown as before. No runs, credentials or old versions are exported.",
"import": "Select definitions from a ZIP. Identical existing definitions are skipped. For differing names, choose keep existing, keep both under a new name, or use imported as a new version (history retained). --on-conflict supplies an explicit automation policy; --yes only approves final publication. Schedules may activate; no run history is imported.",
"new": "Create an optional template copy or a one-stage starter. Opens the configured default harness; --harness opencode|pi|nvim overrides it. No submission is performed.",
"edit": "Omit the name to open fuzzy selection. Original immutable source is copied to a private draft. The configured harness edits that draft, never the live definition.",
"validate": "Uses the same server parser as submission. Does not persist or activate schedules. Structural validity does not prove runtime/tool availability.",
"submit": "Confirm publication at the prompt, or use --yes for explicit automation; schedules activate on acceptance. Managed drafts provide the name and base-version guard. --force never bypasses validation.",
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. --model PROVIDER/MODEL overrides the model for every stage of that run; --stage-model STAGE=PROVIDER/MODEL overrides one stage and is repeatable, with a repeated key last-wins. Precedence is per-stage > whole-run > the model pinned in the pipeline version; overrides apply to that run only and never change the pinned definition. An unknown stage key or a malformed provider/model is refused before the run starts, naming the valid stage keys. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
"retry": "Resume a failed stage by default, restoring the completed prefix handoff; use --mode=restart to start at entry. --stage selects a recorded stage (use --visit for repeated stages). --input-file accepts a bounded JSON object, merged with replacement at the selected stage; '-' reads stdin. Original history and pinned identity are retained; --version selects an explicit published version of the same pipeline only. --wait-timeout leaves an active retry for later read-only inspection or reattachment with the original --request-key, rather than starting a duplicate retry.",
"title": "Rename a run's operator-facing title after the fact without touching its input, steps or execution. The title is collapsed to a single bounded line and recorded as explicitly set. `pipelines runs` and `pipelines inspect` then show it.",
"wait": "Block until each named run reaches a terminal status - succeeded, failed, canceled or interrupted, and also through a run waiting on a person - then print one line per run with its id, status and outcome, its stage and how long it took, and exit 0 even for a run that finished unsuccessfully. Several ids wait together in one command. The exit is non-zero only for a client-side failure: an unknown run id, a connection that could not be established, or --timeout expiring while a run is still active; a timeout says so plainly, never cancels, retries or otherwise touches a run, and is never reported as its outcome. Read-only: it needs no workspace, is not refused by the workspace guard and writes nothing. --json prints the final run envelope instead of the line (one envelope for one id, an array for several).",
"decide": "Choose an option for a run waiting on a human stage. The run must be `waiting`; the option must be one of that stage's non-error edges. The stage is inferred from the run's current stage unless named. The choice and optional --feedback are recorded as the step's evidence (visible in inspect and the TUI) and the run resumes along the chosen edge. An unknown option is refused with the valid list; a repeated decision for an already-decided visit does not double-apply. Works across a server restart because the wait is persisted.",
"debug": "Open a fresh configured harness conversation to investigate a run using inspect/show/runs CLI queries. --harness opencode|pi|nvim chooses another configured harness or read-only nvim packet. Does not resume implementation or retry. --json returns the prompt without launching anything.",
"rpc": "Accepts a JSON protocol envelope; '-' reads stdin. Every TUI server operation is available here too. Mutating commands are not dry runs. The omni-search command is `search` (fields query, optional kind and limit) and answers with ranked `search_results`.",
"chat-adopt": "Adopt a session that already exists on the configured OpenCode endpoint so it appears in chat_sessions and can be continued. The id must be one the endpoint reports; its directory and title are taken from the endpoint's own record (never guessed), and a missing id records nothing. Adopting is always explicit and does not change the server's endpoint choice.",
"retention": "Local, explicit run-history retention; never automatic and never a side effect of another command. `report` is strictly read-only and prints what a policy would remove; `apply` removes it after approval (--yes for scripted use). Policy: --older-than DURATION selects runs created before now minus that rolling UTC duration (Go durations plus d/w; not a calendar boundary), --keep N keeps the newest N runs by run id and selects older ranks (--keep 0 keeps none). Both together are the conservative intersection; at least one is required. Protected and never removed: queued/running runs, runs waiting on a person (waiting run or step), runs with an active cleanup agent, and any run referenced by a retained run's retry provenance (transitively). A run that becomes protected before deletion aborts the whole apply. Artifacts at <root>/<run> are removed with their run and kept with protected/retained runs; the database is VACUUMed and its WAL checkpointed, with measured reclaimed bytes. Definitions and versions are immutable and never touched. --db must name an existing database (default: the standard data directory); --artifacts defaults to artifacts/ beside --db. Works without a running server.",
"search": "One read-only query over pipelines, versions, runs, steps, field text and artifacts. QUERY is free text, a bare status word (queued, running, succeeded, failed, canceled, interrupted) selecting those runs, or a lone date expression (today, yesterday, this week [Monday 00:00 local], or YYYY-MM-DD) selecting the runs created in that window. --type is the only narrowing mechanism and takes one kind: pipeline, version, run, step, field or artifact; an unknown kind is an error. Ranking: an exact name match (3) beats a name substring (2) beats a body-text match (1); equal tiers break by recency, then a stable kind/target tie-break, and every hit reports score, matched_field and recency so a client can re-rank. Bounds: the result cap defaults to 50 and clamps to 200 (0 = default); every kind scans at most 2000 rows and field-text scanning is additionally capped at 2000 steps / 1 MiB; Truncated plus scanned_steps, scanned_bytes and scan_bound_bytes report when a bound applied. Targets are stable: pipeline:<id>, pipeline:<id>:v<version>, run:<id>, run:<id>:<sequence>, run:<id>:<sequence>:<field>, or an artifact URL.",
"resource-pool-configure": "Read a bounded full pool declaration from JSON and replace the declaration only after explicit --yes consent. Use '-' for stdin. This is local operator transport only; it never accepts or returns capabilities or credentials. resource-pool-inspect reads the declared members and status.",
"version": "Identify the local CLI binary without connecting to a server. --version is an alias. Use ping --json to identify the running server separately.",
"ping": "Pong includes the running server's compiled build.version, build.commit and build.dirty (null when unknown). This identifies the server, not the client or runtime workspace. Older servers omit build. The client compares its own build/protocol against the server: a same-protocol build mismatch prints one warning to stderr, a server speaking a newer protocol fails the command, and a response carrying a field this client does not model prints a one-time unrecognized-field notice instead of dropping it silently.",
"connection": "Prints the server endpoint and phone notification endpoints. Like ping, it reports the client/server build relationship and warns or fails rather than silently proceeding against a mismatched server.",
"status": "Reports server, host and ntfy publishing health. The client compares its build/protocol against the server and warns once (or fails on a newer protocol) rather than proceeding silently.",
"export": "Without a name, select pipelines with Space and save a ZIP in the current directory. --output writes a portable definition bundle; --all skips selection. One name without --output prints Markdown as before. No runs, credentials or old versions are exported.",
"import": "Select definitions from a ZIP. Identical existing definitions are skipped. For differing names, choose keep existing, keep both under a new name, or use imported as a new version (history retained). --on-conflict supplies an explicit automation policy; --yes only approves final publication. Schedules may activate; no run history is imported.",
"new": "Create an optional template copy or a one-stage starter. Opens the configured default harness; --harness opencode|pi|nvim overrides it. No submission is performed.",
"edit": "Omit the name to open fuzzy selection. Original immutable source is copied to a private draft. The configured harness edits that draft, never the live definition.",
"validate": "Uses the same server parser as submission. Does not persist or activate schedules. Structural validity does not prove runtime/tool availability.",
"submit": "Confirm publication at the prompt, or use --yes for explicit automation; schedules activate on acceptance. Managed drafts provide the name and base-version guard. --force never bypasses validation.",
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. --model PROVIDER/MODEL overrides the model for every stage of that run; --stage-model STAGE=PROVIDER/MODEL overrides one stage and is repeatable, with a repeated key last-wins. Precedence is per-stage > whole-run > the model pinned in the pipeline version; overrides apply to that run only and never change the pinned definition. An unknown stage key or a malformed provider/model is refused before the run starts, naming the valid stage keys. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
"retry": "Resume a failed stage by default, restoring the completed prefix handoff; use --mode=restart to start at entry. --stage selects a recorded stage (use --visit for repeated stages). --input-file accepts a bounded JSON object, merged with replacement at the selected stage; '-' reads stdin. Original history and pinned identity are retained; --version selects an explicit published version of the same pipeline only. --wait-timeout leaves an active retry for later read-only inspection or reattachment with the original --request-key, rather than starting a duplicate retry.",
"title": "Rename a run's operator-facing title after the fact without touching its input, steps or execution. The title is collapsed to a single bounded line and recorded as explicitly set. `pipelines runs` and `pipelines inspect` then show it.",
"wait": "Block until each named run reaches a terminal status - succeeded, failed, canceled or interrupted, and also through a run waiting on a person - then print one line per run with its id, status and outcome, its stage and how long it took, and exit 0 even for a run that finished unsuccessfully. Several ids wait together in one command. The exit is non-zero only for a client-side failure: an unknown run id, a connection that could not be established, or --timeout expiring while a run is still active; a timeout says so plainly, never cancels, retries or otherwise touches a run, and is never reported as its outcome. Read-only: it needs no workspace, is not refused by the workspace guard and writes nothing. --json prints the final run envelope instead of the line (one envelope for one id, an array for several).",
"decide": "Choose an option for a run waiting on a human stage. The run must be `waiting`; the option must be one of that stage's non-error edges. The stage is inferred from the run's current stage unless named. The choice and optional --feedback are recorded as the step's evidence (visible in inspect and the TUI) and the run resumes along the chosen edge. An unknown option is refused with the valid list; a repeated decision for an already-decided visit does not double-apply. Works across a server restart because the wait is persisted.",
"debug": "Open a fresh configured harness conversation to investigate a run using inspect/show/runs CLI queries. --harness opencode|pi|nvim chooses another configured harness or read-only nvim packet. Does not resume implementation or retry. --json returns the prompt without launching anything.",
"rpc": "Accepts a JSON protocol envelope; '-' reads stdin. Every TUI server operation is available here too. Mutating commands are not dry runs. The omni-search command is `search` (fields query, optional kind and limit) and answers with ranked `search_results`.",
"chat-adopt": "Adopt a session that already exists on the configured OpenCode endpoint so it appears in chat_sessions and can be continued. The id must be one the endpoint reports; its directory and title are taken from the endpoint's own record (never guessed), and a missing id records nothing. Adopting is always explicit and does not change the server's endpoint choice.",
"retention": "Local, explicit run-history retention; never automatic and never a side effect of another command. `report` is strictly read-only and prints what a policy would remove; `apply` removes it after approval (--yes for scripted use). Policy: --older-than DURATION selects runs created before now minus that rolling UTC duration (Go durations plus d/w; not a calendar boundary), --keep N keeps the newest N runs by run id and selects older ranks (--keep 0 keeps none). Both together are the conservative intersection; at least one is required. Protected and never removed: queued/running runs, runs waiting on a person (waiting run or step), runs with an active cleanup agent, and any run referenced by a retained run's retry provenance (transitively). A run that becomes protected before deletion aborts the whole apply. Artifacts at <root>/<run> are removed with their run and kept with protected/retained runs; the database is VACUUMed and its WAL checkpointed, with measured reclaimed bytes. Definitions and versions are immutable and never touched. --db must name an existing database (default: the standard data directory); --artifacts defaults to artifacts/ beside --db. Works without a running server.",
"search": "One read-only query over pipelines, versions, runs, steps, field text and artifacts. QUERY is free text, a bare status word (queued, running, succeeded, failed, canceled, interrupted) selecting those runs, or a lone date expression (today, yesterday, this week [Monday 00:00 local], or YYYY-MM-DD) selecting the runs created in that window. --type is the only narrowing mechanism and takes one kind: pipeline, version, run, step, field or artifact; an unknown kind is an error. Ranking: an exact name match (3) beats a name substring (2) beats a body-text match (1); equal tiers break by recency, then a stable kind/target tie-break, and every hit reports score, matched_field and recency so a client can re-rank. Bounds: the result cap defaults to 50 and clamps to 200 (0 = default); every kind scans at most 2000 rows and field-text scanning is additionally capped at 2000 steps / 1 MiB; Truncated plus scanned_steps, scanned_bytes and scan_bound_bytes report when a bound applied. Targets are stable: pipeline:<id>, pipeline:<id>:v<version>, run:<id>, run:<id>:<sequence>, run:<id>:<sequence>:<field>, or an artifact URL.",
}
var serviceHelp = map[string]string{
+3 -18
View File
@@ -44,12 +44,9 @@ func decodeControl(text string, value any) error {
}
func parseControl(s *Stage, fields map[string]string) error {
if text, ok := fields["resources"]; ok {
if err := decodeControl(text, &s.Resources); err != nil || s.Resources == nil || !validStageResources(*s.Resources) {
return fmt.Errorf("resources must contain exactly one exclusive or pool key (1–128 ASCII letters, digits, _ or -; start with a letter)")
}
delete(fields, "resources")
}
// A resources block from a definition created before the feature was removed
// is ignored: it no longer declares scheduling or validation.
delete(fields, "resources")
if text, ok := fields["limits"]; ok {
if err := decodeControl(text, &s.Limits); err != nil || s.Limits == nil {
return fmt.Errorf("limits must be an object with max_visits and optional on outcomes")
@@ -81,15 +78,7 @@ func parseControl(s *Stage, fields map[string]string) error {
return nil
}
func validStageResources(resources protocol.StageResources) bool {
return (ValidResourceKey(resources.Exclusive) && resources.Pool == "") ||
(resources.Exclusive == "" && ValidResourceKey(resources.Pool))
}
func validateControl(s Stage, stages map[string]bool) error {
if s.Resources != nil && s.Type == "human" {
return fmt.Errorf("stage %s: human stages cannot declare resources", s.Key)
}
if s.Limits != nil {
for _, outcome := range s.Limits.On {
target, exists := s.Edges[outcome]
@@ -109,7 +98,3 @@ func validateControl(s Stage, stages map[string]bool) error {
}
return nil
}
func ValidResourceKey(key string) bool {
return len(key) <= 128 && identifier.MatchString(key)
}
+13
View File
@@ -6,6 +6,19 @@ import (
"testing"
)
func TestRetiredResourcesBlockIsIgnored(t *testing.T) {
source := Template([]string{"worker"}, true) + "```resources\n{\"exclusive\":\"phone\"}\n```\n"
graph, err := Parse(source)
if err != nil {
t.Fatalf("resources block was not ignored: %v", err)
}
for _, stage := range graph.Stages {
if _, exists := stage.Blocks["resources"]; exists {
t.Fatalf("retired resources block was retained on stage %s", stage.Key)
}
}
}
func TestValidation(t *testing.T) {
valid := Template([]string{"implementer", "reviewer"}, true)
cycle := strings.Replace(valid, "success -> #success", "bad_implementation -> implementer\nsuccess -> #success", 1)
-759
View File
@@ -1,759 +0,0 @@
package pipeline
import (
"context"
"crypto/rand"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"fmt"
"strings"
"time"
"unicode/utf8"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
const maxRecoveryText = 512
func boundedRecoveryText(value string) string {
value = strings.TrimSpace(value)
return value
}
// InspectResource exposes only bounded state; capability tokens never leave this
// package or enter a protocol value.
func (s *Store) InspectResource(ctx context.Context, resource string) (protocol.ResourceInspection, error) {
if !ValidResourceKey(resource) {
return protocol.ResourceInspection{}, fmt.Errorf("invalid resource")
}
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
out := protocol.ResourceInspection{Resource: resource, Waiters: []protocol.ResourceWaiter{}, Audits: []protocol.ResourceRecoveryAudit{}}
var runID int64
var stage string
var seq, cleanup int
err := s.db.QueryRowContext(ctx, `SELECT run_id,stage_key,sequence,cleanup_confirmed FROM resource_owners WHERE resource_key=?`, resource).Scan(&runID, &stage, &seq, &cleanup)
if err == nil {
var active int
if err := s.db.QueryRowContext(ctx, `SELECT COALESCE(MAX(agent_active),0) FROM run_steps WHERE run_id=? AND sequence=?`, runID, seq).Scan(&active); err != nil {
return out, err
}
out.Owner = &protocol.ResourceOwner{RunID: runID, Stage: stage, Sequence: seq, CleanupConfirmed: cleanup != 0, Quarantined: cleanup == 0, AgentActive: active != 0}
} else if err != sql.ErrNoRows {
return out, err
}
rows, err := s.db.QueryContext(ctx, `SELECT run_id,stage_key,sequence,queued_at FROM resource_waiters WHERE resource_key=? ORDER BY queue_order LIMIT 32`, resource)
if err != nil {
return out, err
}
for rows.Next() {
var w protocol.ResourceWaiter
if err := rows.Scan(&w.RunID, &w.Stage, &w.Sequence, &w.QueuedAt); err != nil {
rows.Close()
return out, err
}
out.Waiters = append(out.Waiters, w)
}
if err := rows.Err(); err != nil {
rows.Close()
return out, err
}
rows.Close()
audits, err := s.db.QueryContext(ctx, `SELECT run_id,stage_key,sequence,actor,source,evidence,result,created_at FROM resource_recovery_audit WHERE resource_key=? ORDER BY rowid DESC LIMIT 16`, resource)
if err != nil {
return out, err
}
defer audits.Close()
for audits.Next() {
var a protocol.ResourceRecoveryAudit
a.Resource = resource
if err := audits.Scan(&a.RunID, &a.Stage, &a.Sequence, &a.Actor, &a.Source, &a.Evidence, &a.Result, &a.CreatedAt); err != nil {
audits.Close()
return out, err
}
out.Audits = append(out.Audits, a)
}
return out, audits.Err()
}
// ReconcileResource is the sole operator recovery mutation. The persisted
// capability is compared only inside this authoritative store operation.
func (s *Store) ReconcileResource(ctx context.Context, request protocol.ResourceRecoveryRequest) error {
if !ValidResourceKey(request.Resource) || request.RunID <= 0 || request.Sequence <= 0 || !identifier.MatchString(request.Stage) {
return fmt.Errorf("invalid resource recovery identity")
}
if request.Evidence == nil || !request.Evidence.CleanupConfirmed || !validRecoveryEvidence(request.Evidence.Summary) {
return fmt.Errorf("positive cleanup/restoration evidence is required")
}
if strings.TrimSpace(request.Actor) == "" || len(request.Actor) > maxRecoveryText {
return fmt.Errorf("bounded actor is required")
}
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
var token, stage string
var runID int64
var seq, cleanup int
err = tx.QueryRowContext(ctx, `SELECT run_id,stage_key,sequence,token,cleanup_confirmed FROM resource_owners WHERE resource_key=?`, request.Resource).Scan(&runID, &stage, &seq, &token, &cleanup)
if err == sql.ErrNoRows {
var count int
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND result='reconciled'`, request.Resource, request.RunID, request.Stage, request.Sequence).Scan(&count); err != nil {
return err
}
if count > 0 {
return nil
}
return fmt.Errorf("resource owner not found")
}
if err != nil {
return err
}
if runID != request.RunID || stage != request.Stage || seq != request.Sequence {
return fmt.Errorf("resource owner visit is stale")
}
if cleanup != 0 {
return fmt.Errorf("resource owner is not quarantined")
}
var status string
if err := tx.QueryRowContext(ctx, `SELECT status FROM pipeline_runs WHERE id=?`, runID).Scan(&status); err != nil {
return err
}
if status == protocol.RunStatusQueued || status == protocol.RunStatusWaiting || status == protocol.RunStatusRunning {
return fmt.Errorf("owning run is still active")
}
var active int
if err := tx.QueryRowContext(ctx, `SELECT COALESCE(MAX(agent_active),0) FROM run_steps WHERE run_id=? AND sequence=?`, runID, seq).Scan(&active); err != nil {
return err
}
if active != 0 {
return fmt.Errorf("owning agent is still active")
}
evidence := boundedRecoveryText(request.Evidence.Summary)
if request.Evidence.NoMutatingAction {
evidence = boundedRecoveryText("no mutating action occurred; " + evidence)
}
evidenceHash := sha256.Sum256([]byte(evidence))
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_recovery_audit(resource_key,run_id,stage_key,sequence,actor,source,evidence,result) VALUES(?,?,?,?,?,?,?,?)`, request.Resource, runID, stage, seq, boundedRecoveryText(request.Actor), "loopback-operator", "sha256:"+hex.EncodeToString(evidenceHash[:]), "reconciled"); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, `UPDATE resource_owners SET cleanup_confirmed=1,diagnostic='' WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND token=? AND cleanup_confirmed=0`, request.Resource, runID, stage, seq, token); err != nil {
return err
}
owner := ResourceLease{ResourceVisit: ResourceVisit{Resource: request.Resource, RunID: runID, Stage: stage, Sequence: seq}, token: token}
if _, err := deleteResourceOwner(ctx, tx, owner); err != nil {
return err
}
next, err := s.handoffResourceTx(ctx, tx, request.Resource)
if err != nil {
return err
}
if err := tx.Commit(); err != nil {
return err
}
if next.token != "" {
s.resourceWaiters[next.ResourceVisit] <- next
}
return nil
}
func validRecoveryEvidence(value string) bool {
value = strings.TrimSpace(value)
if value == "" || len(value) > maxRecoveryText || !utf8.ValidString(value) {
return false
}
if strings.EqualFold(value, "true") || strings.EqualFold(value, "false") || strings.EqualFold(value, "null") {
return false
}
if strings.HasPrefix(value, "{") || strings.HasPrefix(value, "[") {
return false
}
for _, r := range value {
if r == '\x00' || (r < 0x20 && r != '\t' && r != '\n' && r != '\r') {
return false
}
}
return true
}
// ResourceVisit identifies a pending visit, before it consumes execution accounting.
type ResourceVisit struct {
Resource string
RunID int64
Stage string
Sequence int
}
// ResourceLease is an opaque capability. Its credential is deliberately not exported
// or serializable, and must never be put in model input or public wire metadata.
type ResourceLease struct {
ResourceVisit
Target *protocol.ResourceTarget
pool string
token string
}
const maxPoolMembers = 32
const maxTargetMetadata = 16
const maxTargetMetadataKey = 128
const maxTargetValue = 128
func validateTargetMetadata(metadata map[string]string) error {
if len(metadata) > maxTargetMetadata {
return fmt.Errorf("too many target metadata fields")
}
for k, v := range metadata {
if !identifier.MatchString(k) || len(k) > maxTargetMetadataKey || len(v) > maxTargetValue || !utf8.ValidString(v) {
return fmt.Errorf("invalid target metadata")
}
}
return nil
}
func validateResourcePool(name string, members []protocol.ResourcePoolMember) error {
if !ValidResourceKey(name) || len(members) == 0 || len(members) > maxPoolMembers {
return fmt.Errorf("invalid resource pool")
}
seen := make(map[string]bool, len(members))
for i := range members {
member := &members[i]
if !ValidResourceKey(member.Member) || seen[member.Member] {
return fmt.Errorf("invalid pool member")
}
if err := validateTargetMetadata(member.Metadata); err != nil {
return err
}
seen[member.Member] = true
}
return nil
}
// ConfigureResourcePool replaces a pool declaration atomically. Existing
// owners keep their concrete member; queued requests are evaluated against the
// new declaration on the next release and are never silently transferred.
func (s *Store) ConfigureResourcePool(ctx context.Context, pool protocol.ResourcePool) error {
if err := validateResourcePool(pool.Name, pool.Members); err != nil {
return err
}
encoded, err := json.Marshal(pool.Members)
if err != nil {
return err
}
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
_, err = s.db.ExecContext(ctx, `INSERT INTO resource_pools(name,members_json) VALUES(?,?) ON CONFLICT(name) DO UPDATE SET members_json=excluded.members_json,updated_at=CURRENT_TIMESTAMP`, pool.Name, string(encoded))
return err
}
// InspectResourcePool returns the declared pool, including only bounded,
// non-secret member metadata and status.
func (s *Store) InspectResourcePool(ctx context.Context, name string) (protocol.ResourcePool, error) {
if !ValidResourceKey(name) {
return protocol.ResourcePool{}, fmt.Errorf("invalid resource pool")
}
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
members, err := s.resourcePool(ctx, s.db, name)
if err != nil {
return protocol.ResourcePool{}, err
}
return protocol.ResourcePool{Name: name, Members: members}, nil
}
func (s *Store) resourcePool(ctx context.Context, tx interface {
QueryRowContext(context.Context, string, ...any) *sql.Row
}, name string) ([]protocol.ResourcePoolMember, error) {
var encoded string
if err := tx.QueryRowContext(ctx, "SELECT members_json FROM resource_pools WHERE name=?", name).Scan(&encoded); err != nil {
return nil, err
}
var members []protocol.ResourcePoolMember
if err := json.Unmarshal([]byte(encoded), &members); err != nil {
return nil, fmt.Errorf("invalid persisted pool: %w", err)
}
if err := validateResourcePool(name, members); err != nil {
return nil, fmt.Errorf("invalid persisted pool: %w", err)
}
return members, nil
}
func poolTarget(member protocol.ResourcePoolMember) *protocol.ResourceTarget {
metadata := make(map[string]string, len(member.Metadata))
for k, v := range member.Metadata {
metadata[k] = v
}
return &protocol.ResourceTarget{Member: member.Member, Metadata: metadata}
}
// RequestResourcePool selects one free eligible concrete member while holding
// the same authority as direct resource requests.
func (s *Store) RequestResourcePool(ctx context.Context, pool string, visit ResourceVisit) (ResourceLease, <-chan ResourceLease, *protocol.ResourceWait, error) {
if !ValidResourceKey(pool) || visit.Resource != pool || visit.RunID <= 0 || visit.Sequence <= 0 || !identifier.MatchString(visit.Stage) {
return ResourceLease{}, nil, nil, fmt.Errorf("invalid resource pool visit")
}
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
if ch := s.resourceWaiters[visit]; ch != nil {
wait, err := s.resourceWait(ctx, visit.RunID)
return ResourceLease{}, ch, wait, err
}
credential := make([]byte, 32)
if _, err := rand.Read(credential); err != nil {
return ResourceLease{}, nil, nil, err
}
lease := ResourceLease{ResourceVisit: visit, token: hex.EncodeToString(credential)}
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return ResourceLease{}, nil, nil, err
}
defer tx.Rollback()
var status string
if err := tx.QueryRowContext(ctx, "SELECT status FROM pipeline_runs WHERE id=?", visit.RunID).Scan(&status); err != nil {
return ResourceLease{}, nil, nil, err
}
if status != protocol.RunStatusQueued && status != protocol.RunStatusRunning {
return ResourceLease{}, nil, nil, fmt.Errorf("run is not active")
}
var existing ResourceLease
var existingPool string
err = tx.QueryRowContext(ctx, "SELECT resource_key,run_id,stage_key,sequence,token,pool_name FROM resource_owners WHERE run_id=?", visit.RunID).Scan(&existing.Resource, &existing.RunID, &existing.Stage, &existing.Sequence, &existing.token, &existingPool)
if err == nil {
if existing.RunID != visit.RunID || existing.Stage != visit.Stage || existing.Sequence != visit.Sequence {
return ResourceLease{}, nil, nil, fmt.Errorf("resource owner visit is stale")
}
if existingPool != pool {
return ResourceLease{}, nil, nil, fmt.Errorf("resource owner pool visit is stale")
}
members, poolErr := s.resourcePool(ctx, tx, pool)
if poolErr != nil {
return ResourceLease{}, nil, nil, fmt.Errorf("unknown resource pool: %w", poolErr)
}
for _, member := range members {
if member.Member == existing.Resource {
existing.ResourceVisit = visit
existing.pool = pool
existing.Target = poolTarget(member)
return existing, nil, nil, nil
}
}
return ResourceLease{}, nil, nil, fmt.Errorf("run already owns another resource visit")
}
if err != sql.ErrNoRows {
return ResourceLease{}, nil, nil, err
}
// A queued visit may be re-entered after the process that owned its live
// channel was restarted. Reattach to the durable waiter instead of trying
// to insert a second reservation.
var persistedPool string
err = tx.QueryRowContext(ctx, "SELECT pool_name FROM resource_waiters WHERE run_id=? AND stage_key=? AND sequence=?", visit.RunID, visit.Stage, visit.Sequence).Scan(&persistedPool)
if err == nil {
if persistedPool != pool {
return ResourceLease{}, nil, nil, fmt.Errorf("resource waiter pool visit is stale")
}
wait, waitErr := s.resourceWait(ctx, visit.RunID)
if waitErr != nil {
return ResourceLease{}, nil, nil, waitErr
}
if s.resourceWaiters == nil {
s.resourceWaiters = make(map[ResourceVisit]chan ResourceLease)
}
ch := make(chan ResourceLease, 1)
s.resourceWaiters[visit] = ch
return ResourceLease{}, ch, wait, nil
}
if err != sql.ErrNoRows {
return ResourceLease{}, nil, nil, err
}
members, err := s.resourcePool(ctx, tx, pool)
if err != nil {
return ResourceLease{}, nil, nil, fmt.Errorf("unknown resource pool: %w", err)
}
for _, member := range members {
if !member.Enabled || member.Quarantined {
continue
}
var count int
if err := tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", member.Member).Scan(&count); err != nil {
return ResourceLease{}, nil, nil, err
}
if count != 0 {
continue
}
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_owners(resource_key,run_id,stage_key,sequence,token,pool_name) VALUES(?,?,?,?,?,?)`, member.Member, visit.RunID, visit.Stage, visit.Sequence, lease.token, pool); err != nil {
return ResourceLease{}, nil, nil, err
}
lease.Resource = member.Member
lease.pool = pool
lease.Target = poolTarget(member)
if err := tx.Commit(); err != nil {
return ResourceLease{}, nil, nil, err
}
return lease, nil, nil, nil
}
wait := &protocol.ResourceWait{Resource: pool, Stage: visit.Stage, Sequence: visit.Sequence, QueuedSince: time.Now().UTC().Format(time.RFC3339Nano)}
encoded, err := json.Marshal(wait)
if err != nil {
return ResourceLease{}, nil, nil, err
}
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_waiters(resource_key,pool_name,run_id,stage_key,sequence,token,queued_at) VALUES(?,?,?,?,?,?,?)`, pool, pool, visit.RunID, visit.Stage, visit.Sequence, lease.token, wait.QueuedSince); err != nil {
return ResourceLease{}, nil, nil, err
}
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='queued',current_stage=?,resource_wait=? WHERE id=? AND status IN ('queued','running')", visit.Stage, string(encoded), visit.RunID)); err != nil {
return ResourceLease{}, nil, nil, err
}
if err := tx.Commit(); err != nil {
return ResourceLease{}, nil, nil, err
}
if s.resourceWaiters == nil {
s.resourceWaiters = make(map[ResourceVisit]chan ResourceLease)
}
ch := make(chan ResourceLease, 1)
s.resourceWaiters[visit] = ch
return ResourceLease{}, ch, wait, nil
}
// RequestResource acquires immediately or registers one FIFO waiter/channel. The
// mutex covers only short DB transactions and channel bookkeeping, never execution.
func (s *Store) RequestResource(ctx context.Context, visit ResourceVisit) (ResourceLease, <-chan ResourceLease, *protocol.ResourceWait, error) {
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
if !ValidResourceKey(visit.Resource) || visit.RunID <= 0 || visit.Sequence <= 0 || !identifier.MatchString(visit.Stage) {
return ResourceLease{}, nil, nil, fmt.Errorf("invalid resource visit")
}
if ch := s.resourceWaiters[visit]; ch != nil {
wait, err := s.resourceWait(ctx, visit.RunID)
return ResourceLease{}, ch, wait, err
}
credential := make([]byte, 32)
if _, err := rand.Read(credential); err != nil {
return ResourceLease{}, nil, nil, err
}
lease := ResourceLease{ResourceVisit: visit, token: hex.EncodeToString(credential)}
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return ResourceLease{}, nil, nil, err
}
defer tx.Rollback()
var status string
if err := tx.QueryRowContext(ctx, "SELECT status FROM pipeline_runs WHERE id=?", visit.RunID).Scan(&status); err != nil {
return ResourceLease{}, nil, nil, err
}
if status != protocol.RunStatusQueued && status != protocol.RunStatusRunning {
return ResourceLease{}, nil, nil, fmt.Errorf("run is not active")
}
var existing ResourceLease
var existingPool string
err = tx.QueryRowContext(ctx, "SELECT resource_key,run_id,stage_key,sequence,token,pool_name FROM resource_owners WHERE run_id=?", visit.RunID).Scan(&existing.Resource, &existing.RunID, &existing.Stage, &existing.Sequence, &existing.token, &existingPool)
if err == nil {
if existingPool != "" {
return ResourceLease{}, nil, nil, fmt.Errorf("run already owns a pooled resource visit")
}
if existing.ResourceVisit != visit {
return ResourceLease{}, nil, nil, fmt.Errorf("run already owns another resource visit")
}
return existing, nil, nil, nil
}
if err != sql.ErrNoRows {
return ResourceLease{}, nil, nil, err
}
result, err := tx.ExecContext(ctx, `INSERT INTO resource_owners(resource_key,run_id,stage_key,sequence,token,pool_name)
VALUES(?,?,?,?,?,?) ON CONFLICT(resource_key) DO NOTHING`, visit.Resource, visit.RunID, visit.Stage, visit.Sequence, lease.token, "")
if err != nil {
return ResourceLease{}, nil, nil, err
}
n, err := result.RowsAffected()
if err != nil {
return ResourceLease{}, nil, nil, err
}
if n == 1 {
return lease, nil, nil, tx.Commit()
}
wait := &protocol.ResourceWait{Resource: visit.Resource, Stage: visit.Stage, Sequence: visit.Sequence, QueuedSince: time.Now().UTC().Format(time.RFC3339Nano)}
encoded, err := json.Marshal(wait)
if err != nil {
return ResourceLease{}, nil, nil, err
}
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_waiters(resource_key,run_id,stage_key,sequence,token,queued_at) VALUES(?,?,?,?,?,?)`, visit.Resource, visit.RunID, visit.Stage, visit.Sequence, lease.token, wait.QueuedSince); err != nil {
return ResourceLease{}, nil, nil, err
}
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='queued',current_stage=?,resource_wait=? WHERE id=? AND status IN ('queued','running')", visit.Stage, string(encoded), visit.RunID)); err != nil {
return ResourceLease{}, nil, nil, err
}
if err := tx.Commit(); err != nil {
return ResourceLease{}, nil, nil, err
}
if s.resourceWaiters == nil {
s.resourceWaiters = make(map[ResourceVisit]chan ResourceLease)
}
ch := make(chan ResourceLease, 1)
s.resourceWaiters[visit] = ch
return ResourceLease{}, ch, wait, nil
}
func (s *Store) resourceWait(ctx context.Context, id int64) (*protocol.ResourceWait, error) {
var wait protocol.ResourceWait
err := s.db.QueryRowContext(ctx, "SELECT resource_key,stage_key,sequence,queued_at FROM resource_waiters WHERE run_id=?", id).Scan(&wait.Resource, &wait.Stage, &wait.Sequence, &wait.QueuedSince)
if err == sql.ErrNoRows {
return nil, nil
}
return &wait, err
}
// CancelResource removes a queued visit, or returns a concurrently reserved grant
// to the queue before any execution is allowed. It cannot cancel another visit.
func (s *Store) CancelResource(ctx context.Context, visit ResourceVisit) error {
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
defer delete(s.resourceWaiters, visit)
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
deleted, err := tx.ExecContext(ctx, "DELETE FROM resource_waiters WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=?", visit.Resource, visit.RunID, visit.Stage, visit.Sequence)
if err != nil {
return err
}
n, err := deleted.RowsAffected()
if err != nil {
return err
}
if n > 0 {
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET resource_wait='' WHERE id=?", visit.RunID); err != nil {
return err
}
}
ch := s.resourceWaiters[visit]
// A grant is buffered while this registry entry exists. Cancellation can
// reclaim only that not-yet-accepted capability, never an executing owner.
var grant ResourceLease
if ch != nil {
select {
case grant = <-ch:
default:
}
}
if err := tx.Commit(); err != nil {
return err
}
if grant.token != "" {
if err := s.SetResourceCleanup(ctx, grant, true); err != nil {
return err
}
return s.releaseResourceLocked(ctx, grant)
}
return nil
}
// AcceptResource removes the channel exactly once. Cancellation is rechecked by
// the runner before BeginStep; a canceled accepted grant is released as unused.
func (s *Store) AcceptResource(lease ResourceLease) {
s.resourceMu.Lock()
delete(s.resourceWaiters, lease.ResourceVisit)
s.resourceMu.Unlock()
}
// HasResourceOwner is used only for orphan/restart reconciliation, never for a
// resource-free stage visit.
func (s *Store) HasResourceOwner(ctx context.Context, runID int64) (bool, error) {
var exists bool
err := s.db.QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM resource_owners WHERE run_id=?)", runID).Scan(&exists)
return exists, err
}
func (s *Store) ConfirmRecoveredSession(ctx context.Context, owner SessionOwner) error {
_, err := s.db.ExecContext(ctx, "UPDATE run_steps SET agent_active=0 WHERE run_id=? AND session_id=?", owner.RunID, owner.ID)
return err
}
// SetResourceCleanup records positive quiescence durably before release. False
// quarantines the owner; neither age nor another run can remove the lease.
func (s *Store) SetResourceCleanup(ctx context.Context, lease ResourceLease, confirmed bool) error {
if lease.pool != "" && (lease.Target == nil || !ValidResourceKey(lease.Target.Member)) {
return fmt.Errorf("invalid pooled resource lease")
}
diagnostic := ""
if !confirmed {
diagnostic = "Cleanup/restoration unconfirmed; resource quarantined pending inspected reconciliation"
}
resource := lease.Resource
if lease.pool != "" {
resource = lease.Target.Member
}
return affected(s.db.ExecContext(ctx, `UPDATE resource_owners SET cleanup_confirmed=?,diagnostic=? WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND token=?`, confirmed, diagnostic, resource, lease.RunID, lease.Stage, lease.Sequence, lease.token))
}
// ReleaseResource requires the complete capability AND positive cleanup evidence.
// Mismatches and repeated releases are harmless, including after transfer.
func (s *Store) ReleaseResource(ctx context.Context, lease ResourceLease) error {
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
return s.releaseResourceLocked(ctx, lease)
}
func (s *Store) releaseResourceLocked(ctx context.Context, lease ResourceLease) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ownerResource := lease.Resource
if lease.pool != "" {
if lease.Target == nil || !ValidResourceKey(lease.Target.Member) {
return fmt.Errorf("invalid pooled resource lease")
}
ownerResource = lease.Target.Member
}
lease.Resource = ownerResource
n, err := deleteResourceOwner(ctx, tx, lease)
if err != nil {
return err
}
if n == 0 {
return nil
}
next, err := s.handoffResourceTx(ctx, tx, ownerResource)
if err != nil {
return err
}
if err := tx.Commit(); err != nil {
return err
}
if next.token != "" {
s.resourceWaiters[next.ResourceVisit] <- next
}
return nil
}
// handoffResourceTx is shared by ordinary release and inspected recovery. It
// validates the in-memory waiter before changing durable ownership so a stale
// server cannot grant a capability to a runner that no longer exists.
func (s *Store) handoffResourceTx(ctx context.Context, tx *sql.Tx, resource string) (ResourceLease, error) {
var next ResourceLease
next.Resource = resource
rows, err := tx.QueryContext(ctx, `SELECT queue_order,resource_key,pool_name,run_id,stage_key,sequence,token FROM resource_waiters WHERE resource_key=? OR pool_name<>'' ORDER BY queue_order`, resource)
if err != nil {
return ResourceLease{}, err
}
defer rows.Close()
var selected *protocol.ResourceTarget
var queue int64
var waiterResource string
for rows.Next() {
var candidate ResourceLease
var candidatePool string
if err := rows.Scan(&queue, &waiterResource, &candidatePool, &candidate.RunID, &candidate.Stage, &candidate.Sequence, &candidate.token); err != nil {
return ResourceLease{}, err
}
if candidatePool != "" {
candidate.Resource = candidatePool
candidate.pool = candidatePool
} else {
candidate.Resource = resource
}
if candidatePool == "" {
if waiterResource != resource {
continue
}
selected = &protocol.ResourceTarget{Member: resource}
} else {
members, e := s.resourcePool(ctx, tx, candidatePool)
if e != nil {
continue
}
found := false
for _, member := range members {
if !member.Enabled || member.Quarantined {
continue
}
var count int
if e = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", member.Member).Scan(&count); e != nil {
return ResourceLease{}, e
}
if count == 0 {
selected, found = poolTarget(member), true
break
}
}
if !found {
continue
}
}
if s.resourceWaiters[candidate.ResourceVisit] == nil {
if candidatePool == "" {
return ResourceLease{}, fmt.Errorf("resource waiter has no live runner; refusing handoff")
}
continue
}
next = candidate
next.Target = selected
break
}
if err := rows.Err(); err != nil {
return ResourceLease{}, err
}
if next.token == "" {
return ResourceLease{}, nil
}
ownerResource := next.Resource
if next.pool != "" {
ownerResource = next.Target.Member
}
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_owners(resource_key,run_id,stage_key,sequence,token,pool_name) VALUES(?,?,?,?,?,?)`, ownerResource, next.RunID, next.Stage, next.Sequence, next.token, next.pool); err != nil {
return ResourceLease{}, err
}
if _, err := tx.ExecContext(ctx, `DELETE FROM resource_waiters WHERE queue_order=?`, queue); err != nil {
return ResourceLease{}, err
}
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET resource_wait='' WHERE id=?", next.RunID); err != nil {
return ResourceLease{}, err
}
if next.pool == "" {
next.Resource = ownerResource
}
return next, nil
}
func deleteResourceOwner(ctx context.Context, tx *sql.Tx, lease ResourceLease) (int64, error) {
result, err := tx.ExecContext(ctx, `DELETE FROM resource_owners WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND token=? AND cleanup_confirmed=1`, lease.Resource, lease.RunID, lease.Stage, lease.Sequence, lease.token)
if err != nil {
return 0, err
}
return result.RowsAffected()
}
// recoverResourceOwners runs only at startup, after pending waiters have been
// interrupted and removed. Every reclamation uses the persisted capability;
// absent cleanup evidence is never replaced with a timestamp-based expiry.
func recoverResourceOwners(ctx context.Context, tx *sql.Tx) error {
rows, err := tx.QueryContext(ctx, "SELECT resource_key,run_id,stage_key,sequence,token FROM resource_owners WHERE cleanup_confirmed=1")
if err != nil {
return err
}
var leases []ResourceLease
for rows.Next() {
var lease ResourceLease
if err := rows.Scan(&lease.Resource, &lease.RunID, &lease.Stage, &lease.Sequence, &lease.token); err != nil {
rows.Close()
return err
}
leases = append(leases, lease)
}
err = rows.Err()
rows.Close()
if err != nil {
return err
}
for _, lease := range leases {
if _, err := deleteResourceOwner(ctx, tx, lease); err != nil {
return err
}
}
_, err = tx.ExecContext(ctx, "UPDATE resource_owners SET diagnostic='Server stopped; resource quarantined pending inspected cleanup/restoration'")
return err
}
@@ -0,0 +1,66 @@
package pipeline
import (
"context"
"path/filepath"
"testing"
)
// TestOpenDropsRetiredResourceSchema proves an existing database that still
// carries the removed pool/ownership/wait-queue tables and the run queue
// snapshot column opens cleanly, losing only that retired state. A second open
// proves the removal is idempotent.
func TestOpenDropsRetiredResourceSchema(t *testing.T) {
path := filepath.Join(t.TempDir(), "legacy-resources.db")
store, err := Open(path)
if err != nil {
t.Fatal(err)
}
if _, err := store.db.Exec(`ALTER TABLE pipeline_runs ADD COLUMN resource_wait TEXT NOT NULL DEFAULT '';
CREATE TABLE resource_owners (
resource_key TEXT PRIMARY KEY, run_id INTEGER NOT NULL REFERENCES pipeline_runs(id),
stage_key TEXT NOT NULL, sequence INTEGER NOT NULL, token TEXT NOT NULL,
cleanup_confirmed INTEGER NOT NULL DEFAULT 0, diagnostic TEXT NOT NULL DEFAULT '');
CREATE TABLE resource_waiters (
queue_order INTEGER PRIMARY KEY AUTOINCREMENT, resource_key TEXT NOT NULL,
run_id INTEGER NOT NULL UNIQUE REFERENCES pipeline_runs(id), stage_key TEXT NOT NULL,
sequence INTEGER NOT NULL, token TEXT NOT NULL, queued_at TEXT NOT NULL);
CREATE TABLE resource_pools (
name TEXT PRIMARY KEY, members_json TEXT NOT NULL, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);
CREATE TABLE resource_recovery_audit (
resource_key TEXT NOT NULL, run_id INTEGER NOT NULL, stage_key TEXT NOT NULL,
sequence INTEGER NOT NULL, actor TEXT NOT NULL, source TEXT NOT NULL,
evidence TEXT NOT NULL, result TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);`); err != nil {
t.Fatal(err)
}
if err := store.Close(); err != nil {
t.Fatal(err)
}
for attempt := 0; attempt < 2; attempt++ {
store, err = Open(path)
if err != nil {
t.Fatalf("open existing database (attempt %d): %v", attempt+1, err)
}
if runs, err := store.ListRuns(context.Background()); err != nil || len(runs) != 0 {
t.Fatalf("list runs after migration (attempt %d): %+v %v", attempt+1, runs, err)
}
for _, table := range []string{"resource_owners", "resource_waiters", "resource_pools", "resource_recovery_audit"} {
var name string
err := store.db.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name=?", table).Scan(&name)
if err == nil {
t.Fatalf("retired table %s survived migration", table)
}
}
var count int
if err := store.db.QueryRow("SELECT COUNT(*) FROM pragma_table_info('pipeline_runs') WHERE name='resource_wait'").Scan(&count); err != nil {
t.Fatal(err)
}
if count != 0 {
t.Fatal("retired resource_wait column survived migration")
}
if err := store.Close(); err != nil {
t.Fatal(err)
}
}
}
-779
View File
@@ -1,779 +0,0 @@
package pipeline
import (
"context"
"encoding/json"
"fmt"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
func TestResourceDeclaration(t *testing.T) {
base := Template([]string{"work"}, true)
for _, value := range []string{`{"exclusive":"phone-X"}`, `{ "exclusive" : "A_1" }`} {
graph, err := Parse(base + "```resources\n" + value + "\n```\n")
if err != nil || graph.Stages[0].Resources == nil {
t.Fatalf("%s: %v", value, err)
}
if _, ok := StageDefinition(graph.Stages[0]).Blocks["resources"]; ok {
t.Fatal("runner control reached executor")
}
encoded, _ := json.Marshal(graph)
if !strings.Contains(string(encoded), `"resources"`) {
t.Fatal("resource lost in persisted graph")
}
}
for _, value := range []string{`{}`, `null`, `[]`, `{"exclusive":""}`, `{"exclusive":"a b"}`, `{"exclusive":"a","exclusive":"b"}`, `{"exclusive":"a","extra":1}`, `{"exclusive":["a","b"]}`, fmt.Sprintf(`{"exclusive":%q}`, strings.Repeat("a", 129))} {
if _, err := Parse(base + "```resources\n" + value + "\n```\n"); err == nil {
t.Fatalf("accepted %s", value)
}
}
human := "# stage gate\n```prompt\nChoose\n```\n```configs\ntype=human\n```\n```edges\ngo -> #success\n```\n```resources\n{\"exclusive\":\"phone\"}\n```\n"
if _, err := Parse(human); err == nil {
t.Fatal("human acquired a resource")
}
}
func TestResourcePoolDeclaration(t *testing.T) {
base := Template([]string{"work"}, true)
graph, err := Parse(base + "```resources\n{\"pool\":\"android-targets\"}\n```\n")
if err != nil || graph.Stages[0].Resources == nil || graph.Stages[0].Resources.Pool != "android-targets" {
t.Fatalf("pool declaration: %#v, %v", graph.Stages[0].Resources, err)
}
for _, value := range []string{`{"exclusive":"phone","pool":"targets"}`, `{"pool":""}`, `{"pool":"a b"}`} {
if _, err := Parse(base + "```resources\n" + value + "\n```\n"); err == nil {
t.Fatalf("accepted invalid pool declaration %s", value)
}
}
}
func TestResourcePoolAllocatesConcreteMembersAndExcludesDirectOwner(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "android-targets", Members: []protocol.ResourcePoolMember{
{Member: "phone", Enabled: true, Metadata: map[string]string{"kind": "phone"}},
{Member: "emulator", Enabled: true, Metadata: map[string]string{"kind": "emulator"}},
}}); err != nil {
t.Fatal(err)
}
phone := acquireTestResource(t, s, create("phone"))
visit := create("android-targets")
lease, ch, _, err := s.RequestResourcePool(ctx, "android-targets", visit)
if err != nil || ch != nil || lease.Target == nil || lease.Target.Member != "emulator" {
t.Fatalf("pool selected %#v, channel=%v, err=%v", lease.Target, ch, err)
}
if lease.Target.Metadata["kind"] != "emulator" {
t.Fatal("missing bounded target metadata")
}
releaseTestResource(t, s, lease)
releaseTestResource(t, s, phone)
}
func TestResourcePoolFIFOWakeAndQuarantineIsolation(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
{Member: "held", Enabled: true}, {Member: "free", Enabled: true}, {Member: "bad", Enabled: false, Quarantined: true},
}}); err != nil {
t.Fatal(err)
}
held := acquireTestResource(t, s, create("held"))
first := create("targets")
firstLease, firstCh, _, err := s.RequestResourcePool(ctx, "targets", first)
if err != nil || firstCh != nil || firstLease.Target == nil {
t.Fatalf("expected immediate free member: %v", err)
}
// The second request waits once both concrete members are held.
second := create("targets")
_, secondCh, _, err := s.RequestResourcePool(ctx, "targets", second)
if err != nil || secondCh == nil {
t.Fatalf("expected pool waiter: %v", err)
}
releaseTestResource(t, s, firstLease)
select {
case next := <-secondCh:
if next.Target == nil || next.Target.Member != "free" {
t.Fatalf("wrong handoff: %#v", next.Target)
}
s.AcceptResource(next)
releaseTestResource(t, s, next)
case <-time.After(time.Second):
t.Fatal("pool waiter not woken")
}
releaseTestResource(t, s, held)
}
func TestResourcePoolHandoffUsesAnotherMemberWhenReleasedTargetQuarantined(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
pool := protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
{Member: "phone", Enabled: true}, {Member: "emulator", Enabled: false},
}}
if err := s.ConfigureResourcePool(ctx, pool); err != nil {
t.Fatal(err)
}
owner, ready, _, err := s.RequestResourcePool(ctx, pool.Name, create(pool.Name))
if err != nil || ready != nil || owner.Target.Member != "phone" {
t.Fatalf("owner: %+v %v", owner, err)
}
_, waiter, _, err := s.RequestResourcePool(ctx, pool.Name, create(pool.Name))
if err != nil || waiter == nil {
t.Fatalf("waiter: %v", err)
}
pool.Members[0].Quarantined = true
pool.Members[1].Enabled = true
if err := s.ConfigureResourcePool(ctx, pool); err != nil {
t.Fatal(err)
}
releaseTestResource(t, s, owner)
select {
case next := <-waiter:
if next.Target == nil || next.Target.Member != "emulator" {
t.Fatalf("wrong target: %+v", next.Target)
}
s.AcceptResource(next)
releaseTestResource(t, s, next)
case <-time.After(time.Second):
t.Fatal("quarantined released phone blocked the free emulator")
}
}
func TestResourcePoolRejectsStaleVisitForExistingOwner(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
{Member: "phone", Enabled: true},
}}); err != nil {
t.Fatal(err)
}
owner := acquireTestResource(t, s, create("phone"))
stale := owner.ResourceVisit
stale.Resource = "targets"
stale.Sequence++
if _, ch, _, err := s.RequestResourcePool(ctx, "targets", stale); err == nil || ch != nil {
t.Fatalf("stale pool visit was accepted: channel=%v err=%v", ch, err)
}
if err := s.SetResourceCleanup(ctx, owner, true); err != nil {
t.Fatal(err)
}
if err := s.ReleaseResource(ctx, owner); err != nil {
t.Fatal(err)
}
}
func TestResourcePoolAndDirectVisitsCannotBeReinterpreted(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{{Member: "phone", Enabled: true}}}); err != nil {
t.Fatal(err)
}
pooled := create("targets")
lease, ch, _, err := s.RequestResourcePool(ctx, "targets", pooled)
if err != nil || ch != nil {
t.Fatalf("pool acquire: %v %v", ch, err)
}
if _, ch, _, err := s.RequestResource(ctx, ResourceVisit{Resource: "phone", RunID: pooled.RunID, Stage: pooled.Stage, Sequence: pooled.Sequence}); err == nil || ch != nil {
t.Fatalf("pooled owner was accepted as direct lease: channel=%v err=%v", ch, err)
}
releaseTestResource(t, s, lease)
}
func TestResourcePoolConfigurationAndOwnerSurviveReopen(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
{Member: "phone", Enabled: true, Metadata: map[string]string{"kind": "phone"}},
}}); err != nil {
t.Fatal(err)
}
visit := create("targets")
owner, ch, _, err := s.RequestResourcePool(ctx, "targets", visit)
if err != nil || ch != nil {
t.Fatalf("pool acquire: %v %v", ch, err)
}
path := s.path
if err := s.Close(); err != nil {
t.Fatal(err)
}
reopened, err := Open(path)
if err != nil {
t.Fatal(err)
}
defer reopened.Close()
recovered, ch, _, err := reopened.RequestResourcePool(ctx, "targets", visit)
if err != nil || ch != nil || recovered.Target == nil || recovered.Target.Member != "phone" || recovered.Target.Metadata["kind"] != "phone" {
t.Fatalf("reopened pool owner not recovered: target=%#v channel=%v err=%v", recovered.Target, ch, err)
}
if err := reopened.SetResourceCleanup(ctx, recovered, true); err != nil {
t.Fatal(err)
}
if err := reopened.ReleaseResource(ctx, recovered); err != nil {
t.Fatal(err)
}
_ = owner
}
func TestResourcePoolAllocatesDistinctMembersConcurrently(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
{Member: "phone", Enabled: true}, {Member: "emulator", Enabled: true},
}}); err != nil {
t.Fatal(err)
}
visits := []ResourceVisit{create("targets"), create("targets")}
leases := make(chan ResourceLease, len(visits))
errs := make(chan error, len(visits))
var wg sync.WaitGroup
for _, visit := range visits {
wg.Add(1)
go func(v ResourceVisit) {
defer wg.Done()
lease, ch, _, err := s.RequestResourcePool(ctx, "targets", v)
if err != nil {
errs <- err
return
}
if ch != nil {
select {
case lease = <-ch:
case <-time.After(time.Second):
errs <- fmt.Errorf("pool request remained queued")
return
}
s.AcceptResource(lease)
}
leases <- lease
}(visit)
}
wg.Wait()
close(errs)
for err := range errs {
t.Fatal(err)
}
got := map[string]bool{}
for range visits {
lease := <-leases
if lease.Target == nil || got[lease.Target.Member] {
t.Fatalf("members were not distinct: %#v", lease.Target)
}
got[lease.Target.Member] = true
releaseTestResource(t, s, lease)
}
}
func TestResourcePoolRejectsInvalidDeclarations(t *testing.T) {
s, _ := resourceStore(t)
ctx := context.Background()
cases := []protocol.ResourcePool{
{Name: "missing-members"},
{Name: "duplicate", Members: []protocol.ResourcePoolMember{{Member: "phone"}, {Member: "phone"}}},
{Name: "bad-member", Members: []protocol.ResourcePoolMember{{Member: "not valid"}}},
{Name: "bad-metadata", Members: []protocol.ResourcePoolMember{{Member: "phone", Metadata: map[string]string{"bad key": "x"}}}},
{Name: "oversized-metadata-key", Members: []protocol.ResourcePoolMember{{Member: "phone", Metadata: map[string]string{strings.Repeat("k", maxTargetMetadataKey+1): "x"}}}},
}
for _, pool := range cases {
if err := s.ConfigureResourcePool(ctx, pool); err == nil {
t.Fatalf("accepted invalid pool %#v", pool)
}
}
visit := ResourceVisit{Resource: "unknown", RunID: 1, Stage: "work", Sequence: 1}
if _, _, _, err := s.RequestResourcePool(ctx, "unknown", visit); err == nil {
t.Fatal("accepted unknown pool")
}
}
func resourceStore(t *testing.T) (*Store, func(string) ResourceVisit) {
t.Helper()
ctx := context.Background()
s, err := Open(filepath.Join(t.TempDir(), "resources.db"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { s.Close() })
if _, err := s.Submit(ctx, "resource", Template([]string{"work"}, true)); err != nil {
t.Fatal(err)
}
items, _ := s.List(ctx)
d, _ := s.Get(ctx, items[0].ID, 1)
count := 0
return s, func(resource string) ResourceVisit {
count++
id, _, err := s.CreateRun(ctx, d, fmt.Sprint(count), "workspace", false, nil, nil)
if err != nil {
t.Fatal(err)
}
return ResourceVisit{Resource: resource, RunID: id, Stage: "work", Sequence: 1}
}
}
func acquireTestResource(t *testing.T, s *Store, visit ResourceVisit) ResourceLease {
t.Helper()
l, ch, _, err := s.RequestResource(context.Background(), visit)
if err != nil || ch != nil {
t.Fatalf("acquire: %v %v", ch, err)
}
return l
}
func releaseTestResource(t *testing.T, s *Store, lease ResourceLease) {
t.Helper()
if err := s.SetResourceCleanup(context.Background(), lease, true); err != nil {
t.Fatal(err)
}
if err := s.ReleaseResource(context.Background(), lease); err != nil {
t.Fatal(err)
}
}
func TestResourceFIFOOwnershipCancellationAndRegistry(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
a := acquireTestResource(t, s, create("X"))
duplicate := acquireTestResource(t, s, a.ResourceVisit)
if duplicate != a {
t.Fatal("duplicate acquisition changed capability")
}
var visits []ResourceVisit
var channels []<-chan ResourceLease
for i := 0; i < 4; i++ {
v := create("X")
_, ch, wait, err := s.RequestResource(ctx, v)
if err != nil || ch == nil || wait == nil {
t.Fatal("missing waiter", err)
}
_, again, _, err := s.RequestResource(ctx, v)
if err != nil || again != ch {
t.Fatal("duplicate waiter registered", err)
}
visits = append(visits, v)
channels = append(channels, ch)
}
if err := s.ReleaseResource(ctx, a); err != nil {
t.Fatal(err)
} // no cleanup receipt
for _, ch := range channels {
select {
case <-ch:
t.Fatal("unconfirmed release granted")
default:
}
}
if err := s.SetResourceCleanup(ctx, a, true); err != nil {
t.Fatal(err)
}
for _, corrupt := range []func(*ResourceLease){func(l *ResourceLease) { l.RunID++ }, func(l *ResourceLease) { l.Stage = "other" }, func(l *ResourceLease) { l.Sequence++ }, func(l *ResourceLease) { l.token = "wrong" }} {
bad := a
corrupt(&bad)
if err := s.ReleaseResource(ctx, bad); err != nil {
t.Fatal(err)
}
}
for _, ch := range channels {
select {
case <-ch:
t.Fatal("unconfirmed/mismatched release granted")
default:
}
}
if err := s.CancelResource(ctx, visits[1]); err != nil {
t.Fatal(err)
}
releaseTestResource(t, s, a)
for _, i := range []int{0, 2, 3} {
var grant ResourceLease
select {
case grant = <-channels[i]:
case <-time.After(time.Second):
t.Fatal("FIFO grant not delivered")
}
s.AcceptResource(grant)
if grant.ResourceVisit != visits[i] {
t.Fatal("wrong FIFO owner")
}
// An old owner's repeated release cannot release the next owner.
if err := s.ReleaseResource(ctx, a); err != nil {
t.Fatal(err)
}
for _, j := range []int{2, 3} {
if j > i {
select {
case <-channels[j]:
t.Fatal("multiple grants")
default:
}
}
}
releaseTestResource(t, s, grant)
}
if len(s.resourceWaiters) != 0 {
t.Fatal("channel registry leaked")
}
for i := 0; i < 30; i++ {
owner := acquireTestResource(t, s, create("X"))
v := create("X")
_, _, _, err := s.RequestResource(ctx, v)
if err != nil {
t.Fatal(err)
}
releaseTestResource(t, s, owner) // race window: granted but not accepted
if err := s.CancelResource(ctx, v); err != nil {
t.Fatal(err)
}
if len(s.resourceWaiters) != 0 {
t.Fatal("grant/cancel leaked channel")
}
}
for i := 0; i < 30; i++ {
owner := acquireTestResource(t, s, create("X"))
v := create("X")
if _, _, _, err := s.RequestResource(ctx, v); err != nil {
t.Fatal(err)
}
if err := s.SetResourceCleanup(ctx, owner, true); err != nil {
t.Fatal(err)
}
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
if err := s.ReleaseResource(ctx, owner); err != nil {
t.Error(err)
}
}()
go func() {
defer wg.Done()
if err := s.CancelResource(ctx, v); err != nil {
t.Error(err)
}
}()
wg.Wait()
if len(s.resourceWaiters) != 0 {
t.Fatal("concurrent grant/cancel leaked channel")
}
}
var owners, waiters int
s.db.QueryRow("SELECT COUNT(*) FROM resource_owners").Scan(&owners)
s.db.QueryRow("SELECT COUNT(*) FROM resource_waiters").Scan(&waiters)
if owners != 0 || waiters != 0 {
t.Fatalf("stranded owners/waiters: %d/%d", owners, waiters)
}
}
func TestResourceCancellationPersistenceFailureRemovesChannel(t *testing.T) {
s, create := resourceStore(t)
acquireTestResource(t, s, create("X"))
v := create("X")
if _, _, _, err := s.RequestResource(context.Background(), v); err != nil {
t.Fatal(err)
}
if err := s.Close(); err != nil {
t.Fatal(err)
}
if err := s.CancelResource(context.Background(), v); err == nil {
t.Fatal("closed DB cancellation claimed success")
}
if len(s.resourceWaiters) != 0 {
t.Fatal("persistence failure leaked channel")
}
}
func TestResourceRestartQuarantineAndRetention(t *testing.T) {
for _, confirmed := range []bool{false, true} {
t.Run(fmt.Sprint(confirmed), func(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
a := acquireTestResource(t, s, create("X"))
b := create("X")
if _, _, _, err := s.RequestResource(ctx, b); err != nil {
t.Fatal(err)
}
if confirmed {
if err := s.SetResourceCleanup(ctx, a, true); err != nil {
t.Fatal(err)
}
}
path := s.path
if err := s.Close(); err != nil {
t.Fatal(err)
}
reopened, err := Open(path)
if err != nil {
t.Fatal(err)
}
defer reopened.Close()
if err := reopened.RecoverRuns(ctx); err != nil {
t.Fatal(err)
}
if err := reopened.RecoverRuns(ctx); err != nil {
t.Fatal(err)
}
detail, err := reopened.GetRun(ctx, b.RunID)
if err != nil || detail.Status != "interrupted" || detail.ResourceWait != nil || len(detail.Steps) != 0 || !strings.Contains(detail.Diagnostic, "resource") {
t.Fatal("queued visit replayed/lost diagnostic", detail, err)
}
exists, err := reopened.HasResourceOwner(ctx, a.RunID)
if err != nil || exists == confirmed {
t.Fatal("incorrect recovery disposition", exists, err)
}
report, err := reopened.RetentionReport(ctx, RetentionPolicy{Keep: 0}, "", time.Now())
if err != nil {
t.Fatal(err)
}
protected := false
for _, p := range report.Protected {
if p.ID == a.RunID {
protected = true
}
}
if protected == confirmed {
t.Fatal("quarantined evidence not protected")
}
})
}
}
func TestResourceReconcileMissingWaiterPreservesQuarantine(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
owner := acquireTestResource(t, s, create("X"))
if _, err := s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
t.Fatal(err)
}
waiter := create("X")
if _, ch, _, err := s.RequestResource(ctx, waiter); err != nil || ch == nil {
t.Fatalf("register waiter: %v", err)
}
// Simulate a runner disappearing while its durable waiter remains. The
// authoritative operation must reject the handoff before audit or cleanup.
delete(s.resourceWaiters, waiter)
err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
Resource: owner.Resource,
RunID: owner.RunID,
Stage: owner.Stage,
Sequence: owner.Sequence,
Actor: "operator",
Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, NoMutatingAction: true, Summary: "inspection confirmed no mutating action occurred"},
})
if err == nil {
t.Fatal("reconciled without a live waiter channel")
}
var cleanup, audits, owners int
if err := s.db.QueryRow("SELECT cleanup_confirmed FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&cleanup); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=?", owner.Resource).Scan(&audits); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&owners); err != nil {
t.Fatal(err)
}
if cleanup != 0 || audits != 0 || owners != 1 {
t.Fatalf("failed recovery changed state: cleanup=%d audits=%d owners=%d", cleanup, audits, owners)
}
}
func TestResourceReconcileTransfersAndAudits(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
owner := acquireTestResource(t, s, create("X"))
if _, err := s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
t.Fatal(err)
}
waiter := create("X")
_, ch, _, err := s.RequestResource(ctx, waiter)
if err != nil || ch == nil {
t.Fatalf("register waiter: %v", err)
}
if err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
Resource: owner.Resource,
RunID: owner.RunID,
Stage: owner.Stage,
Sequence: owner.Sequence,
Actor: "operator",
Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, NoMutatingAction: true, Summary: "inspection confirmed no mutating action occurred"},
}); err != nil {
t.Fatal(err)
}
select {
case next := <-ch:
s.AcceptResource(next)
if next.ResourceVisit != waiter {
t.Fatalf("wrong handoff: %#v", next.ResourceVisit)
}
case <-time.After(time.Second):
t.Fatal("reconciliation did not wake waiter")
}
var audits int
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=? AND result='reconciled'", owner.Resource).Scan(&audits); err != nil {
t.Fatal(err)
}
if audits != 1 {
t.Fatalf("recovery audit count=%d", audits)
}
}
func TestResourceReconcileReplayAfterSuccessorHandoff(t *testing.T) {
s, create := resourceStore(t)
ctx := context.Background()
owner := acquireTestResource(t, s, create("X"))
if _, err := s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
t.Fatal(err)
}
successor := create("X")
_, ch, _, err := s.RequestResource(ctx, successor)
if err != nil || ch == nil {
t.Fatalf("register successor: %v", err)
}
request := protocol.ResourceRecoveryRequest{
Resource: owner.Resource,
RunID: owner.RunID,
Stage: owner.Stage,
Sequence: owner.Sequence,
Actor: "operator",
Evidence: &protocol.ResourceRecoveryEvidence{
CleanupConfirmed: true,
NoMutatingAction: true,
Summary: "inspection confirmed no mutating action occurred",
},
}
if err := s.ReconcileResource(ctx, request); err != nil {
t.Fatal(err)
}
var successorLease ResourceLease
select {
case successorLease = <-ch:
case <-time.After(time.Second):
t.Fatal("reconciliation did not hand off to successor")
}
if successorLease.ResourceVisit != successor {
t.Fatalf("wrong successor visit: %#v", successorLease.ResourceVisit)
}
if successorLease.token == "" {
t.Fatal("successor capability was empty")
}
s.AcceptResource(successorLease)
// The old request is an idempotent replay after the handoff. It must not
// match or release the successor's private capability.
if err := s.ReconcileResource(ctx, request); err == nil {
t.Fatal("old recovery replay was accepted against successor owner")
}
inspection, err := s.InspectResource(ctx, owner.Resource)
if err != nil {
t.Fatal(err)
}
if inspection.Owner == nil || inspection.Owner.RunID != successor.RunID || inspection.Owner.Stage != successor.Stage || inspection.Owner.Sequence != successor.Sequence {
t.Fatalf("successor was not preserved: %#v", inspection.Owner)
}
if len(inspection.Waiters) != 0 || len(inspection.Audits) != 1 || inspection.Audits[0].RunID != owner.RunID {
t.Fatalf("replay changed handoff state: waiters=%d audits=%#v", len(inspection.Waiters), inspection.Audits)
}
if len(s.resourceWaiters) != 0 {
t.Fatal("successor channel registry leaked")
}
var owners, waiters, audits int
if err := s.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&owners); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_waiters WHERE resource_key=?", owner.Resource).Scan(&waiters); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=?", owner.Resource).Scan(&audits); err != nil {
t.Fatal(err)
}
if owners != 1 || waiters != 0 || audits != 1 {
t.Fatalf("unexpected post-replay counts: owners=%d waiters=%d audits=%d", owners, waiters, audits)
}
}
func TestResourceReconcileRequiresQuarantinedInactiveOwner(t *testing.T) {
ctx := context.Background()
test := func(t *testing.T, setup func(*Store, ResourceLease) int) {
s, create := resourceStore(t)
owner := acquireTestResource(t, s, create("X"))
expectedCleanup := setup(s, owner)
err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
Resource: owner.Resource, RunID: owner.RunID, Stage: owner.Stage, Sequence: owner.Sequence,
Actor: "operator", Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, Summary: "verified"},
})
if err == nil {
t.Fatal("reconciled unsafe owner")
}
var owners, audits, cleanup int
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_owners").Scan(&owners); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit").Scan(&audits); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRow("SELECT cleanup_confirmed FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&cleanup); err != nil {
t.Fatal(err)
}
if owners != 1 || audits != 0 || cleanup != expectedCleanup {
t.Fatalf("unsafe recovery changed state: owners=%d audits=%d cleanup=%d", owners, audits, cleanup)
}
}
t.Run("active run", func(t *testing.T) {
test(t, func(s *Store, owner ResourceLease) int {
if _, err := s.db.Exec("UPDATE pipeline_runs SET status=? WHERE id=?", protocol.RunStatusWaiting, owner.RunID); err != nil {
t.Fatal(err)
}
return 0
})
})
t.Run("active agent", func(t *testing.T) {
test(t, func(s *Store, owner ResourceLease) int {
if _, err := s.db.Exec("UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
t.Fatal(err)
}
if _, err := s.db.Exec("INSERT INTO run_steps(run_id,sequence,stage_key,status,agent_active,input_json) VALUES(?,?,?,?,?, '{}')", owner.RunID, owner.Sequence, owner.Stage, "running", 1); err != nil {
t.Fatal(err)
}
return 0
})
})
t.Run("already confirmed", func(t *testing.T) {
test(t, func(s *Store, owner ResourceLease) int {
if err := s.SetResourceCleanup(ctx, owner, true); err != nil {
t.Fatal(err)
}
return 1
})
})
}
func TestResourceReconcileRejectsMalformedEvidenceBeforeMutation(t *testing.T) {
ctx := context.Background()
for _, summary := range []string{"", "true", "FALSE", "null", "verified\x00cleanup", strings.Repeat("x", maxRecoveryText+1)} {
t.Run(fmt.Sprintf("%q", summary), func(t *testing.T) {
s, create := resourceStore(t)
owner := acquireTestResource(t, s, create("X"))
err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
Resource: owner.Resource, RunID: owner.RunID, Stage: owner.Stage, Sequence: owner.Sequence,
Actor: "operator", Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, Summary: summary},
})
if err == nil {
t.Fatal("accepted malformed evidence")
}
var owners, audits, cleanup int
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_owners").Scan(&owners); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit").Scan(&audits); err != nil {
t.Fatal(err)
}
if err := s.db.QueryRow("SELECT cleanup_confirmed FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&cleanup); err != nil {
t.Fatal(err)
}
if owners != 1 || audits != 0 || cleanup != 0 {
t.Fatalf("invalid evidence changed state: owners=%d audits=%d cleanup=%d", owners, audits, cleanup)
}
})
}
}
+7 -22
View File
@@ -130,12 +130,11 @@ type retentionQuerier interface {
// retentionRunInfo is one pipeline_runs row plus the step aggregates protection
// needs, read in the same statement so the report never issues a per-run query.
type retentionRunInfo struct {
id int64
status string
createdAt time.Time
waitingSteps int
activeSteps int
resourceOwners int
id int64
status string
createdAt time.Time
waitingSteps int
activeSteps int
}
// protectionReasons lists why a run must not be removed for a reason intrinsic
@@ -156,9 +155,6 @@ func (r retentionRunInfo) protectionReasons() []string {
if r.activeSteps > 0 {
reasons = append(reasons, "cleanup agent still active (agent_active=1)")
}
if r.resourceOwners > 0 {
reasons = append(reasons, "resource ownership retained (possibly quarantined)")
}
return reasons
}
@@ -175,20 +171,9 @@ type retentionSelection struct {
// report is a bounded number of round trips rather than one query per run.
func selectRetentionRuns(ctx context.Context, q retentionQuerier, policy RetentionPolicy, now time.Time) (retentionSelection, error) {
var sel retentionSelection
// Maintenance is deliberately non-migrating and may inspect a database
// whose server predates resource support.
var resources bool
if err := q.QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE type='table' AND name='resource_owners')").Scan(&resources); err != nil {
return sel, err
}
owners := "0"
if resources {
owners = "(SELECT COUNT(*) FROM resource_owners o WHERE o.run_id=r.id)"
}
rows, err := q.QueryContext(ctx, `SELECT r.id,r.status,strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),
COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.status=?),0),
COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.agent_active=1),0),
`+owners+` `+
COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.agent_active=1),0) `+
runFrom+` ORDER BY r.id DESC`, protocol.StepStatusWaiting)
if err != nil {
return sel, err
@@ -197,7 +182,7 @@ COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.agent_activ
for rows.Next() {
var run retentionRunInfo
var created string
if err := rows.Scan(&run.id, &run.status, &created, &run.waitingSteps, &run.activeSteps, &run.resourceOwners); err != nil {
if err := rows.Scan(&run.id, &run.status, &created, &run.waitingSteps, &run.activeSteps); err != nil {
return sel, err
}
run.createdAt = parseRetentionTime(created)
+3 -8
View File
@@ -21,19 +21,14 @@ const runStamps = `strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),COALESCE(strftime
const runSelect = `SELECT r.id,r.pipeline_id,p.name,r.version,v.content_hash,r.status,r.outcome,r.current_stage,
` + runStamps + `,r.diagnostic,
CASE WHEN r.status='succeeded' THEN r.output_json ELSE COALESCE((SELECT output_json FROM run_steps WHERE run_id=r.id AND status='completed' ORDER BY sequence DESC LIMIT 1),'{}') END,
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title,r.model,r.stage_models,r.resource_wait
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title,r.model,r.stage_models
` + runFrom + ` JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version`
func scanRun(row interface{ Scan(...any) error }, r *protocol.RunSummary) error {
var output, stageModels, resourceWait string
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title, &r.Model, &stageModels, &resourceWait); err != nil {
var output, stageModels string
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title, &r.Model, &stageModels); err != nil {
return err
}
if resourceWait != "" {
if err := json.Unmarshal([]byte(resourceWait), &r.ResourceWait); err != nil {
return err
}
}
if stageModels != "" && stageModels != "{}" {
if err := json.Unmarshal([]byte(stageModels), &r.StageModels); err != nil {
return err
+2 -2
View File
@@ -44,7 +44,7 @@ func (s *Store) BeginStep(ctx context.Context, id int64, sequence int, key strin
return err
}
defer tx.Rollback()
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='running',current_stage=?,resource_wait='' WHERE id=? AND status IN ('queued','running')", key, id)); err != nil {
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='running',current_stage=? WHERE id=? AND status IN ('queued','running')", key, id)); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, "INSERT INTO run_steps(run_id,sequence,stage_key,input_json) VALUES (?,?,?,?)", id, sequence, key, string(encoded)); err != nil {
@@ -310,5 +310,5 @@ func (s *Store) FinishRun(ctx context.Context, id int64, status, outcome, diagno
if err != nil {
return err
}
return affected(s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status=?,outcome=?,diagnostic=?,output_json=?,finished_at=CURRENT_TIMESTAMP,resource_wait='' WHERE id=? AND status IN ('queued','running')", status, outcome, diagnostic, string(data), id))
return affected(s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status=?,outcome=?,diagnostic=?,output_json=?,finished_at=CURRENT_TIMESTAMP WHERE id=? AND status IN ('queued','running')", status, outcome, diagnostic, string(data), id))
}
+50 -45
View File
@@ -55,48 +55,64 @@ run_id INTEGER PRIMARY KEY REFERENCES pipeline_runs(id), original_run_id INTEGER
CREATE TABLE IF NOT EXISTS retry_checkpoints (
run_id INTEGER PRIMARY KEY REFERENCES pipeline_runs(id), source_run_id INTEGER NOT NULL REFERENCES pipeline_runs(id),
mode TEXT NOT NULL, failed_sequence INTEGER NOT NULL, handoff_json TEXT NOT NULL,
source_version INTEGER NOT NULL, source_hash TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);
CREATE TABLE IF NOT EXISTS resource_owners (
resource_key TEXT PRIMARY KEY, run_id INTEGER NOT NULL REFERENCES pipeline_runs(id),
stage_key TEXT NOT NULL, sequence INTEGER NOT NULL, token TEXT NOT NULL, pool_name TEXT NOT NULL DEFAULT '',
cleanup_confirmed INTEGER NOT NULL DEFAULT 0, diagnostic TEXT NOT NULL DEFAULT '',
acquired_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);
CREATE INDEX IF NOT EXISTS resource_owner_run ON resource_owners(run_id);
CREATE UNIQUE INDEX IF NOT EXISTS resource_owner_visit ON resource_owners(run_id);
CREATE TABLE IF NOT EXISTS resource_waiters (
queue_order INTEGER PRIMARY KEY AUTOINCREMENT, resource_key TEXT NOT NULL,
run_id INTEGER NOT NULL UNIQUE REFERENCES pipeline_runs(id), stage_key TEXT NOT NULL,
sequence INTEGER NOT NULL, token TEXT NOT NULL, queued_at TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS resource_wait_fifo ON resource_waiters(resource_key,queue_order);`)
source_version INTEGER NOT NULL, source_hash TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);`)
if err != nil {
return err
}
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS resource_pools (
name TEXT PRIMARY KEY, members_json TEXT NOT NULL, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
);`); err != nil {
return err
}
if err := s.addColumnIfMissing("resource_waiters", "pool_name", "TEXT NOT NULL DEFAULT ''"); err != nil {
return err
}
if err := s.addColumnIfMissing("resource_owners", "pool_name", "TEXT NOT NULL DEFAULT ''"); err != nil {
return err
}
if err := s.migrateRunTitles(); err != nil {
return err
}
if err := s.addColumnIfMissing("pipeline_runs", "resource_wait", "TEXT NOT NULL DEFAULT ''"); err != nil {
return err
}
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS resource_recovery_audit (
resource_key TEXT NOT NULL, run_id INTEGER NOT NULL, stage_key TEXT NOT NULL,
sequence INTEGER NOT NULL, actor TEXT NOT NULL, source TEXT NOT NULL,
evidence TEXT NOT NULL, result TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP)`); err != nil {
if err := s.dropResourceSchema(); err != nil {
return err
}
return s.migrateRunModels()
}
// dropResourceSchema removes the retired resource pool, ownership, wait queue
// and recovery tables, plus the queue snapshot column. It is idempotent: a
// database that predates the feature or was already migrated opens cleanly,
// while an older database loses only the removed feature's state.
func (s *Store) dropResourceSchema() error {
for _, table := range []string{"resource_owners", "resource_waiters", "resource_pools", "resource_recovery_audit"} {
if _, err := s.db.Exec("DROP TABLE IF EXISTS " + table); err != nil {
return err
}
}
return s.dropColumnIfMissing("pipeline_runs", "resource_wait")
}
// dropColumnIfMissing removes a retired column only when it is still present, so
// repeated Open calls and databases created after the removal stay idempotent.
func (s *Store) dropColumnIfMissing(table, column string) error {
rows, err := s.db.Query("PRAGMA table_info(" + table + ")")
if err != nil {
return err
}
found := false
for rows.Next() {
var cid, notnull, pk int
var name, ctype string
var dflt sql.NullString
if err := rows.Scan(&cid, &name, &ctype, &notnull, &dflt, &pk); err != nil {
rows.Close()
return err
}
if name == column {
found = true
}
}
if err := rows.Err(); err != nil {
rows.Close()
return err
}
rows.Close()
if !found {
return nil
}
_, err = s.db.Exec("ALTER TABLE " + table + " DROP COLUMN " + column)
return err
}
// migrateRunModels adds the launch-time model-override columns to a database
// created before per-run overrides existed. Each column is added only when
// missing, so an upgrade is idempotent across repeated Open calls and existing
@@ -158,29 +174,18 @@ func (s *Store) addColumnIfMissing(table, column, definition string) error {
}
func (s *Store) RecoverRuns(ctx context.Context) error {
s.resourceMu.Lock()
defer s.resourceMu.Unlock()
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
// No replay: pending visits never began. Uncertain owners survive recovery,
// including the crash window before BeginStep/session registration.
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET diagnostic='Server stopped while queued for a resource; pending visit was not resumed',status='interrupted',finished_at=CURRENT_TIMESTAMP,resource_wait='' WHERE id IN (SELECT run_id FROM resource_waiters); DELETE FROM resource_waiters"); err != nil {
return err
}
if err := recoverResourceOwners(ctx, tx); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, "UPDATE run_steps SET status='interrupted',agent_active=CASE WHEN EXISTS(SELECT 1 FROM resource_owners o WHERE o.run_id=run_steps.run_id AND o.sequence=run_steps.sequence) THEN agent_active ELSE 0 END,diagnostic='Server stopped; stage was not resumed',finished_at=CURRENT_TIMESTAMP WHERE (status='running' OR agent_active=1) AND status<>?", protocol.StepStatusWaiting); err != nil {
// No replay: pending visits never began. A cleanup agent that was still
// active is marked interrupted, and its run is interrupted below.
if _, err := tx.ExecContext(ctx, "UPDATE run_steps SET status='interrupted',agent_active=0,diagnostic='Server stopped; stage was not resumed',finished_at=CURRENT_TIMESTAMP WHERE (status='running' OR agent_active=1) AND status<>?", protocol.StepStatusWaiting); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted',diagnostic='Server stopped; run was not resumed',finished_at=CURRENT_TIMESTAMP WHERE status IN ('queued','running') AND status<>?", protocol.RunStatusWaiting); err != nil {
return err
}
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET diagnostic='Server stopped; resource ownership quarantined: inspect retained session, tool termination and restoration before reconciliation' WHERE id IN (SELECT run_id FROM resource_owners)"); err != nil {
return err
}
return tx.Commit()
}
+7
View File
@@ -29,6 +29,13 @@ func (s *Store) RunSession(ctx context.Context, id int64, sequence int, session
return tx.Commit()
}
// ConfirmRecoveredSession clears the agent-active marker after startup has
// interrupted a previously active session, so it no longer blocks new work.
func (s *Store) ConfirmRecoveredSession(ctx context.Context, owner SessionOwner) error {
_, err := s.db.ExecContext(ctx, "UPDATE run_steps SET agent_active=0 WHERE run_id=? AND session_id=?", owner.RunID, owner.ID)
return err
}
type SessionOwner struct {
ID string
Backend string
+4 -6
View File
@@ -18,12 +18,10 @@ import (
)
type Store struct {
db *sql.DB
dir string
path string
mu sync.Mutex
resourceMu sync.Mutex
resourceWaiters map[ResourceVisit]chan ResourceLease
db *sql.DB
dir string
path string
mu sync.Mutex
}
// openDB opens one SQLite connection pool for a DSN. Every open path shares it
+15 -34
View File
@@ -3,31 +3,22 @@ package protocol
import "git.a-wan.net/alan/pipelines/internal/buildinfo"
const (
Ping = "ping"
Pong = "pong"
Error = "error"
Submit = "submit"
Submitted = "submitted"
List = "list_pipelines"
PipelineList = "pipelines"
GetPipeline = "get_pipeline"
Pipeline = "pipeline"
ListTemplates = "list_templates"
Templates = "templates"
Validate = "validate"
Validated = "validated"
InspectResource = "inspect_resource"
ResourceInspectionResult = "resource_inspection"
ReconcileResource = "reconcile_resource"
ResourceReconciled = "resource_reconciled"
InspectResourcePool = "inspect_resource_pool"
ResourcePoolInspection = "resource_pool_inspection"
ConfigureResourcePool = "configure_resource_pool"
ResourcePoolConfigured = "resource_pool_configured"
Ping = "ping"
Pong = "pong"
Error = "error"
Submit = "submit"
Submitted = "submitted"
List = "list_pipelines"
PipelineList = "pipelines"
GetPipeline = "get_pipeline"
Pipeline = "pipeline"
ListTemplates = "list_templates"
Templates = "templates"
Validate = "validate"
Validated = "validated"
)
type Stage struct {
Resources *StageResources `json:"resources,omitempty"`
Limits *StageLimits `json:"limits,omitempty"`
Memory map[string]string `json:"memory,omitempty"`
Notify *NotificationPolicy `json:"notify,omitempty"`
@@ -42,12 +33,6 @@ type Stage struct {
Blocks map[string]string `json:"blocks,omitempty"`
}
// StageResources declares one capacity-one resource, local to this store authority.
type StageResources struct {
Exclusive string `json:"exclusive"`
Pool string `json:"pool"`
}
// Limits are enforced by the runner and never included in the model prompt.
type StageLimits struct {
MaxVisits int `json:"max_visits"`
@@ -211,10 +196,6 @@ type Message struct {
// RunEvent is the server-pushed run state delta (see runevents.go) and
// RunIDs the optional subscribe scope, so a client asks for the runs it
// displays. Both are optional so older peers ignore them.
RunEvent *RunEventFrame `json:"run_event,omitempty"`
RunIDs []int64 `json:"run_ids,omitempty"`
Resource string `json:"resource,omitempty"`
ResourceInspection *ResourceInspection `json:"resource_inspection,omitempty"`
ResourceRecovery *ResourceRecoveryRequest `json:"resource_recovery,omitempty"`
ResourcePool *ResourcePool `json:"resource_pool,omitempty"`
RunEvent *RunEventFrame `json:"run_event,omitempty"`
RunIDs []int64 `json:"run_ids,omitempty"`
}
-66
View File
@@ -1,66 +0,0 @@
package protocol
type ResourceInspection struct {
Resource string `json:"resource"`
Owner *ResourceOwner `json:"owner,omitempty"`
Waiters []ResourceWaiter `json:"waiters,omitempty"`
Audits []ResourceRecoveryAudit `json:"audits,omitempty"`
Pool *ResourcePool `json:"pool,omitempty"`
}
// ResourceTarget is bounded, non-secret metadata for the concrete member that
// was selected. Credentials and capabilities are intentionally absent.
type ResourceTarget struct {
Member string `json:"member"`
Metadata map[string]string `json:"metadata,omitempty"`
}
type ResourcePoolMember struct {
Member string `json:"member"`
Metadata map[string]string `json:"metadata,omitempty"`
Enabled bool `json:"enabled"`
Quarantined bool `json:"quarantined,omitempty"`
}
type ResourcePool struct {
Name string `json:"name"`
Members []ResourcePoolMember `json:"members"`
}
type ResourceOwner struct {
RunID int64 `json:"run_id"`
Stage string `json:"stage"`
Sequence int `json:"sequence"`
CleanupConfirmed bool `json:"cleanup_confirmed"`
Quarantined bool `json:"quarantined"`
AgentActive bool `json:"agent_active"`
}
type ResourceWaiter struct {
RunID int64 `json:"run_id"`
Stage string `json:"stage"`
Sequence int `json:"sequence"`
QueuedAt string `json:"queued_at"`
}
type ResourceRecoveryAudit struct {
Resource string `json:"resource"`
RunID int64 `json:"run_id"`
Stage string `json:"stage"`
Sequence int `json:"sequence"`
Actor string `json:"actor"`
Source string `json:"source"`
Evidence string `json:"evidence"`
Result string `json:"result"`
CreatedAt string `json:"created_at"`
}
type ResourceRecoveryEvidence struct {
CleanupConfirmed bool `json:"cleanup_confirmed"`
NoMutatingAction bool `json:"no_mutating_action"`
Summary string `json:"summary"`
}
type ResourceRecoveryRequest struct {
Resource string `json:"resource"`
RunID int64 `json:"run_id"`
Stage string `json:"stage"`
Sequence int `json:"sequence"`
Evidence *ResourceRecoveryEvidence `json:"evidence,omitempty"`
Actor string `json:"actor"`
}
-29
View File
@@ -1,29 +0,0 @@
package protocol
import (
"encoding/json"
"strings"
"testing"
)
func TestResourceWaitOlderClientCompatibility(t *testing.T) {
data, err := json.Marshal(RunSummary{ID: 1, Status: RunStatusQueued, ResourceWait: &ResourceWait{Resource: "X", Stage: "review", Sequence: 2, QueuedSince: "now"}})
if err != nil {
t.Fatal(err)
}
var old struct {
ID int64 `json:"id"`
Status string `json:"status"`
}
if err := json.Unmarshal(data, &old); err != nil || old.Status != "queued" {
t.Fatal("old client no longer waits", old, err)
}
clear, err := json.Marshal(RunEventFrame{RunID: 1, ResourceWaitSet: true})
if err != nil || !strings.Contains(string(clear), `"resource_wait_set":true`) {
t.Fatal("no explicit clear", string(clear), err)
}
var current RunSummary
if err := json.Unmarshal([]byte(`{"id":2,"status":"running"}`), &current); err != nil || current.ResourceWait != nil {
t.Fatal("missing field changed old payload", err)
}
}
+2 -5
View File
@@ -24,11 +24,8 @@ const (
// from the frame instead of re-reading the whole list. They repeat on every
// frame (pipeline identity never changes mid-run).
type RunEventFrame struct {
// ResourceWaitSet makes a nil ResourceWait an explicit cache clear.
ResourceWaitSet bool `json:"resource_wait_set,omitempty"`
ResourceWait *ResourceWait `json:"resource_wait,omitempty"`
RunID int64 `json:"run_id"`
Workspace string `json:"workspace,omitempty"`
RunID int64 `json:"run_id"`
Workspace string `json:"workspace,omitempty"`
// PipelineID, Name, Title and Version identify the run's pipeline and its
// operator-facing title, so a new run's row can be built without a fetch.
PipelineID int64 `json:"pipeline_id,omitempty"`
+8 -31
View File
@@ -118,10 +118,9 @@ type SessionExport struct {
}
type RunSummary struct {
ResourceWait *ResourceWait `json:"resource_wait,omitempty"`
ID int64 `json:"id"`
PipelineID int64 `json:"pipeline_id"`
Name string `json:"name"`
ID int64 `json:"id"`
PipelineID int64 `json:"pipeline_id"`
Name string `json:"name"`
// Title is the operator-facing, bounded single-line description of what the
// run is for, generated from its prompt. It is never empty for a new run;
// historical rows predating the title column may omit it.
@@ -146,19 +145,6 @@ type RunSummary struct {
StageModels map[string]string `json:"stage_models,omitempty"`
}
// ResourceWait is bounded public queue metadata. Ownership credentials stay private.
type ResourceWait struct {
Resource string `json:"resource"`
Stage string `json:"stage"`
Sequence int `json:"sequence"`
QueuedSince string `json:"queued_since"`
}
// ResourceCleanupConfirmedKey lets a stage report failed restoration without
// pretending that tool idleness also proves the resource is ready. An explicit
// false (or malformed value) vetoes release; it never authorizes release alone.
const ResourceCleanupConfirmedKey = "resource_cleanup_confirmed"
// Label is the operator-facing name of a run: the generated Title when the
// server stored one, otherwise the pipeline Name (historical rows predate
// titles). It is the single definition shared by the CLI and TUI.
@@ -169,14 +155,6 @@ func (r RunSummary) Label() string {
return r.Name
}
// StatusLabel separates resource contention from a human decision everywhere.
func (r RunSummary) StatusLabel() string {
if r.ResourceWait != nil {
return "Waiting for resource " + r.ResourceWait.Resource
}
return r.Status
}
type RunStep struct {
Control *ControlEvidence `json:"control,omitempty"`
FileChanges *StageFileChanges `json:"file_changes,omitempty"`
@@ -331,12 +309,11 @@ type MemoryRead struct {
}
type ControlEvidence struct {
Visit int `json:"visit"`
MaxVisits int `json:"max_visits"`
LimitReached bool `json:"limit_reached,omitempty"`
ModelTarget string `json:"model_target,omitempty"`
Memory map[string]MemoryRead `json:"memory,omitempty"`
ResourceTarget *ResourceTarget `json:"resource_target,omitempty"`
Visit int `json:"visit"`
MaxVisits int `json:"max_visits"`
LimitReached bool `json:"limit_reached,omitempty"`
ModelTarget string `json:"model_target,omitempty"`
Memory map[string]MemoryRead `json:"memory,omitempty"`
}
// StageFileChanges is observed workspace evidence, not a model's claim of authorship.
-511
View File
@@ -1,511 +0,0 @@
package server
import (
"bytes"
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"net"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/cli"
"git.a-wan.net/alan/pipelines/internal/client"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
"github.com/coder/websocket"
"github.com/coder/websocket/wsjson"
)
type fixedRemoteListener struct{ net.Listener }
func (l fixedRemoteListener) Accept() (net.Conn, error) {
conn, err := l.Listener.Accept()
if err != nil {
return nil, err
}
return fixedRemoteConn{Conn: conn}, nil
}
type fixedRemoteConn struct{ net.Conn }
func (fixedRemoteConn) RemoteAddr() net.Addr {
return &net.TCPAddr{IP: net.ParseIP("203.0.113.9"), Port: 443}
}
// This is a real CLI/WebSocket/scratch-SQLite exercise with an explicitly FAKE
// executor. It never calls inference, a phone, or a deployed server.
func TestResourceScratchCLIWaitingThenProgress(t *testing.T) {
t.Setenv("PIPELINES_HARNESS_CONFIG", filepath.Join(t.TempDir(), "harnesses.json"))
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "scratch.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
source := pipeline.Template([]string{"work"}, true) + "```resources\n{\"exclusive\":\"fake-phone\"}\n```\n"
if _, err := store.Submit(ctx, "scratch-resource", source); err != nil {
t.Fatal(err)
}
handler := Handler(ctx, store).(*runHandler)
started, release := make(chan struct{}), make(chan struct{})
handler.runner.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
if input["prompt"] == "A" {
close(started)
select {
case <-release:
case <-ctx.Done():
return stage.Result{}, ctx.Err()
}
}
return stage.Result{Outcome: "success", Output: stage.Values{"execution": "FAKE executor; no device or inference"}}, nil
}
handler.runner.events.interval = time.Millisecond
events := make(chan protocol.RunEventFrame, 64)
sub := handler.runner.events.subscribe("cli-evidence", "", nil, func(m protocol.Message) error {
if m.RunEvent != nil {
events <- *m.RunEvent
}
return nil
})
sub.ready()
defer sub.close()
srv := httptest.NewServer(handler)
defer srv.Close()
defer func() { cancel(); WaitRuns(handler) }()
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
run := func(args ...string) string {
var out bytes.Buffer
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
if err := app.Run(ctx, append([]string{"--url", endpoint}, args...)); err != nil {
t.Fatalf("CLI %v: %v\n%s", args, err, out.String())
}
t.Logf("pipelines %s\n%s", strings.Join(args, " "), out.String())
return out.String()
}
parseRun := func(output string) int64 {
var m protocol.Message
if err := json.Unmarshal([]byte(output), &m); err != nil || m.RunDetail == nil {
t.Fatal("CLI run response", output, err)
}
return m.RunDetail.ID
}
run("validate", "testdata/android-resource-draft.md", "--json")
a := parseRun(run("run", "scratch-resource", "A", "--workspace", t.TempDir(), "--yes", "--json", "--request-key", "a"))
<-started
b := parseRun(run("run", "scratch-resource", "B", "--workspace", t.TempDir(), "--yes", "--json", "--request-key", "b"))
awaitResourceEvent(t, events, b, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
var detail protocol.RunDetail
if err := json.Unmarshal([]byte(run("inspect", fmt.Sprint(b), "--json")), &detail); err != nil || detail.ResourceWait == nil || detail.Status != "queued" || len(detail.Steps) != 0 {
t.Fatal("CLI did not observe queue", err)
}
if output := run("runs"); !strings.Contains(output, "Waiting for resource fake-phone") {
t.Fatal("list omitted resource status", output)
}
var snapshot *protocol.RunEventFrame
handleRunEvents(protocol.Message{Type: protocol.Subscribe, RunIDs: []int64{b}}, handler.runner.events, handler.runner, "snapshot-evidence", func(m protocol.Message) error {
if m.RunEvent != nil {
snapshot = m.RunEvent
}
return nil
})
handler.runner.events.unsubscribeClient("snapshot-evidence")
if snapshot == nil || snapshot.ResourceWait == nil || !snapshot.ResourceWaitSet {
t.Fatal("subscription snapshot omitted resource wait")
}
close(release)
run("wait", fmt.Sprint(a), fmt.Sprint(b), "--json")
detail = protocol.RunDetail{}
if err := json.Unmarshal([]byte(run("inspect", fmt.Sprint(b), "--json")), &detail); err != nil || detail.Status != "succeeded" || detail.ResourceWait != nil || len(detail.Steps) != 1 {
t.Fatal("automatic progress not observed", err)
}
t.Log("Observed queued resource wait followed by automatic progress; FAKE executor only, no phone/inference/deployment.")
}
func TestResourceOperatorTransportCLIRecoveryBoundary(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "operator.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
source := pipeline.Template([]string{"work"}, true) + "```resources\n{\"exclusive\":\"scratch-resource\"}\n```\n"
if _, err := store.Submit(ctx, "operator-resource", source); err != nil {
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
t.Fatal(err)
}
detail, err := store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
runID, _, err := store.CreateRun(ctx, detail, "operator-recovery", t.TempDir(), false, nil, nil)
if err != nil {
t.Fatal(err)
}
visit := pipeline.ResourceVisit{Resource: "scratch-resource", RunID: runID, Stage: "work", Sequence: 1}
if _, ch, _, err := store.RequestResource(ctx, visit); err != nil || ch != nil {
t.Fatalf("acquire resource: channel=%v err=%v", ch, err)
}
if err := store.RecoverRuns(ctx); err != nil {
t.Fatal(err)
}
handler := Handler(ctx, store)
srv := httptest.NewServer(handler)
defer srv.Close()
normal, err := client.Open(ctx, "ws"+strings.TrimPrefix(srv.URL, "http")+"/connect")
if err != nil {
t.Fatal(err)
}
if _, err := normal.Call(ctx, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource}); err == nil || !strings.Contains(err.Error(), "operator transport") {
t.Fatalf("ordinary connect accepted recovery: %v", err)
}
normal.Close()
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
runCLI := func(args ...string) string {
var out bytes.Buffer
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
if err := app.Run(ctx, append([]string{"--url", endpoint}, args...)); err != nil {
t.Fatalf("CLI %v: %v\n%s", args, err, out.String())
}
return out.String()
}
inspection := runCLI("resource-inspect", visit.Resource, "--json")
if strings.Contains(inspection, "token") || strings.Contains(inspection, "credential") {
t.Fatalf("inspection leaked capability: %s", inspection)
}
runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), visit.Stage, fmt.Sprint(visit.Sequence), "--evidence", "verified no mutating action", "--yes", "--json")
inspection = runCLI("resource-inspect", visit.Resource, "--json")
var recovered protocol.ResourceInspection
if err := json.Unmarshal([]byte(inspection), &recovered); err != nil {
t.Fatal(err)
}
if recovered.Owner != nil || len(recovered.Audits) != 1 || recovered.Audits[0].RunID != runID || recovered.Audits[0].Result != "reconciled" {
t.Fatalf("CLI recovery state was not persisted: %s", inspection)
}
}
func TestResourceOperatorCLIRejectsStaleAndNonQuarantinedRecovery(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "operator-rejections.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "operator-rejections", pipeline.Template([]string{"work"}, true)+"```resources\n{\"exclusive\":\"scratch-resource\"}\n```\n"); err != nil {
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
t.Fatal(err)
}
detail, err := store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
runID, _, err := store.CreateRun(ctx, detail, "operator-rejections", t.TempDir(), false, nil, nil)
if err != nil {
t.Fatal(err)
}
visit := pipeline.ResourceVisit{Resource: "scratch-resource", RunID: runID, Stage: "work", Sequence: 1}
lease, ch, _, err := store.RequestResource(ctx, visit)
if err != nil || ch != nil {
t.Fatalf("acquire resource: channel=%v err=%v", ch, err)
}
if err := store.RecoverRuns(ctx); err != nil {
t.Fatal(err)
}
handler := Handler(ctx, store)
srv := httptest.NewServer(handler)
defer srv.Close()
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
runCLI := func(args ...string) error {
var out bytes.Buffer
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
err := app.Run(ctx, append([]string{"--url", endpoint}, args...))
if strings.Contains(out.String(), "token") || strings.Contains(out.String(), "credential") {
t.Fatalf("private capability appeared in CLI output: %s", out.String())
}
return err
}
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID+1), visit.Stage, fmt.Sprint(visit.Sequence), "--evidence", "verified", "--yes", "--json"); err == nil {
t.Fatal("accepted wrong owner")
}
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), "stale-stage", fmt.Sprint(visit.Sequence), "--evidence", "verified", "--yes", "--json"); err == nil {
t.Fatal("accepted stale stage")
}
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), visit.Stage, "2", "--evidence", "verified", "--yes", "--json"); err == nil {
t.Fatal("accepted stale sequence")
}
if err := store.SetResourceCleanup(ctx, lease, true); err != nil {
t.Fatal(err)
}
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), visit.Stage, fmt.Sprint(visit.Sequence), "--evidence", "verified", "--yes", "--json"); err == nil {
t.Fatal("accepted non-quarantined owner")
}
inspection, err := store.InspectResource(ctx, visit.Resource)
if err != nil {
t.Fatal(err)
}
if inspection.Owner == nil || !inspection.Owner.CleanupConfirmed || len(inspection.Audits) != 0 {
t.Fatalf("rejected recovery changed state: owner=%#v audits=%d", inspection.Owner, len(inspection.Audits))
}
}
func TestResourceOperatorTransportDoesNotTrustRemoteOrForwardedHeaders(t *testing.T) {
for _, remote := range []string{"203.0.113.9:443", "198.51.100.7:8080", "not-an-address"} {
if isLoopbackPeer(remote) {
t.Fatalf("accepted non-loopback peer %q", remote)
}
}
for _, remote := range []string{"127.0.0.1:443", "[::1]:443"} {
if !isLoopbackPeer(remote) {
t.Fatalf("rejected loopback peer %q", remote)
}
}
// X-Forwarded-For is deliberately not consulted: a proxy-supplied header
// cannot turn a remote WebSocket peer into the local operator transport.
}
func TestResourceOperatorTransportRejectsForgedForwardedPeer(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "operator-transport.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "operator-transport", pipeline.Template([]string{"work"}, true)+"```resources\n{\"exclusive\":\"scratch-resource\"}\n```\n"); err != nil {
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
t.Fatal(err)
}
detail, err := store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
runID, _, err := store.CreateRun(ctx, detail, "operator-transport", t.TempDir(), false, nil, nil)
if err != nil {
t.Fatal(err)
}
visit := pipeline.ResourceVisit{Resource: "scratch-resource", RunID: runID, Stage: "work", Sequence: 1}
if _, ch, _, err := store.RequestResource(ctx, visit); err != nil || ch != nil {
t.Fatalf("acquire resource: channel=%v err=%v", ch, err)
}
if err := store.RecoverRuns(ctx); err != nil {
t.Fatal(err)
}
before, err := store.InspectResource(ctx, visit.Resource)
if err != nil {
t.Fatal(err)
}
server := httptest.NewUnstartedServer(Handler(ctx, store))
server.Listener = fixedRemoteListener{Listener: server.Listener}
server.Start()
defer server.Close()
conn, _, err := websocket.Dial(ctx, "ws"+strings.TrimPrefix(server.URL, "http")+"/operator-connect", &websocket.DialOptions{HTTPHeader: map[string][]string{"X-Forwarded-For": {"127.0.0.1"}}})
if err != nil {
t.Fatal(err)
}
defer conn.CloseNow()
var welcome protocol.Message
if err := wsjson.Read(ctx, conn, &welcome); err != nil {
t.Fatal(err)
}
if err := wsjson.Write(ctx, conn, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource, ID: "forged"}); err != nil {
t.Fatal(err)
}
var refused protocol.Message
if err := wsjson.Read(ctx, conn, &refused); err != nil {
t.Fatal(err)
}
if refused.Type != protocol.Error || !strings.Contains(refused.Error, "operator transport") {
t.Fatalf("forged peer was accepted: %#v", refused)
}
after, err := store.InspectResource(ctx, visit.Resource)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(before, after) {
t.Fatalf("forged request mutated recovery state: before=%#v after=%#v", before, after)
}
trusted := httptest.NewServer(Handler(ctx, store))
defer trusted.Close()
normal, err := client.Open(ctx, "ws"+strings.TrimPrefix(trusted.URL, "http")+"/connect")
if err != nil {
t.Fatal(err)
}
if _, err := normal.Call(ctx, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource}); err == nil || !strings.Contains(err.Error(), "operator transport") {
t.Fatalf("ordinary connect accepted recovery: %v", err)
}
normal.Close()
operator, err := client.Open(ctx, "ws"+strings.TrimPrefix(trusted.URL, "http")+"/operator-connect")
if err != nil {
t.Fatal(err)
}
defer operator.Close()
if _, err := operator.Call(ctx, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource}); err != nil {
t.Fatalf("trusted operator transport refused inspection: %v", err)
}
}
func TestResourcePoolOperatorCLIConfigureReadbackAndPreserveLease(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
dbPath := filepath.Join(t.TempDir(), "pool-operator.db")
store, err := pipeline.Open(dbPath)
if err != nil {
t.Fatal(err)
}
defer store.Close()
if err := store.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{{Member: "phone", Enabled: true, Metadata: map[string]string{"serial": "old"}}}}); err != nil {
t.Fatal(err)
}
if _, err := store.Submit(ctx, "pool-owner", pipeline.Template([]string{"work"}, true)+"```resources\n{\"pool\":\"targets\"}\n```\n"); err != nil {
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil || len(items) != 1 {
t.Fatal("pool pipeline missing", err)
}
detail, err := store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
runID, _, err := store.CreateRun(ctx, detail, "pool-owner", t.TempDir(), false, nil, nil)
if err != nil {
t.Fatal(err)
}
lease, ready, _, err := store.RequestResourcePool(ctx, "targets", pipeline.ResourceVisit{Resource: "targets", RunID: runID, Stage: "work", Sequence: 1})
if err != nil || ready != nil {
t.Fatalf("pool lease: ready=%v err=%v", ready, err)
}
handler := Handler(ctx, store)
srv := httptest.NewServer(handler)
defer srv.Close()
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
normal, err := client.Open(ctx, endpoint)
if err != nil {
t.Fatal(err)
}
if _, err := normal.Call(ctx, protocol.Message{Type: protocol.ConfigureResourcePool, ResourcePool: &protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{{Member: "phone", Enabled: true}}}}); err == nil || !strings.Contains(err.Error(), "operator transport") {
t.Fatalf("remote configuration was accepted: %v", err)
}
normal.Close()
declaration := filepath.Join(t.TempDir(), "pool.json")
if err := os.WriteFile(declaration, []byte(`{"name":"targets","members":[{"member":"phone","enabled":true,"metadata":{"serial":"new"}},{"member":"emulator","enabled":false}]}`), 0600); err != nil {
t.Fatal(err)
}
var out bytes.Buffer
for _, fromStdin := range []bool{false, true} {
oversized := strings.Repeat(" ", 256*1024+1)
path := filepath.Join(t.TempDir(), "oversized.json")
if err := os.WriteFile(path, []byte(oversized), 0600); err != nil {
t.Fatal(err)
}
if fromStdin {
path = "-"
}
bounded := cli.App{Out: &out, Err: &out, In: strings.NewReader(oversized)}
err := bounded.Run(ctx, []string{"--url", endpoint, "resource-pool-configure", path, "--yes", "--json"})
if err == nil || !strings.Contains(err.Error(), "exceeds 262144 bytes") {
t.Fatalf("oversized pool input (stdin=%v): %v", fromStdin, err)
}
out.Reset()
}
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
if err := app.Run(ctx, []string{"--url", "ws" + strings.TrimPrefix(srv.URL, "http"), "resource-pool-configure", declaration, "--yes", "--json"}); err != nil {
t.Fatalf("configure CLI: %v\n%s", err, out.String())
}
out.Reset()
app = cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
if err := app.Run(ctx, []string{"--url", "ws" + strings.TrimPrefix(srv.URL, "http"), "resource-pool-inspect", "targets", "--json"}); err != nil {
t.Fatalf("inspect CLI: %v\n%s", err, out.String())
}
var pool protocol.ResourcePool
if err := json.Unmarshal(out.Bytes(), &pool); err != nil || len(pool.Members) != 2 || pool.Members[0].Metadata["serial"] != "new" {
t.Fatalf("bad pool readback: %s (%v)", out.String(), err)
}
inspection, err := store.InspectResource(ctx, "phone")
if err != nil || inspection.Owner == nil || inspection.Owner.RunID != runID {
t.Fatalf("configuration released active lease: %#v (%v)", inspection, err)
}
if err := store.SetResourceCleanup(ctx, lease, true); err != nil {
t.Fatal(err)
}
if err := store.ReleaseResource(ctx, lease); err != nil {
t.Fatal(err)
}
srv.Close()
store.Close()
reopened, err := pipeline.Open(dbPath)
if err != nil {
t.Fatal(err)
}
defer reopened.Close()
persisted, err := reopened.InspectResourcePool(ctx, "targets")
if err != nil || persisted.Members[0].Metadata["serial"] != "new" {
t.Fatalf("pool did not survive reopen: %#v (%v)", persisted, err)
}
}
func TestResourceAndroidDraftPublishedV3Provenance(t *testing.T) {
baseline, err := os.ReadFile("testdata/android-v3-published.md")
if err != nil {
t.Fatal(err)
}
if hash := fmt.Sprintf("%x", sha256.Sum256(baseline)); hash != "4a3a4b8b947eb20c115c5470e6e378f117d5fa291a7b6c8ae966a3745bc826de" {
t.Fatal("baseline is not retrieved Android v3", hash)
}
draft, err := os.ReadFile("testdata/android-resource-draft.md")
if err != nil {
t.Fatal(err)
}
old, err := pipeline.Parse(string(baseline))
if err != nil {
t.Fatal(err)
}
updated, err := pipeline.Parse(string(draft))
if err != nil {
t.Fatal(err)
}
if len(old.Stages) != len(updated.Stages) {
t.Fatal("draft changed graph shape")
}
for i := range old.Stages {
a, b := old.Stages[i], updated.Stages[i]
if b.Configs["provider"] != "openai" || b.Configs["model"] != a.Configs["model"] {
t.Fatal("draft changed approved model")
}
if b.Key == "ui_reviewer" {
if b.Resources == nil || b.Resources.Exclusive != "phone-47241FDAS00DLK" {
t.Fatal("reviewer resource missing")
}
} else if b.Resources != nil {
t.Fatal("non-UI stage acquired phone")
}
a.Prompt, b.Prompt = "", ""
a.Resources, b.Resources = nil, nil
if !reflect.DeepEqual(a, b) {
t.Fatal("draft changed fields outside approved role split/resource", a.Key)
}
}
}
-72
View File
@@ -1,72 +0,0 @@
package server
import (
"context"
"errors"
"time"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
)
func (r *runner) acquireResource(ctx context.Context, visit pipeline.ResourceVisit, workspace string, completed int) (pipeline.ResourceLease, error) {
return r.acquireResourceWithPool(ctx, visit, "", workspace, completed)
}
func (r *runner) acquireResourceWithPool(ctx context.Context, visit pipeline.ResourceVisit, pool string, workspace string, completed int) (pipeline.ResourceLease, error) {
var lease pipeline.ResourceLease
var ready <-chan pipeline.ResourceLease
var wait *protocol.ResourceWait
var err error
if pool != "" {
lease, ready, wait, err = r.store.RequestResourcePool(ctx, pool, visit)
} else {
lease, ready, wait, err = r.store.RequestResource(ctx, visit)
}
if err != nil {
return lease, err
}
if ready != nil {
r.emitRun(protocol.RunEventFrame{RunID: visit.RunID, Workspace: workspace, Status: protocol.RunStatusQueued, CurrentStage: visit.Stage, CompletedSteps: completed, ResourceWaitSet: true, ResourceWait: wait}, false)
select {
case lease = <-ready:
r.store.AcceptResource(lease)
case <-ctx.Done():
cleanup, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := r.store.CancelResource(cleanup, visit); err != nil {
return lease, errors.Join(ctx.Err(), err)
}
return lease, ctx.Err()
}
}
if ctx.Err() != nil {
err := r.releaseResource(lease, true) // Never launched, including a concurrent grant.
return pipeline.ResourceLease{}, errors.Join(ctx.Err(), err)
}
r.emitRun(protocol.RunEventFrame{RunID: visit.RunID, Workspace: workspace, ResourceWaitSet: true}, false)
return lease, nil
}
func (r *runner) releaseResource(lease pipeline.ResourceLease, confirmed bool) error {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := r.store.SetResourceCleanup(ctx, lease, confirmed); err != nil {
return err
}
if confirmed {
return r.store.ReleaseResource(ctx, lease)
}
return nil
}
func (r *runner) resourceQuiescent(id int64, executionErr error, panicked bool) bool {
if panicked || errors.Is(executionErr, stage.ErrStageCleanup) {
return false
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
sessions, err := r.store.ActiveAgentSessionsForRun(ctx, id)
return err == nil && len(sessions) == 0
}
-418
View File
@@ -1,418 +0,0 @@
package server
import (
"context"
"database/sql"
"errors"
"fmt"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
)
func resourceRunner(t *testing.T) (*runner, <-chan protocol.RunEventFrame, func(string, string, string) *protocol.RunDetail) {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
s, err := pipeline.Open(filepath.Join(t.TempDir(), "resources.db"))
if err != nil {
t.Fatal(err)
}
r, err := newRunner(ctx, s)
if err != nil {
t.Fatal(err)
}
r.events = newRunEventHub()
r.events.interval = time.Millisecond
events := make(chan protocol.RunEventFrame, 128)
sub := r.events.subscribe("resource-test", "", nil, func(m protocol.Message) error {
if m.RunEvent != nil {
events <- *m.RunEvent
}
return nil
})
sub.ready()
t.Cleanup(func() { cancel(); r.wg.Wait(); sub.close(); s.Close() })
count := 0
return r, events, func(resource, timeout, prompt string) *protocol.RunDetail {
count++
source := strings.Replace(pipeline.Template([]string{"work"}, true), "timeout=1h", "timeout="+timeout, 1)
if resource != "" {
source += "```resources\n{\"exclusive\":\"" + resource + "\"}\n```\n"
}
name := fmt.Sprint("resource-", count)
if _, err := s.Submit(ctx, name, source); err != nil {
t.Fatal(err)
}
items, _ := s.List(ctx)
var pid int64
for _, p := range items {
if p.Name == name {
pid = p.ID
}
}
d, err := r.start(protocol.Message{PipelineID: pid, Version: 1, RequestKey: name, Workspace: t.TempDir(), Input: map[string]any{"prompt": prompt}})
if err != nil {
t.Fatal(err)
}
return d
}
}
func awaitResourceEvent(t *testing.T, events <-chan protocol.RunEventFrame, id int64, predicate func(protocol.RunEventFrame) bool) protocol.RunEventFrame {
t.Helper()
timer := time.NewTimer(3 * time.Second)
defer timer.Stop()
for {
select {
case event := <-events:
if event.RunID == id && predicate(event) {
return event
}
case <-timer.C:
t.Fatalf("resource event missing for run %d", id)
return protocol.RunEventFrame{}
}
}
}
func TestResourcePauseBeyondTimeoutAndAutomaticResume(t *testing.T) {
r, events, start := resourceRunner(t)
aStarted, releaseA := make(chan struct{}), make(chan struct{})
bStarted := make(chan struct{}, 1)
var bCalls atomic.Int32
r.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
switch input["prompt"] {
case "A":
close(aStarted)
select {
case <-releaseA:
case <-ctx.Done():
return stage.Result{}, ctx.Err()
}
case "B":
bCalls.Add(1)
deadline, ok := ctx.Deadline()
if !ok || time.Until(deadline) < 15*time.Millisecond {
return stage.Result{}, errors.New("execution timeout was consumed in queue")
}
bStarted <- struct{}{}
}
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
a := start("X", "2s", "A")
<-aStarted
b := start("X", "30ms", "B")
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
<-time.After(70 * time.Millisecond) // Deliberately longer than B's whole execution timeout.
d, err := r.store.GetRun(context.Background(), b.ID)
if err != nil || d.Status != protocol.RunStatusQueued || d.ResourceWait == nil || len(d.Steps) != 0 || bCalls.Load() != 0 {
t.Fatal("queued execution consumed accounting", d, err, bCalls.Load())
}
if _, err := r.decide(protocol.Message{RunID: b.ID, Option: "success"}); err == nil {
t.Fatal("resource wait exposed a human decision")
}
c := start("Y", "1s", "C")
awaitResourceEvent(t, events, c.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
select {
case <-bStarted:
t.Fatal("B ran while A held X")
default:
}
owner, err := r.store.GetRun(context.Background(), a.ID)
if err != nil || owner.Status != "running" {
t.Fatal("A did not remain owner", err)
}
close(releaseA)
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
r.wg.Wait()
d, err = r.store.GetRun(context.Background(), b.ID)
if err != nil || bCalls.Load() != 1 || d.ResourceWait != nil || len(d.Steps) != 1 || d.Steps[0].Control.Visit != 1 || d.Steps[0].Sequence != 1 {
t.Fatal("resume did not execute exactly one fresh visit", d, err)
}
}
func TestResourcePoolRunnerHandoffUsesConcreteTargetAfterQueue(t *testing.T) {
r, events, _ := resourceRunner(t)
ctx := context.Background()
source := pipeline.Template([]string{"work"}, true) + "```resources\n{\"pool\":\"targets\"}\n```\n"
if _, err := r.store.Submit(ctx, "pool-runner", source); err != nil {
t.Fatal(err)
}
items, err := r.store.List(ctx)
if err != nil || len(items) != 1 {
t.Fatal("pool definition missing", err)
}
definition, err := r.store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
if err := r.store.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
{Member: "phone", Enabled: true, Metadata: map[string]string{"serial": "phone-serial", "type": "phone"}},
{Member: "emulator", Enabled: true, Metadata: map[string]string{"serial": "emu-serial", "type": "emulator"}},
}}); err != nil {
t.Fatal(err)
}
ownerRun, _, err := r.store.CreateRun(ctx, definition, "phone-owner", t.TempDir(), false, nil, nil)
if err != nil {
t.Fatal(err)
}
phone, ch, _, err := r.store.RequestResource(ctx, pipeline.ResourceVisit{Resource: "phone", RunID: ownerRun, Stage: "work", Sequence: 1})
if err != nil || ch != nil {
t.Fatalf("phone owner acquire: channel=%v err=%v", ch, err)
}
defer r.releaseResource(phone, true)
aStarted, releaseA := make(chan struct{}), make(chan struct{})
bRan := make(chan *protocol.ResourceTarget, 1)
r.execute = func(stageCtx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
target, ok := stage.ResourceTarget(stageCtx)
if !ok || target == nil {
return stage.Result{}, errors.New("missing resource target context")
}
if input["prompt"] == "A" {
if target.Member != "emulator" || target.Metadata["serial"] != "emu-serial" || target.Metadata["type"] != "emulator" {
return stage.Result{}, fmt.Errorf("wrong first target: %#v", target)
}
close(aStarted)
<-releaseA
} else {
if deadline, ok := stageCtx.Deadline(); !ok || time.Until(deadline) < 15*time.Millisecond {
return stage.Result{}, errors.New("execution deadline was consumed in pool queue")
}
bRan <- target
}
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
start := func(key, prompt string) *protocol.RunDetail {
run, err := r.start(protocol.Message{PipelineID: definition.ID, Version: 1, RequestKey: key, Workspace: t.TempDir(), Input: map[string]any{"prompt": prompt}})
if err != nil {
t.Fatal(err)
}
return run
}
a := start("pool-a", "A")
<-aStarted
b := start("pool-b", "B")
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
time.Sleep(80 * time.Millisecond)
select {
case <-bRan:
t.Fatal("pool waiter ran before either member was released")
default:
}
if err := r.releaseResource(phone, true); err != nil {
t.Fatal(err)
}
var target *protocol.ResourceTarget
select {
case target = <-bRan:
case <-time.After(3 * time.Second):
t.Fatal("pool waiter did not resume")
}
if target.Member != "phone" || target.Metadata["serial"] != "phone-serial" {
t.Fatalf("wrong queued target: %#v", target)
}
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
close(releaseA)
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
r.wg.Wait()
detail, err := r.store.GetRun(ctx, b.ID)
if err != nil || len(detail.Steps) != 1 || detail.Steps[0].Control.ResourceTarget == nil {
t.Fatalf("pool target evidence missing: detail=%#v err=%v", detail, err)
}
}
func TestResourceCancellationWaitsForOwnerCleanup(t *testing.T) {
r, events, start := resourceRunner(t)
started, cleaning, cleanupDone := make(chan struct{}), make(chan struct{}), make(chan struct{})
var waitingCalls atomic.Int32
r.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
if input["prompt"] == "owner" {
close(started)
<-ctx.Done()
close(cleaning)
<-cleanupDone
return stage.Result{}, ctx.Err()
}
waitingCalls.Add(1)
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
a := start("X", "2s", "owner")
<-started
b := start("X", "30ms", "waiter")
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
if err := r.cancel(a.ID); err != nil {
t.Fatal(err)
}
<-cleaning
select {
case <-time.After(50 * time.Millisecond):
if waitingCalls.Load() != 0 {
t.Fatal("handoff before cleanup")
}
}
close(cleanupDone)
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
if waitingCalls.Load() != 1 {
t.Fatal("waiter did not resume exactly once")
}
}
func TestResourceQuarantineAndQueuedCancellation(t *testing.T) {
for _, failure := range []string{"cleanup", "panic", "restoration"} {
t.Run(failure, func(t *testing.T) {
r, events, start := resourceRunner(t)
started, release := make(chan struct{}), make(chan struct{})
var calls atomic.Int32
r.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
if input["prompt"] == "owner" {
close(started)
select {
case <-release:
case <-ctx.Done():
}
if failure == "panic" {
panic("unknown cleanup")
}
if failure == "restoration" {
return stage.Result{Outcome: "success", Output: stage.Values{protocol.ResourceCleanupConfirmedKey: false}}, nil
}
return stage.Result{}, stage.ErrStageCleanup
}
calls.Add(1)
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
a := start("X", "1s", "owner")
<-started
b := start("X", "30ms", "waiter")
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
close(release)
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "failed" })
if err := r.cancel(b.ID); err != nil {
t.Fatal(err)
}
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool {
return e.Status == "canceled" && e.ResourceWaitSet && e.ResourceWait == nil
})
r.wg.Wait()
if calls.Load() != 0 {
t.Fatal("quarantined resource handed off")
}
exists, err := r.store.HasResourceOwner(context.Background(), a.ID)
if err != nil || !exists {
t.Fatal("quarantine evidence lost", err)
}
d, err := r.store.GetRun(context.Background(), b.ID)
if err != nil || d.ResourceWait != nil || len(d.Steps) != 0 {
t.Fatal("canceled queue consumed step", err)
}
})
}
}
func TestResourceEventClearCoalescing(t *testing.T) {
frame := protocol.RunEventFrame{ResourceWaitSet: true, ResourceWait: &protocol.ResourceWait{Resource: "X"}}
mergeRunEventFrame(&frame, &protocol.RunEventFrame{ResourceWaitSet: true})
if !frame.ResourceWaitSet || frame.ResourceWait != nil {
t.Fatal("coalescer retained stale resource wait")
}
}
func TestResourceReleaseOnBeginAndPersistenceFailure(t *testing.T) {
for _, failure := range []string{"begin", "persist"} {
t.Run(failure, func(t *testing.T) {
r, events, start := resourceRunner(t)
db, err := sql.Open("sqlite", filepath.Join(r.store.Directory(), "resources.db"))
if err != nil {
t.Fatal(err)
}
defer db.Close()
trigger := `CREATE TRIGGER resource_fail BEFORE INSERT ON run_steps WHEN json_extract(NEW.input_json,'$.prompt')='fail' BEGIN SELECT RAISE(FAIL,'test BeginStep failure'); END`
if failure == "persist" {
trigger = `CREATE TRIGGER resource_fail BEFORE UPDATE OF output_json ON run_steps WHEN json_extract(OLD.input_json,'$.prompt')='fail' BEGIN SELECT RAISE(FAIL,'test persistence failure'); END`
}
if _, err := db.Exec(trigger); err != nil {
t.Fatal(err)
}
var calls atomic.Int32
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
calls.Add(1)
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
a := start("X", "1s", "fail")
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "failed" })
r.wg.Wait()
if exists, err := r.store.HasResourceOwner(context.Background(), a.ID); err != nil || exists {
t.Fatal("safe failure stranded resource", err)
}
if failure == "begin" && calls.Load() != 0 {
t.Fatal("BeginStep failure executed work")
}
b := start("X", "1s", "pass")
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
})
}
}
func TestResourceCleanupQueryFailureQuarantines(t *testing.T) {
r, events, start := resourceRunner(t)
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
if err := r.store.Close(); err != nil {
return stage.Result{}, err
}
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
a := start("X", "1s", "query-failure")
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "failed" })
r.wg.Wait()
reopened, err := pipeline.Open(filepath.Join(r.store.Directory(), "resources.db"))
if err != nil {
t.Fatal(err)
}
defer reopened.Close()
if err := reopened.RecoverRuns(context.Background()); err != nil {
t.Fatal(err)
}
if exists, err := reopened.HasResourceOwner(context.Background(), a.ID); err != nil || !exists {
t.Fatal("query failure released uncertain owner", err)
}
}
func TestResourceSkippedVisitDoesNotAcquire(t *testing.T) {
r, events, _ := resourceRunner(t)
ctx := context.Background()
source := strings.Replace(pipeline.Template([]string{"review"}, true), "success -> #success", "success -> #success\nskipped -> #success", 1) + "```resources\n{\"exclusive\":\"X\"}\n```\n"
if _, err := r.store.Submit(ctx, "skip", source); err != nil {
t.Fatal(err)
}
items, _ := r.store.List(ctx)
d, _ := r.store.Get(ctx, items[0].ID, 1)
owner, _, err := r.store.CreateRun(ctx, d, "held", t.TempDir(), false, nil, nil)
if err != nil {
t.Fatal(err)
}
lease, ch, _, err := r.store.RequestResource(ctx, pipeline.ResourceVisit{Resource: "X", RunID: owner, Stage: "review", Sequence: 1})
if err != nil || ch != nil {
t.Fatal("owner fixture failed", err)
}
defer r.releaseResource(lease, true)
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
t.Error("skipped stage executed")
return stage.Result{}, nil
}
run, err := r.start(protocol.Message{PipelineID: d.ID, Version: 1, RequestKey: "skip", Workspace: t.TempDir(), Input: map[string]any{"prompt": "no UI"}})
if err != nil {
t.Fatal(err)
}
awaitResourceEvent(t, events, run.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
detail, err := r.store.GetRun(ctx, run.ID)
if err != nil || len(detail.Steps) != 1 || detail.Steps[0].Outcome != "skipped" {
t.Fatal("skip not preserved", detail, err)
}
}
+12 -17
View File
@@ -424,9 +424,6 @@ func (s *runSubscription) close() {
}
func mergeRunEventFrame(into, in *protocol.RunEventFrame) {
if in.ResourceWaitSet {
into.ResourceWaitSet, into.ResourceWait = true, in.ResourceWait
}
if in.Workspace != "" {
into.Workspace = in.Workspace
}
@@ -503,20 +500,18 @@ func handleRunEvents(request protocol.Message, events *runEventHub, runs *runner
continue
}
frame := protocol.RunEventFrame{
ResourceWaitSet: true,
ResourceWait: summary.ResourceWait,
RunID: summary.ID,
Workspace: summary.Workspace,
PipelineID: summary.PipelineID,
Name: summary.Name,
Title: summary.Title,
Version: summary.Version,
Status: summary.Status,
Outcome: summary.Outcome,
CurrentStage: summary.CurrentStage,
CompletedSteps: summary.CompletedSteps,
CreatedAt: summary.CreatedAt,
FinishedAt: summary.FinishedAt,
RunID: summary.ID,
Workspace: summary.Workspace,
PipelineID: summary.PipelineID,
Name: summary.Name,
Title: summary.Title,
Version: summary.Version,
Status: summary.Status,
Outcome: summary.Outcome,
CurrentStage: summary.CurrentStage,
CompletedSteps: summary.CompletedSteps,
CreatedAt: summary.CreatedAt,
FinishedAt: summary.FinishedAt,
}
if !sub.writeSnapshot(protocol.Message{Type: protocol.RunEvent, RunEvent: &frame}) {
// The connection write failed; the subscription is closed and the
+15 -83
View File
@@ -86,10 +86,6 @@ func newRunner(ctx context.Context, s *pipeline.Store) (*runner, error) {
}
cancel()
if err != nil {
if resource, checkErr := s.HasResourceOwner(ctx, owner.RunID); checkErr == nil && resource {
log.Printf("resource owner run=%d remains quarantined: cleanup not confirmed", owner.RunID)
continue
}
return nil, fmt.Errorf("cannot interrupt previously active agent %s: %w", owner.ID, err)
}
if err := s.ConfirmRecoveredSession(ctx, owner); err != nil {
@@ -224,9 +220,6 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
// workspace) must not block new work.
for _, owner := range owners {
if _, ok := r.active[owner.RunID]; !ok {
if resource, checkErr := r.store.HasResourceOwner(r.ctx, owner.RunID); checkErr == nil && resource && owner.Directory != workspace {
continue
}
r.mu.Unlock()
return nil, fmt.Errorf("an agent may still be active; restart recovery must interrupt it before new work")
}
@@ -452,7 +445,7 @@ func retryInputOverride(base map[string]any, override map[string]any) (map[strin
// retryStageContractEqual compares the complete persisted stage contract. A
// retry may carry completed output across a published version only when the
// stages that produced that output are unchanged; comparing only effective
// types would allow prompt, config, edge, resource, or limit changes through.
// types would allow prompt, config, edge, or limit changes through.
func retryStageContractEqual(left, right protocol.Stage) bool {
a, err := json.Marshal(left)
if err != nil {
@@ -1182,14 +1175,13 @@ func (r *runner) finishRunState(id int64, name, workspace, status, outcome, diag
// Push the terminal delta before the attention notification, so a slow ntfy
// never delays the live update the app folds.
r.emitRun(protocol.RunEventFrame{
RunID: id,
Workspace: workspace,
Status: status,
ResourceWaitSet: true,
Outcome: outcome,
CurrentStage: currentStage,
CompletedSteps: completed,
FinishedAt: time.Now().UTC().Format(time.RFC3339),
RunID: id,
Workspace: workspace,
Status: status,
Outcome: outcome,
CurrentStage: currentStage,
CompletedSteps: completed,
FinishedAt: time.Now().UTC().Format(time.RFC3339),
}, true)
r.notifyRun(id, name, status, sequence, currentStage, diagnostic)
// The run is terminal: nothing more can notify for it.
@@ -1257,21 +1249,6 @@ func (r *runner) finishStage(id int64, name, workspace string, selected protocol
}
func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, name, workspace string, request protocol.Message, state *runState) {
var held *pipeline.ResourceLease
resourceQuiet := true
release := func() error {
if held != nil {
err := r.releaseResource(*held, resourceQuiet)
held = nil
return err
}
return nil
}
defer func() {
if err := release(); err != nil {
log.Printf("resource cleanup run=%d: %v", id, err)
}
}()
finish := func(status, outcome, diagnostic string) {
r.finishRunState(id, name, workspace, status, outcome, diagnostic, state.completed, state.nextSequence-1, state.key, state.input)
}
@@ -1336,7 +1313,7 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
maxVisits = selected.Limits.MaxVisits
}
// Validate the next visit without consuming it. An already-exhausted
// stage must not queue for a resource it can never execute.
// stage must not begin work it can never execute.
if !isHuman {
if state.visits[key]+1 > 5 {
finish("failed", "error", "stage visit budget exceeded")
@@ -1349,27 +1326,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
}
_, declaresSkip := def.Edges["skipped"]
skip := !isHuman && declaresSkip && !state.uiSurface
if selected.Resources != nil && !skip {
if isHuman || (selected.Resources.Exclusive == "" && selected.Resources.Pool == "") || (selected.Resources.Exclusive != "" && selected.Resources.Pool != "") || (selected.Resources.Exclusive != "" && !pipeline.ValidResourceKey(selected.Resources.Exclusive)) || (selected.Resources.Pool != "" && !pipeline.ValidResourceKey(selected.Resources.Pool)) {
finish("failed", "error", "invalid persisted resource declaration")
return
}
resource := selected.Resources.Exclusive
pool := selected.Resources.Pool
if pool != "" {
resource = pool
}
lease, err := r.acquireResourceWithPool(ctx, pipeline.ResourceVisit{Resource: resource, RunID: id, Stage: key, Sequence: sequence}, pool, workspace, state.completed)
if err != nil {
if ctx.Err() != nil {
finish("canceled", "", err.Error())
} else {
finish("failed", "error", err.Error())
}
return
}
held, resourceQuiet = &lease, true
}
state.nextSequence = sequence + 1
state.visits[key]++
// Publish the run's server-captured changed paths into the input memory
@@ -1403,13 +1359,12 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
}
// queued -> running: report the new current stage and the step start.
r.emitRun(protocol.RunEventFrame{
RunID: id,
Workspace: workspace,
Status: protocol.RunStatusRunning,
CurrentStage: key,
ResourceWaitSet: true,
CompletedSteps: state.completed,
Step: &protocol.RunStepDelta{Sequence: sequence, Stage: key, Status: protocol.StepStatusRunning},
RunID: id,
Workspace: workspace,
Status: protocol.RunStatusRunning,
CurrentStage: key,
CompletedSteps: state.completed,
Step: &protocol.RunStepDelta{Sequence: sequence, Stage: key, Status: protocol.StepStatusRunning},
}, false)
if isHuman {
// A human step is O(1): one BeginStep plus one MarkWaiting, no
@@ -1516,9 +1471,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
var piClient *stage.PiClient
var err error
stageCtx := ctx
if held != nil && held.Target != nil {
stageCtx = stage.WithResourceTarget(stageCtx, held.Target)
}
var modelConfig stage.ModelConfig
var budgetBefore stage.BudgetSnapshot
if stage.EffectiveType(def.Type) == stage.ModelType {
@@ -1542,9 +1494,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
snapshotCancel()
var stageSession string
panicked := false
if held != nil {
resourceQuiet = false
}
func() {
defer func() {
if recover() != nil {
@@ -1600,12 +1549,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
result, err = stage.DefaultRegistry().Run(stageCtx, def, input)
}
}()
if held != nil {
resourceQuiet = r.resourceQuiescent(id, err, panicked)
if confirmed, reported := result.Output[protocol.ResourceCleanupConfirmedKey]; reported && confirmed != true {
resourceQuiet = false
}
}
// Capture after cleanup, even when the stage's context was canceled.
snapshotCtx, snapshotCancel = context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
after := filediff.Capture(snapshotCtx, workspace, before)
@@ -1698,9 +1641,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
}
step.Target = def.Edges[step.Outcome]
step.Control = &protocol.ControlEvidence{Visit: state.visits[key], MaxVisits: maxVisits, Memory: reads}
if held != nil && held.Target != nil {
step.Control.ResourceTarget = held.Target
}
if err == nil && selected.Limits != nil && state.visits[key] >= maxVisits {
for _, outcome := range selected.Limits.On {
if outcome == step.Outcome {
@@ -1765,14 +1705,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
return
}
}
if held != nil && !resourceQuiet {
finish("failed", "error", "resource cleanup/restoration not confirmed; ownership quarantined pending inspection")
return
}
if err := release(); err != nil {
finish("failed", "error", "resource release failed; retained ownership needs reconciliation: "+err.Error())
return
}
if ctx.Err() != nil {
finish("canceled", "", ctx.Err().Error())
return
-312
View File
@@ -1,312 +0,0 @@
# stage implementer
```limits
{"max_visits":5,"on":["blocked_retryable"]}
```
```memory
{"task": "input.prompt", "code_findings": "reviewer.findings?", "ui_findings": "ui_reviewer.findings?"}
```
```output
{ task, acceptance_checks, changed_files, implementation_summary, new_helpers }
```
```prompt
Implement input.prompt in the run directory using the repository's actual languages and
file types; create files as needed.
Read android/AGENTS.md first. It is the map of this project - how to build, test, run and capture here, where
the pieces live, and the conventions that apply. Do not re-derive what it tells you and do not go
looking for what it already names; if it is wrong or out of date, say so in your output.
Read the surrounding code before editing so you extend the existing helpers and patterns rather than
duplicating them. Keep each change as small as the task allows and follow the repository's own
conventions and error handling.
This is the Android app. Build and test it with the project's own Gradle invocation as the map
gives it, and treat the map's geometry tokens as the only source of dimensions - a UI change that
adds a raw value fails the repository's own check. Perform code/build/unit tests only;
no adb, install, capture, device-setting changes or restoring main's build. The UI reviewer
alone verifies rendered surfaces under its declared runner resource lease.
Record the acceptance checks in acceptance_checks, the actual changes in changed_files, and any new
shared helper in new_helpers. Verify with the checks this repository uses - as the map gives them -
and report the exact commands and observed output. Address any concrete findings the runner supplies
and note the actual fixes. Return blocked if the required scope or tools are unavailable. Do not
commit or push; the committer stage lands the work.
```
```tools
read
glob
grep
edit
shell
```
```configs
provider=openai
model=gpt-5.6-terra
retries=1
timeout=30m
```
```edges
success -> reviewer
blocked_retryable -> implementer
blocked -> #failure
error -> #failure
```
# stage reviewer
```limits
{"max_visits": 3, "on": ["rejected"]}
```
```memory
{"task": "input.prompt", "acceptance_checks": "implementer.acceptance_checks", "changed_files": "implementer.changed_files", "implementation": "implementer.implementation_summary?"}
```
```output
{ summary, changed_files, verification_evidence, duplication_review, performance_review, findings }
```
```prompt
Verify with builds, unit tests and source review as android/AGENTS.md gives them.
Do not use adb, install, capture, change device settings or restore main's build.
Device verification belongs exclusively to the UI reviewer under its runner resource lease.
Enforce the map's geometry-token rule: a raw dimension outside the token file is a must-fix finding.
Review the actual changes against the task in input.prompt and the acceptance
checks the implementer recorded. Run the repository's relevant verification checks
with the stage tools (for example build, lint, tests or type checks) and report the
exact commands and observed results in verification_evidence.
Independently assess code quality:
- Duplication: search the run directory for repeated or near-identical functions,
blocks and copy-pasted logic introduced by the change; identify code that should be
shared and record concrete locations in duplication_review.
- Performance and resources: flag redundant computation, unnecessary dependencies,
and repeated I/O or database/filesystem work that wastes time, CPU or memory;
record it in performance_review.
- Workarounds: flag hacks, suppressed errors and shortcuts that trade reliability for
speed in findings.
Keep verification proportionate to the change: exercise the files and screens the diff
actually touches, capture at most one screenshot per changed surface plus one regression
screen, and do not repeat sweeps or measurements that an earlier visit already recorded.
Return rejected ONLY when at least one must-fix finding exists: a defect that blocks
acceptance and that you can locate concretely (file and line), with evidence and a
remedy. Do not return rejected for advisory-only, cosmetic or nit items. When there
is no must-fix finding but you have non-blocking notes (duplication, performance or
workaround observations), record them in findings and return advisory, which
continues the run without another implementer visit. Return success only when the
observed checks pass and you have nothing further to record. Do not edit files or
claim unexecuted checks.
```
```tools
read
glob
grep
shell
```
```configs
provider=openai
model=gpt-5.6-luna
retries=1
timeout=45m
```
```edges
success -> ui_reviewer
advisory -> ui_reviewer
rejected -> implementer
error -> #failure
```
# stage ui_reviewer
```resources
{"exclusive":"phone-47241FDAS00DLK"}
```
```limits
{"max_visits": 3, "on": ["rejected"]}
```
```memory
{"task": "input.prompt", "acceptance_checks": "implementer.acceptance_checks", "changed_files": "implementer.changed_files", "implementation": "implementer.implementation_summary?"}
```
```output
{ summary, changed_files, ui_applicable, screenshots, ui_review, platform_conventions, findings }
```
```prompt
Decide whether this change touches a user-facing UI. If it does not (a backend,
library, CLI, config or data-only change with no rendered surface), set
ui_applicable=false, explain briefly in summary, and return skipped. Do not invent
screenshots or visual work.
When the change does affect a user-facing UI, verify it visually:
Read android/AGENTS.md first. You alone perform phone work while the runner holds
this stage's declared resource lease. Use controlled fixtures for changed states,
including human-decision sheets, on a scratch server/database and throwaway session.
Never mutate an operator session or restart deployed services. Record original
settings and connection/workspace; use the configured authoritative endpoint, not a
hardcoded scratch default. Keep every shell/ADB operation foreground and awaited.
Restore original settings and main's build before successful return, under the same
lease. Report capture/restoration failures explicitly; do not approve incomplete
verification or advertise readiness when cleanup/restoration is uncertain.
Include resource_cleanup_confirmed in output: true only after confirmed restoration;
false on any uncertainty (a release veto that quarantines ownership even if tools are idle).
- Build and launch the application the way the repository actually does it, then
capture screenshots of every relevant screen, state and viewport (default, empty,
loading, error, long content, small and large sizes) with the target platform's
screenshot tooling through shell. Prefer a headless or scripted capture when the
project supports one. Save the images under the run directory and list the exact
capture commands and image paths in screenshots.
- Read the captured images and review them as a UI/UX designer: layout, spacing,
alignment, typographic hierarchy, color and contrast, iconography, interaction
and focus states, loading/empty/error states, accessibility, and consistency.
- Judge the UI against established industry patterns and, above all, the native
conventions of the platform the app targets (operating system, toolkit or web
standards). Record the platform you reviewed against in platform_conventions,
and any deviations and concrete fixes in findings. Infer the target platform from
the task and the repository's actual UI toolkit when it is unstated.
- Approve only when the screenshots show the task and the acceptance checks are met
and the UI is correct, consistent and native to the target platform; otherwise
return rejected with specific, actionable findings the implementer can fix.
Keep the capture pass proportionate: capture each changed state once (for example one
collapsed and one expanded section, one run state, one stuck header) instead of every
screen at multiple viewports or pan directions, and reuse what this run already captured
rather than re-doing it.
If the change is UI-facing but screenshots genuinely cannot be captured
(for example no runnable UI or no display), return blocked rather than approving.
Do not edit files, and do not approve a UI you did not actually see.
Return rejected only for a must-fix visual defect with a concrete location, evidence
and remedy; record non-blocking polish, duplication or convention notes in findings
and return advisory, which continues the run without another implementer visit.
```
```tools
read
glob
grep
shell
```
```configs
provider=openai
model=gpt-5.6-luna
retries=1
timeout=45m
```
```edges
skipped -> committer
approved -> committer
advisory -> committer
rejected -> implementer
blocked -> #failure
error -> #failure
```
# stage committer
```memory
{"task": "input.prompt", "changed_files": "implementer.changed_files", "review": "reviewer.summary?", "ui_review": "ui_reviewer.summary?"}
```
```output
{ commit, files, conflicts_resolved, summary }
```
```prompt
Commit the work this run produced, and resolve any conflict that is present. This stage
runs after the reviews have approved and is the only stage allowed to commit.
Before committing, check that the change is reflected where it belongs: the packaged
templates, the packaged
authoring skill references. Update anything the change makes stale, and state in the
commit body what you checked, saying explicitly when none of them needs updating.
Stage exactly the files this run changed - the paths in changed_files that still exist -
using the repository's own version control commands, and commit them as one commit whose
subject says what the change does and whose body summarises it. Do not stage unrelated
files, generated or ignored files, or anything another change owns. Do not invent work
that was not done; describe what is actually in the diff.
If the workspace is in a conflicted state - unmerged paths, or a merge, rebase or
cherry-pick left unfinished - resolve each conflict: keep both intents where they are
compatible, and choose the result that satisfies the task where they are not, then
complete the operation so the tree is clean and the commit is finished. Never commit
conflict markers, and never discard the other side's work silently; record what you
resolved.
Do not push, tag, revert, or rewrite unrelated history. If there is nothing to commit,
report that instead of creating an empty commit. Report the commit identifier, the files
committed, whatever you resolved, and anything you deliberately left uncommitted.
Finally, bring this branch up to date with main, in this same stage. After committing the work
above, merge main into this branch here, resolve any conflict here - keeping both intents where they
are compatible, and choosing the result that satisfies this task where they are not - run the
repository's checks again after resolving, and commit the completed merge with a message naming what
was combined. The point is that this branch can later be merged back without conflict: leave the
tree clean and the merge finished. Never leave conflict markers, never discard the other side's work
silently. If the merge genuinely cannot be completed, say so with the path and the reason instead of
leaving the merge half-finished.
Then land it: this branch is the work and main is where it belongs, so merge this branch into main
as the last act of this stage. Find the repository's main working tree with `git worktree list` (the
entry checked out on main, not this one) and merge this branch into it there with `--no-ff`, with a
message saying what the change does. Two lanes can land at the same moment, so take a lock first -
`mkdir` a lock directory beside main's `git` directory in a bounded retry loop, and remove it when
done - and retry briefly if another lane holds it. Then verify with
`git merge-base --is-ancestor <this branch> main` and report the commit main now has. If main cannot
be updated, say exactly why and leave this branch committed and clean rather than leaving main
half-merged. There is no separate merge pipeline and nothing to hand this to: if it cannot be landed
here, it is reported here.
Do not edit README or other shared documentation files: an item records what it changed and which documentation is now stale in its own commit body, and the shared documents are folded by a single writer.
```
```tools
read
glob
grep
edit
shell
```
```configs
provider=openai
model=gpt-5.6-terra
retries=1
timeout=30m
```
```edges
success -> #success
blocked -> #failure
error -> #failure
```
-50
View File
@@ -2,9 +2,7 @@ package server
import (
"context"
"fmt"
"log"
"net"
"net/http"
"strings"
"sync"
@@ -270,44 +268,6 @@ func HandlerWithOptions(ctx context.Context, options Options, stores ...*pipelin
response.Error = err.Error()
}
}
} else if (request.Type == protocol.InspectResource || request.Type == protocol.ReconcileResource || request.Type == protocol.InspectResourcePool || request.Type == protocol.ConfigureResourcePool) && len(stores) > 0 {
var err error
response.Type = protocol.ResourceInspectionResult
// Recovery has a separate transport path. A stage/model client may
// also be local, so loopback alone is not an authority boundary.
// The CLI uses /operator-connect; the ordinary /connect path is
// deliberately refused even from loopback.
if r.URL.Path != "/operator-connect" || !isLoopbackPeer(r.RemoteAddr) {
response.Type = protocol.Error
response.Error = "resource recovery requires the local operator transport"
} else if request.Type == protocol.InspectResourcePool {
response.Type = protocol.ResourcePoolInspection
var pool protocol.ResourcePool
pool, err = stores[0].InspectResourcePool(ctx, request.Resource)
response.ResourcePool = &pool
} else if request.Type == protocol.ConfigureResourcePool {
response.Type = protocol.ResourcePoolConfigured
if request.ResourcePool == nil {
err = fmt.Errorf("resource pool declaration is required")
} else {
err = stores[0].ConfigureResourcePool(ctx, *request.ResourcePool)
}
} else if request.Type == protocol.InspectResource {
var inspection protocol.ResourceInspection
inspection, err = stores[0].InspectResource(ctx, request.Resource)
response.ResourceInspection = &inspection
} else {
response.Type = protocol.ResourceReconciled
if request.ResourceRecovery == nil {
err = fmt.Errorf("resource recovery evidence is required")
} else {
err = stores[0].ReconcileResource(ctx, *request.ResourceRecovery)
}
}
if err != nil {
response.Type = protocol.Error
response.Error = err.Error()
}
} else if runs != nil && (request.Type == protocol.RunPipeline || request.Type == protocol.RetryRun || request.Type == protocol.ListRuns || request.Type == protocol.GetRun || request.Type == protocol.CancelRun || request.Type == protocol.DecideRun || request.Type == protocol.SetRunTitle || request.Type == protocol.GetRunEvidence) {
var err error
switch request.Type {
@@ -365,19 +325,9 @@ func HandlerWithOptions(ctx context.Context, options Options, stores ...*pipelin
}
}
mux.HandleFunc("GET /connect", connect)
mux.HandleFunc("GET /operator-connect", connect)
return &runHandler{Handler: mux, runner: runs}
}
func isLoopbackPeer(remoteAddr string) bool {
host, _, err := net.SplitHostPort(remoteAddr)
if err != nil {
return false
}
ip := net.ParseIP(host)
return ip != nil && ip.IsLoopback()
}
type runHandler struct {
http.Handler
runner *runner
+9 -19
View File
@@ -8,8 +8,6 @@ import (
"sort"
"strings"
"time"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
var ErrInvalidModelResult = errors.New("invalid model stage result")
@@ -38,13 +36,6 @@ func harnessUnreachable(err error) error {
}
func modelPrompt(def Definition, spec ModelSpec, input Values) (string, error) {
return modelPromptForTarget(def, spec, input, nil)
}
// modelPromptForTarget carries a server-selected resource target separately from
// task input. The target comes only from the runner context; task input remains
// data and cannot select or replace it.
func modelPromptForTarget(def Definition, spec ModelSpec, input Values, target *protocol.ResourceTarget) (string, error) {
outcomes := make([]string, 0, len(def.Edges))
for name := range def.Edges {
outcomes = append(outcomes, name)
@@ -68,16 +59,15 @@ func modelPromptForTarget(def Definition, spec ModelSpec, input Values, target *
toolsLine = "This call has these stage tools: " + strings.Join(tools, ", ") + "."
}
data, err := json.Marshal(struct {
Key string `json:"stage"`
Instructions string `json:"instructions"`
OutputContract string `json:"output_contract"`
Outcomes []string `json:"allowed_outcomes"`
Input Values `json:"input"`
ResourceTarget *protocol.ResourceTarget `json:"resource_target,omitempty"`
ReportFileArtifacts bool `json:"report_file_artifacts"`
ExecutionTime string `json:"execution_time"`
Timezone string `json:"timezone"`
}{def.Key, spec.Prompt, spec.Output, outcomes, input, target, def.ReportFileArtifacts, time.Now().Format(time.RFC3339), time.Local.String()})
Key string `json:"stage"`
Instructions string `json:"instructions"`
OutputContract string `json:"output_contract"`
Outcomes []string `json:"allowed_outcomes"`
Input Values `json:"input"`
ReportFileArtifacts bool `json:"report_file_artifacts"`
ExecutionTime string `json:"execution_time"`
Timezone string `json:"timezone"`
}{def.Key, spec.Prompt, spec.Output, outcomes, input, def.ReportFileArtifacts, time.Now().Format(time.RFC3339), time.Local.String()})
if err != nil {
return "", fmt.Errorf("model stage input is not JSON-serializable: %w", err)
}
+1 -2
View File
@@ -16,8 +16,7 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
return Result{}, err
}
spec.Config.ResultSchema = resultSchema(def.Edges)
target, _ := ResourceTarget(ctx)
prompt, err := modelPromptForTarget(def, spec, input, target)
prompt, err := modelPrompt(def, spec, input)
if err != nil {
return Result{}, err
}
-103
View File
@@ -9,8 +9,6 @@ import (
"strings"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
func modelDefinition() Definition {
@@ -196,107 +194,6 @@ func TestModelRunUsesInstructionsAndStructuredOutput(t *testing.T) {
}
}
func TestModelRunForwardsOnlyServerGrantedResourceTarget(t *testing.T) {
target := &protocol.ResourceTarget{Member: "emulator", Metadata: map[string]string{"serial": "emu-5554", "type": "emulator"}}
budget := NewBudget(0, 0)
input := Values{"resource_target": map[string]any{"member": "user-selected"}, "prompt": "task"}
ctx := WithBudget(WithResourceTarget(context.Background(), target), budget)
var received ModelConfig
client := generatorFunc(func(_ context.Context, config ModelConfig, prompt string) (string, error) {
received = config
assertPromptResourceTarget(t, prompt, target, input)
return `{"outcome":"success","output":{}}`, nil
})
if _, err := (ModelStage{Client: client}).Run(ctx, modelDefinition(), input); err != nil {
t.Fatal(err)
}
if received.Budget != budget {
t.Fatal("budget context was not preserved with resource target")
}
}
func TestModelRunForwardsResourceTargetToNativeOpenCode(t *testing.T) {
target := &protocol.ResourceTarget{Member: "phone", Metadata: map[string]string{"serial": "phone-serial"}}
var prompt string
client := &OpenCodeClient{Directory: "/work/repo", command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
switch args[2] {
case "/api/session":
return []byte(`{"data":{"id":"ses_target"}}`), nil
case "/api/session/ses_target/generate":
var body struct {
Prompt string `json:"prompt"`
}
if err := json.Unmarshal([]byte(args[4]), &body); err != nil {
t.Fatal(err)
}
prompt = body.Prompt
return []byte(`{"data":{"text":"{\"outcome\":\"success\",\"output\":{}}"}}`), nil
default:
t.Fatalf("unexpected fake OpenCode endpoint %q", args[2])
return nil, nil
}
}}
if _, err := (ModelStage{Client: client}).Run(WithResourceTarget(context.Background(), target), modelDefinition(), Values{"prompt": "task"}); err != nil {
t.Fatal(err)
}
assertPromptResourceTarget(t, prompt, target, Values{"prompt": "task"})
}
func TestModelPromptWithoutResourceTargetPreservesLegacyInput(t *testing.T) {
input := Values{"prompt": "task", "count": "1"}
spec, err := (ModelStage{}).Spec(modelDefinition())
if err != nil {
t.Fatal(err)
}
prompt, err := modelPrompt(modelDefinition(), spec, input)
if err != nil {
t.Fatal(err)
}
var payload struct {
Input Values `json:"input"`
ResourceTarget *protocol.ResourceTarget `json:"resource_target"`
}
if _, data, ok := strings.Cut(prompt, "Stage definition and runtime input:\n"); !ok || json.Unmarshal([]byte(data), &payload) != nil {
t.Fatal("could not decode legacy runtime payload")
}
if payload.ResourceTarget != nil || !reflect.DeepEqual(payload.Input, input) {
t.Fatalf("legacy runtime input changed: %#v", payload)
}
}
func assertPromptResourceTarget(t *testing.T, prompt string, want *protocol.ResourceTarget, wantInput Values) {
t.Helper()
_, data, ok := strings.Cut(prompt, "Stage definition and runtime input:\n")
if !ok {
t.Fatal("missing runtime payload")
}
var payload struct {
Input Values `json:"input"`
ResourceTarget *protocol.ResourceTarget `json:"resource_target"`
}
if err := json.Unmarshal([]byte(data), &payload); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(payload.ResourceTarget, want) {
t.Fatalf("resource target = %#v, want %#v", payload.ResourceTarget, want)
}
var fields map[string]json.RawMessage
if err := json.Unmarshal([]byte(data), &fields); err != nil {
t.Fatal(err)
}
var targetFields map[string]json.RawMessage
if err := json.Unmarshal(fields["resource_target"], &targetFields); err != nil {
t.Fatal(err)
}
if len(targetFields) != 2 || targetFields["member"] == nil || targetFields["metadata"] == nil {
t.Fatalf("resource target leaked non-target fields: %s", fields["resource_target"])
}
if !reflect.DeepEqual(payload.Input, wantInput) {
t.Fatalf("input = %#v, want %#v", payload.Input, wantInput)
}
}
func TestModelResultValidationAndRetryBudget(t *testing.T) {
for _, text := range []string{
`not json`, `{"outcome":"missing","output":{}}`, `{"outcome":"success"}`,
+1 -1
View File
@@ -240,7 +240,7 @@ func (c *PiClient) Generate(ctx context.Context, config ModelConfig, prompt stri
close(heartbeatDone)
// agent_end/abort acknowledgement and the parent's exit alone do not
// prove its shell descendants exited. A surviving process group retains
// ownership rather than handing the resource to another stage.
// ownership rather than handing it to another stage.
quiet = quiet && errors.Is(syscall.Kill(-c.cmd.Process.Pid, 0), syscall.ESRCH)
if quiet {
_ = os.WriteFile(filepath.Join(c.Reference, "quiescent"), []byte("quiescent"), 0600)
@@ -12,7 +12,7 @@ import (
"time"
)
func TestResourcePiReceiptRequiresProcessExit(t *testing.T) {
func TestPiReceiptRequiresProcessExit(t *testing.T) {
// A test-owned process only: no Pi provider, shell/ADB command or phone.
cmd := exec.Command("sleep", "5")
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
-26
View File
@@ -5,34 +5,8 @@ package stage
import (
"context"
"fmt"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
type resourceTargetContextKey struct{}
// WithResourceTarget attaches the server-granted concrete target to a stage's
// runtime context. It is deliberately separate from model input and contains
// no capability or credential material.
func WithResourceTarget(ctx context.Context, target *protocol.ResourceTarget) context.Context {
if target == nil {
return ctx
}
copy := &protocol.ResourceTarget{Member: target.Member}
if target.Metadata != nil {
copy.Metadata = map[string]string{}
for key, value := range target.Metadata {
copy.Metadata[key] = value
}
}
return context.WithValue(ctx, resourceTargetContextKey{}, copy)
}
func ResourceTarget(ctx context.Context) (*protocol.ResourceTarget, bool) {
target, ok := ctx.Value(resourceTargetContextKey{}).(*protocol.ResourceTarget)
return target, ok
}
// Values are runtime data, not the prose output contract in a definition.
type Values map[string]any
-43
View File
@@ -1,43 +0,0 @@
package tui
import (
"fmt"
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/protocol"
"github.com/charmbracelet/lipgloss"
"github.com/charmbracelet/x/ansi"
"github.com/muesli/termenv"
)
func TestResourceWaitFrames(t *testing.T) {
previous := lipgloss.ColorProfile()
defer lipgloss.SetColorProfile(previous)
for _, width := range []int{60, 120} {
for _, plain := range []bool{false, true} {
t.Run(fmt.Sprintf("width-%d-plain-%t", width, plain), func(t *testing.T) {
if plain {
lipgloss.SetColorProfile(termenv.Ascii)
} else {
lipgloss.SetColorProfile(termenv.TrueColor)
}
m := decisionModel()
m.width = width
m.runDetail.Status = protocol.RunStatusQueued
m.runDetail.Steps = nil
m.runDetail.ResourceWait = &protocol.ResourceWait{Resource: "phone-X", Stage: "review", Sequence: 1, QueuedSince: "2026-10-18T10:00:00Z"}
frame := ansi.Strip(m.runsDetailsView(24))
if !strings.Contains(frame, "Waiting for resource phone-X") || strings.Contains(frame, "c decide") || strings.Contains(frame, "Waiting for a person") {
t.Fatal("wrong resource wait frame:\n", frame)
}
for _, line := range strings.Split(frame, "\n") {
if ansi.StringWidth(line) > width {
t.Fatalf("overflow: %q", line)
}
}
t.Log("Rendered resource wait (no phone):\n" + frame)
})
}
}
}
-3
View File
@@ -11,9 +11,6 @@ import (
)
func resultLabel(r protocol.RunSummary) string {
if r.ResourceWait != nil {
return r.StatusLabel() + " · stage " + r.ResourceWait.Stage
}
preview := ansi.Strip(r.ResultPreview)
if r.Status == "succeeded" {
if preview != "" {
-3
View File
@@ -276,9 +276,6 @@ func runStatus(status string) string {
}
func (m Model) runSummaryStatus(run protocol.RunSummary) string {
if run.ResourceWait != nil {
return run.StatusLabel()
}
return m.progress(run.Status)
}