Merge branch 'main' into web-shell-polish
This commit is contained in:
-15
@@ -16,21 +16,6 @@ service manager directly. Legacy Python-managed installations require an explici
|
||||
For source development, use `go run ./cmd/pipelines help`; start the native server
|
||||
separately. Set `PIPELINES_URL` or pass `--url` to select another server.
|
||||
|
||||
## Local resource pools
|
||||
|
||||
`pipelines resource-pool-inspect NAME --json` reads a declared pool's members,
|
||||
enabled/quarantined status and non-secret metadata. It does not report lease tokens.
|
||||
`pipelines resource-pool-configure pool.json --yes --json` replaces the full pool
|
||||
declaration; use `-` instead of a file to read stdin. Input is limited to 256 KiB.
|
||||
Without `--yes`, interactive confirmation is required. Both commands require the
|
||||
loopback operator transport; ordinary remote/stage clients are refused.
|
||||
|
||||
Declarations contain `name` and `members`, each with `member`, `enabled`, optional
|
||||
`quarantined` and bounded string `metadata`. Existing validation limits the member
|
||||
count to 32 and metadata to 16 fields per member. Configuration does not release
|
||||
active owners, discard waiters or prove a device is ready; inspect ownership and
|
||||
verify readiness before activating a target. Never put credentials in metadata.
|
||||
|
||||
## Dashboard and status
|
||||
|
||||
`pipelines --url ws://HOST:PORT/connect ping --json` returns `pong` with the
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
# Android device pool — 2026-10-19
|
||||
|
||||
The Android UI reviewer used to request one hard-coded device
|
||||
(`{"exclusive":"phone-47241FDAS00DLK"}`), so a second UI run had to wait behind
|
||||
the single Pixel even when another device was idle. Android UI stages now request
|
||||
the `android-devices` pool instead; the runner grants one free member and passes
|
||||
it to the stage as `resource_target`, and the stage takes its `adb -s` target
|
||||
from `resource_target.metadata.adb_serial`. A stage still never chooses a device.
|
||||
|
||||
## Pool members
|
||||
|
||||
| member | adb serial | metadata |
|
||||
| --- | --- | --- |
|
||||
| `phone-47241FDAS00DLK` | `47241FDAS00DLK` | `kind=phone`, `model=Pixel_9_Pro_XL` |
|
||||
| `emulator-Pipelines_Fold_A` | `emulator-5554` | `kind=emulator`, `avd=Pipelines_Fold_A` |
|
||||
|
||||
The emulator is a single managed **foldable** AVD (`Pipelines_Fold_A`, unfolded
|
||||
built-in display 2076x2152, cover display 1080x2424), started by the operator. It
|
||||
is the only emulator in scope: ad-hoc emulators are still not started per task.
|
||||
|
||||
The operator's **Pixel 10 Pro Fold** (`58281FDCG000JF`) is delivery-only and is
|
||||
deliberately not a pool member — never a test target.
|
||||
|
||||
## Rule reconciliation
|
||||
|
||||
`android/AGENTS.md` previously said "never start or use an emulator" and pinned
|
||||
the Pixel as the only device. That rule is replaced by the pool above so the map,
|
||||
the runner resource declaration and the live device set agree; the safety rules
|
||||
(UI-reviewer-only device work, lease-scoped fixtures, settings/build restoration,
|
||||
fail-closed cleanup) are unchanged.
|
||||
|
||||
## Scope
|
||||
|
||||
This note, the map update and the pool declaration are configuration and
|
||||
documentation only. No device probing, health check or autostart is added: the
|
||||
emulator must already be running and the pool declaration is a static list, so an
|
||||
offline member is simply not grantable until it is running again.
|
||||
@@ -1,25 +0,0 @@
|
||||
# Android target pool allocation — 2026-10-18, phase one not activated
|
||||
|
||||
This item adds a reusable store contract only. Pool declarations are not
|
||||
activated in live jobs, runner handoff, server configuration, Android maps,
|
||||
shared documentation, or deployment.
|
||||
|
||||
`Store.ConfigureResourcePool(ctx, protocol.ResourcePool)` replaces a bounded
|
||||
pool declaration. Each member has a unique 1–128 byte resource key, an enabled
|
||||
flag, quarantine state, and at most 16 bounded ASCII-keyed metadata values
|
||||
(each value is at most 128 bytes). A pool has 1–32 explicit members. Invalid,
|
||||
empty, duplicate, unknown, or malformed declarations are rejected. Updating a
|
||||
pool never transfers a live owner; queued requests use the new declaration on
|
||||
the next eligible release.
|
||||
|
||||
`Store.RequestResourcePool(ctx, pool, visit)` atomically returns a lease for one
|
||||
free eligible concrete member, or a FIFO wait channel. The lease's `Target`
|
||||
contains only the concrete member and bounded non-secret metadata. Capability
|
||||
tokens remain private to the store and are never protocol/model metadata.
|
||||
Direct concrete-resource leases and pool leases share the owner table, so they
|
||||
cannot overlap. Disabled or quarantined members are skipped; if none are
|
||||
eligible, the request queues without inference. Release, cancellation, stale
|
||||
credentials, recovery, and restart retain the existing no-expiry semantics.
|
||||
|
||||
This is scheduler coordination only: no device probing, emulator creation,
|
||||
credentials, activation, or live job changes are included.
|
||||
@@ -1,19 +0,0 @@
|
||||
# Inspected resource recovery
|
||||
|
||||
The operator recovery contract now keeps quarantined resource ownership and its
|
||||
audit transactionally coupled to FIFO handoff. Inspection is redacted and
|
||||
reconciliation is accepted only over a loopback operator connection with the
|
||||
exact owner visit plus positive cleanup/restoration evidence. The persisted
|
||||
capability remains private to the authoritative live store; missing waiter
|
||||
channels fail closed without changing owner or audit state.
|
||||
|
||||
Use the configured `PIPELINES_URL` endpoint, not a disposable URL. First run
|
||||
the redacted resource inspection, then reconcile only after the parked visit
|
||||
has been inspected and quiesced, supplying its exact run/stage/sequence and
|
||||
explicit evidence (including `no mutating action occurred` where applicable).
|
||||
The old deployed server does not support this action; deploy a compatible
|
||||
server before attempting recovery. This is an operator trust boundary, not an
|
||||
OS sandbox or unrestricted authorization grant.
|
||||
|
||||
The shared README and packaged OpenCode skill references should be updated by
|
||||
the manual `docs-fold` process; they are intentionally not edited here.
|
||||
@@ -1,158 +0,0 @@
|
||||
# Stage-exclusive resources — 2026-10-18, not deployed
|
||||
|
||||
This is source work for run 355, **not a deployment, installation or publication**.
|
||||
No physical phone, real operator session or deployed service was used or changed.
|
||||
The Android role split does not retrospectively gate pinned versions or existing runs.
|
||||
|
||||
## Contract and scope
|
||||
|
||||
An executable non-human stage can declare one runner-owned control block:
|
||||
|
||||
```resources
|
||||
{"exclusive":"phone-47241FDAS00DLK"}
|
||||
```
|
||||
|
||||
The key is 1–128 ASCII identifier bytes (letter first, then letters, digits, `_`, `-`).
|
||||
Duplicate/unknown fields, multiple resources and human-stage resources are rejected.
|
||||
One capacity-one resource per stage visit, one server/store authority; no expressions,
|
||||
nested acquisition, distributed scheduling, capacity settings or multi-resource requests.
|
||||
Resource-free definitions retain their execution path. Old servers reject this new block;
|
||||
do not publish to an incompatible server.
|
||||
|
||||
Acquisition follows selection and automatic UI-skip evaluation, before BeginStep,
|
||||
visits/sequences, artifacts, filediff, stage timeout, budget setup or executor calls.
|
||||
Queued runs retain workspace reservation, use legacy `queued` (never human `waiting`),
|
||||
and carry bounded `resource_wait` identity/stage/sequence/queued-since metadata.
|
||||
Events use `resource_wait_set=true` with absent `resource_wait` to clear caches.
|
||||
CLI lists and TUI/Android details show “Waiting for resource …”; human decisions and
|
||||
attention notifications remain separate. Older clients still see an unfinished run.
|
||||
|
||||
SQLite owners and FIFO waiters are indexed by resource/order and run identity.
|
||||
Private random capabilities attribute ownership to resource/run/stage/sequence and
|
||||
never enter wire data, model input, transcripts or artifacts. Only the full capability
|
||||
and positive cleanup evidence authorize release; mismatches and repeated release cannot
|
||||
release a subsequent owner. One buffered channel per live queued run, no resource
|
||||
polling, waiter tickers, model-based acquisition, queue driver or new daemon.
|
||||
|
||||
The gate holds through tool cleanup. OpenCode interruption now confirms session idle/
|
||||
V1 settlement before clearing session ownership. Pi recovery requires both its receipt
|
||||
and process/process-group exit, not merely abort acknowledgement. Executor panic,
|
||||
ErrStageCleanup, retained active sessions and failed cleanup queries quarantine the owner.
|
||||
An explicit `output.resource_cleanup_confirmed=false` (or malformed value) also vetoes
|
||||
release for failed restoration; true never substitutes for harness cleanup evidence.
|
||||
The generic runner does not inspect Android settings or magically verify restoration.
|
||||
The Android UI reviewer must keep all device work foreground/awaited, restore original
|
||||
settings/connection and main's build, and report uncertainty with the release veto.
|
||||
|
||||
## Restart and retention
|
||||
|
||||
Automatic FIFO progress is promised only within a live server process. Startup interrupts
|
||||
persisted queued resource visits, removes their waiter rows, and never replays an executor.
|
||||
Durably cleanup-confirmed owners can be reclaimed through the token-checked delete helper.
|
||||
Unconfirmed owners remain quarantined, including crashes before BeginStep, before session
|
||||
registration, or after cleanup but before its confirmation is persisted. Session interruption
|
||||
alone cannot prove resource-specific restoration. Age never authorizes reclamation.
|
||||
Quarantined ownership/session evidence survives recovery and protects even terminal runs
|
||||
from retention. Reconciliation requires inspected cleanup and restoration, not lock deletion.
|
||||
There is intentionally no force-unlock CLI or automatic uncertain-owner reclamation.
|
||||
|
||||
## Android draft, provenance and activation
|
||||
|
||||
Read-only retrieval: `pipelines show Android --version 2 --json` returned pipeline 20,
|
||||
v2, hash `8c93aeb9b76f1e6e7696415d6308d786ec08d805a1a346fbabe4d252d5d65032`.
|
||||
`python3` recomputed SHA-256 of the returned source and asserted equality.
|
||||
The exact source is retained as `internal/server/testdata/android-v2-published.md`;
|
||||
the proposed source is `internal/server/testdata/android-resource-draft.md`.
|
||||
The provenance test compares graph fields/outcomes unchanged, apart from the approved
|
||||
three stage prompts and the UI review resource; every stage keeps its published v3 model: Terra for implementer/committer and Luna for reviewer/UI reviewer. No unrelated published definition was edited.
|
||||
|
||||
Artifact handoff recovery: the validated v3 draft was copied to this run's retained
|
||||
artifact directory as `android-resource-draft.md`. The similarly named artifact from
|
||||
run 355 remains the old v2 baseline and must not be published. The proposed source in
|
||||
this workspace and this run's artifact have matching SHA-256 digests
|
||||
`a10d8b9fa3dc6a1875b10469061ed25fb4a7a404d463d1a2d304a309d55aa5d9`.
|
||||
|
||||
Activation, requiring separate explicit operator authorization:
|
||||
1. Review/land the source change and independently rerun the required checks.
|
||||
2. Arrange a compatible deployment separately; nothing here deploys or restarts it.
|
||||
3. Reconcile every existing phone user, pinned/live run and any quarantined owner.
|
||||
4. Transfer the **workspace proposed draft**, provenance and this note into the retained
|
||||
stage artifacts after resolving the artifact edit permission.
|
||||
5. Validate that retained draft against the configured authoritative compatible endpoint;
|
||||
only then explicitly publish it. Do not use a disposable endpoint as a project default.
|
||||
6. Opt in other device users explicitly before relying on mutual exclusion system-wide.
|
||||
|
||||
Read-only published inventory (`pipelines show <name> --json`, v2 for each):
|
||||
- UI sweep: `survey` and `ui_reviewer` request visual/screenshots work; phone work there
|
||||
needs explicit resource opt-in and restoration/fixture policy.
|
||||
- Generic: `reviewer` and `ui_reviewer` have visual/device verification wording; when
|
||||
targeting a phone, separate code review from leased device review first.
|
||||
- Plan and land: `implementer`, `reviewer`, `ui_reviewer` contain rendered-surface work;
|
||||
they need an explicit technology-specific role split/lease before phone use.
|
||||
- Probe: one harness smoke stage; no default phone declaration. Any device-specific
|
||||
probe needs explicit opt-in, never an inferred resource or Android stage-name special case.
|
||||
Server/TUI/other definitions were not changed. Inventory is not activation evidence.
|
||||
|
||||
## Verification evidence and remaining limits
|
||||
|
||||
Executed focused checks:
|
||||
- Run 358 recovery revalidation: `go build ./... && go vet ./... && go test ./...`
|
||||
passed; `go test -race ./internal/pipeline ./internal/server ./internal/stage -run
|
||||
'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1` passed; and the
|
||||
Android map invocation with `JAVA_HOME`/`ANDROID_HOME`, installed Gradle,
|
||||
`--offline --no-daemon --rerun-tasks :app:assembleDebug :app:testDebugUnitTest`
|
||||
executed all 41 tasks and reported `BUILD SUCCESSFUL`. These remain build/unit and
|
||||
fake-executor scheduler evidence,
|
||||
not physical-phone verification or screenshots.
|
||||
- Recovery after integrating main `88595eb`: `go build ./...`, `go vet ./...`,
|
||||
`go test ./internal/server -run Resource -count=1`,
|
||||
`go test ./internal/pipeline ./internal/protocol ./internal/tui -run Resource -count=1`,
|
||||
`go test ./internal/stage -count=1`,
|
||||
`go test -race ./internal/pipeline ./internal/server ./internal/stage -run 'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1`, and
|
||||
`go test ./...` all passed. `gofmt -l .` and `git diff --check` produced no output.
|
||||
- Recovery Android build/unit check with the map's offline, no-daemon Gradle invocation:
|
||||
`:app:assembleDebug :app:testDebugUnitTest` passed. This remains build/unit evidence,
|
||||
not phone verification or screenshots.
|
||||
- `go test ./internal/server ./internal/pipeline ./internal/protocol ./internal/tui -run Resource -count=1`:
|
||||
passed. Includes fake-executor queue >30ms timeout, independent-resource progress,
|
||||
FIFO/mismatch/duplicate/grant-cancel cycles, blocked cleanup, panic/quarantine,
|
||||
persistence/BeginStep/query failure, no-acquire skip, restart/retention, wire defaults,
|
||||
narrow/wide/plain TUI frames and real CLI/WebSocket scratch exercise.
|
||||
- `go test ./internal/stage -count=1`: passed; the later full suite also included
|
||||
`TestResourcePiReceiptRequiresProcessExit` (a test-owned sleep process, not a device).
|
||||
- Android map invocation with `JAVA_HOME`/`ANDROID_HOME` and installed Gradle,
|
||||
`--offline --no-daemon :app:assembleDebug :app:testDebugUnitTest`: BUILD SUCCESSFUL.
|
||||
These are build/unit checks, not device verification or screenshots.
|
||||
- After the Go tree settled: `go build ./...`, `go vet ./...`, `go test ./...`
|
||||
all exited zero (including author-package tests).
|
||||
- `go test -race ./internal/pipeline ./internal/server ./internal/stage -run 'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1`:
|
||||
all three packages passed, including concurrent grant/cancel cycles.
|
||||
- `gofmt -l .` and `git diff --check`: exited zero with no output.
|
||||
|
||||
The scratch CLI test uses the real CLI and WebSocket handler with temporary SQLite and
|
||||
an explicitly **FAKE executor**. It validates the proposed Android draft on compatible
|
||||
scratch code, observes B as queued with zero steps while A holds fake-phone, inspects
|
||||
subscription metadata, releases A, runs `wait`, and observes B succeeded automatically.
|
||||
It does not call a model or use a physical phone. Test log output is evidence, not a
|
||||
live-deployment claim; no retained CLI transcript was written to the denied artifact path.
|
||||
|
||||
Live OpenCode/Pi shell/ADB termination and Android visual/device restoration are **not
|
||||
tested** here. The checks prove scheduler behavior with controlled fakes, plus test-owned
|
||||
process-exit checks, not device readiness on the current deployment.
|
||||
Commit/merge/landing belong to the later committer stage; no commit or push here.
|
||||
|
||||
## Stale shared documentation for docs-fold (do not edit in this item)
|
||||
|
||||
- `docs/cli.md`: resource queue presentation, inspection metadata, unchanged wait/decide semantics.
|
||||
- `internal/author/skill/references/format.md`: optional resources block and validation/key bounds.
|
||||
- `internal/author/skill/references/control.md`: runner-owned declaration, timeout/accounting boundary,
|
||||
release veto and automatic skip placement.
|
||||
- `internal/author/skill/references/queue.md`: FIFO resource waits vs workspace exclusion and human waits.
|
||||
- `internal/author/skill/references/operations.md`: authority, restart interruption/quarantine,
|
||||
inspected recovery, retention and explicit compatible-deployment/publication prerequisites.
|
||||
- `internal/author/skill/references/verification.md`: fake-vs-device evidence, fixtures and leased restoration.
|
||||
- `internal/author/skill/references/orchestration.md`: no queue drivers/polling/extra agents; explicit
|
||||
opt-in of every device user, no retroactive gating or distributed guarantees.
|
||||
|
||||
No packaged skill/reference, shared README/topic documentation, pipeline registry,
|
||||
installation, deployment, provider setting or `internal/stage/model_failure.go` was edited.
|
||||
@@ -1,7 +0,0 @@
|
||||
# Resource-pool operator configuration — 2026-10-18
|
||||
|
||||
`resource-pool-configure <pool.json|-> --yes` replaces one bounded pool declaration
|
||||
through the loopback-only operator transport. `resource-pool-inspect <pool>` reads
|
||||
the declared members, enabled/quarantine status, and non-secret metadata. Existing
|
||||
owners and waiters remain under the store's normal ownership, quarantine, and FIFO
|
||||
rules; configuration does not release active leases.
|
||||
@@ -1,13 +0,0 @@
|
||||
# Resource-pool runner handoff — 2026-10-18
|
||||
|
||||
Stage `resources` declarations now accept exactly one of `exclusive` or `pool`.
|
||||
Pool stages wait on the named pool, then receive the selected concrete member and
|
||||
bounded metadata through explicit runtime context. Model execution serializes that
|
||||
server-granted context as a separate `resource_target` runtime field for both
|
||||
native and plain harnesses; the task input map remains unchanged and cannot select
|
||||
the target. Capability tokens remain private to the store. Retained control
|
||||
evidence records the concrete target; queued run state names the pool while an
|
||||
active owner names the selected member.
|
||||
|
||||
This is parser/runner handoff only. It does not add Android definitions, pool
|
||||
configuration rollout, device setup, probing, or deployment.
|
||||
@@ -21,7 +21,7 @@ Use `pipelines wait` for runs you started, without a polling orchestration loop.
|
||||
|
||||
Inspect diffs, run proportionate checks, resolve conflicts and finish landing.
|
||||
Never discard user work or delete branches. Verify ancestry after the actual
|
||||
merge or server gate. Keep device leases and do not disrupt active shared work.
|
||||
merge or server gate. Do not disrupt active shared work.
|
||||
|
||||
The user's request authorizes routine steps needed to complete it. Ask only about
|
||||
real ambiguity, destructive actions or changes outside that scope. Report actual
|
||||
|
||||
@@ -20,8 +20,8 @@ does not freeze unrelated work. Continue at the necessary stage or recover direc
|
||||
## Definitions
|
||||
|
||||
Edit a retained draft, not immutable history. Give each stage one simple job and
|
||||
only the input it needs. Preserve review gates, runtime limits, permissions,
|
||||
resource declarations and the user's constraints unless asked to change them.
|
||||
only the input it needs. Preserve review gates, runtime limits, permissions and
|
||||
the user's constraints unless asked to change them.
|
||||
Use `pipelines validate DRAFT --json` before publication. Publish or execute when
|
||||
the user requests it; do not ask again for routine steps already authorized.
|
||||
Keep version guards. `--yes` is not a conflict policy; never silently force an
|
||||
|
||||
@@ -20,9 +20,8 @@ delivery gate succeeds with a clean checked candidate; the server lands afterwar
|
||||
Read-only, nonvisual work does not require device screenshots. Visual acceptance
|
||||
requires a ready fixture, verified connection and actual captured interactions.
|
||||
|
||||
Preserve exclusive target ownership; do not force-unlock uncertain cleanup.
|
||||
Inspect tool/process termination before reconciling an inactive resource. Stop and
|
||||
await exact owned fixture processes, not just a go-run parent. Do not disrupt a
|
||||
different active lease or restart a shared service over running work.
|
||||
Preserve exclusive fixture ownership; inspect tool/process termination before
|
||||
reuse. Stop and await exact owned fixture processes, not just a go-run parent. Do
|
||||
not disrupt another active fixture or restart a shared service over running work.
|
||||
|
||||
Report the exact blocker and next action briefly. Keep independent work moving.
|
||||
|
||||
@@ -23,7 +23,7 @@ Wait for shared work to quiesce before restarting its service. Back up durable
|
||||
state, preserve endpoint/configuration, verify the new runtime identity and keep
|
||||
a rollback path. Do not wipe data or remove credentials to repair deployment.
|
||||
|
||||
Respect target leases and the user's device boundaries. A test-device authorization
|
||||
Respect the user's device boundaries. A test-device authorization
|
||||
does not authorize use of the personal phone. Installation on the personal Fold
|
||||
requires the user's delivery request; preserve app data and verify the exact APK.
|
||||
Use `android/scripts/capture_png.sh -s SERIAL PATH` for app captures, not a raw
|
||||
|
||||
+3
-146
@@ -7,7 +7,6 @@ import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
@@ -47,9 +46,7 @@ type options struct {
|
||||
keepSet bool
|
||||
all bool
|
||||
json, source, noOpen, yes, allow, createOnly, force, wait bool
|
||||
version, expected, limit, recoverySequence, retryVisit int
|
||||
recoveryRun int64
|
||||
recoveryStage, resource, poolFile string
|
||||
version, expected, limit, retryVisit int
|
||||
timeout, waitTimeout time.Duration
|
||||
stageModels stageModelFlags
|
||||
}
|
||||
@@ -186,18 +183,6 @@ func (a App) flags(command string) (*flag.FlagSet, *options) {
|
||||
case "decide":
|
||||
f.StringVar(&o.option, "option", "", "Option key to choose among the waiting stage's edges (required)")
|
||||
f.StringVar(&o.feedback, "feedback", "", "Optional free-text feedback recorded with the decision")
|
||||
case "resource-inspect":
|
||||
f.StringVar(&o.resource, "resource", "", "Exclusive resource key")
|
||||
case "resource-reconcile":
|
||||
f.StringVar(&o.evidence, "evidence", "", "Bounded positive cleanup/restoration evidence (required)")
|
||||
f.StringVar(&o.actor, "actor", "operator", "Bounded operator identity")
|
||||
f.BoolVar(&o.allow, "no-mutating-action", false, "Evidence explicitly records that no mutating action occurred")
|
||||
f.BoolVar(&o.yes, "yes", false, "Confirm reconciliation without prompting")
|
||||
case "resource-pool-inspect":
|
||||
f.StringVar(&o.resource, "pool", "", "Resource pool name")
|
||||
case "resource-pool-configure":
|
||||
f.StringVar(&o.poolFile, "file", "", "JSON pool declaration, or - for stdin")
|
||||
f.BoolVar(&o.yes, "yes", false, "Confirm this mutation without prompting")
|
||||
case "debug":
|
||||
f.StringVar(&o.harness, "harness", "", "Debugging harness: configured opencode/pi, or nvim")
|
||||
f.BoolVar(&o.yes, "yes", false, "Approve opening a debugging session without prompting")
|
||||
@@ -484,13 +469,6 @@ func (a App) Run(ctx context.Context, args []string) error {
|
||||
requestTimeout = 30 * time.Second
|
||||
}
|
||||
connectURL := o.url
|
||||
if command == "resource-inspect" || command == "resource-reconcile" || command == "resource-pool-inspect" || command == "resource-pool-configure" {
|
||||
var transportErr error
|
||||
connectURL, transportErr = operatorTransportURL(o.url)
|
||||
if transportErr != nil {
|
||||
return transportErr
|
||||
}
|
||||
}
|
||||
dialCtx, cancel := context.WithTimeout(ctx, requestTimeout)
|
||||
c, err := client.Open(dialCtx, connectURL)
|
||||
cancel()
|
||||
@@ -803,7 +781,7 @@ func (a App) Run(ctx context.Context, args []string) error {
|
||||
}
|
||||
case "runs":
|
||||
for _, run := range r.Runs {
|
||||
fmt.Fprintf(a.Out, "%d\t%s\t%s\tv%d\t%s\n", run.ID, run.Label(), run.Name, run.Version, run.StatusLabel())
|
||||
fmt.Fprintf(a.Out, "%d\t%s\t%s\tv%d\t%s\n", run.ID, run.Label(), run.Name, run.Version, run.Status)
|
||||
}
|
||||
case "ping":
|
||||
fmt.Fprintln(a.Out, "Connected ·", r.Time)
|
||||
@@ -926,117 +904,6 @@ func (a App) Run(ctx context.Context, args []string) error {
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("template %q not found", key)
|
||||
case "resource-inspect":
|
||||
if len(pos) > 1 {
|
||||
return fmt.Errorf("resource-inspect takes one resource key")
|
||||
}
|
||||
if o.resource == "" && len(pos) == 1 {
|
||||
o.resource = pos[0]
|
||||
}
|
||||
if strings.TrimSpace(o.resource) == "" {
|
||||
return fmt.Errorf("resource-inspect requires a resource key")
|
||||
}
|
||||
r, err := call(protocol.Message{Type: protocol.InspectResource, Resource: o.resource})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if r.ResourceInspection == nil {
|
||||
return fmt.Errorf("server did not return resource inspection")
|
||||
}
|
||||
return a.output(r.ResourceInspection)
|
||||
case "resource-reconcile":
|
||||
if len(pos) != 4 {
|
||||
return fmt.Errorf("resource-reconcile requires resource, run-id, stage and sequence")
|
||||
}
|
||||
runID, err := strconv.ParseInt(pos[1], 10, 64)
|
||||
if err != nil || runID <= 0 {
|
||||
return fmt.Errorf("run ID must be positive")
|
||||
}
|
||||
sequence, err := strconv.Atoi(pos[3])
|
||||
if err != nil || sequence <= 0 {
|
||||
return fmt.Errorf("sequence must be positive")
|
||||
}
|
||||
if strings.TrimSpace(o.evidence) == "" {
|
||||
return fmt.Errorf("--evidence is required")
|
||||
}
|
||||
if err := a.confirmAction(fmt.Sprintf("Reconcile quarantined resource %s for run #%d?", pos[0], runID), o.yes); err != nil {
|
||||
return err
|
||||
}
|
||||
r, err := call(protocol.Message{Type: protocol.ReconcileResource, Resource: pos[0], ResourceRecovery: &protocol.ResourceRecoveryRequest{Resource: pos[0], RunID: runID, Stage: pos[2], Sequence: sequence, Actor: o.actor, Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, NoMutatingAction: o.allow, Summary: o.evidence}}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.json {
|
||||
return a.output(r)
|
||||
}
|
||||
_, err = fmt.Fprintf(a.Out, "Reconciled resource %s for run #%d\n", pos[0], runID)
|
||||
return err
|
||||
case "resource-pool-inspect":
|
||||
if len(pos) > 1 {
|
||||
return fmt.Errorf("resource-pool-inspect takes one pool name")
|
||||
}
|
||||
if o.resource == "" && len(pos) == 1 {
|
||||
o.resource = pos[0]
|
||||
}
|
||||
if strings.TrimSpace(o.resource) == "" {
|
||||
return fmt.Errorf("resource-pool-inspect requires a pool name")
|
||||
}
|
||||
r, err := call(protocol.Message{Type: protocol.InspectResourcePool, Resource: o.resource})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if r.ResourcePool == nil {
|
||||
return fmt.Errorf("server did not return resource pool")
|
||||
}
|
||||
return a.output(r.ResourcePool)
|
||||
case "resource-pool-configure":
|
||||
if len(pos) > 1 {
|
||||
return fmt.Errorf("resource-pool-configure takes one JSON file")
|
||||
}
|
||||
file := o.poolFile
|
||||
if file == "" && len(pos) == 1 {
|
||||
file = pos[0]
|
||||
}
|
||||
if file == "" {
|
||||
return fmt.Errorf("resource-pool-configure requires a JSON file")
|
||||
}
|
||||
var data []byte
|
||||
if file == "-" {
|
||||
input := a.In
|
||||
if input == nil {
|
||||
input = os.Stdin
|
||||
}
|
||||
data, err = io.ReadAll(io.LimitReader(input, 256*1024+1))
|
||||
} else {
|
||||
var input *os.File
|
||||
input, err = os.Open(file)
|
||||
if err == nil {
|
||||
data, err = io.ReadAll(io.LimitReader(input, 256*1024+1))
|
||||
_ = input.Close()
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(data) > 256*1024 {
|
||||
return fmt.Errorf("resource pool input exceeds 262144 bytes")
|
||||
}
|
||||
var pool protocol.ResourcePool
|
||||
if err := json.Unmarshal(data, &pool); err != nil {
|
||||
return fmt.Errorf("invalid resource pool JSON: %w", err)
|
||||
}
|
||||
if err := a.confirmAction(fmt.Sprintf("Configure resource pool %s?", pool.Name), o.yes); err != nil {
|
||||
return err
|
||||
}
|
||||
r, err := call(protocol.Message{Type: protocol.ConfigureResourcePool, ResourcePool: &pool})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.json {
|
||||
return a.output(r)
|
||||
}
|
||||
_, err = fmt.Fprintf(a.Out, "Configured resource pool %s\n", pool.Name)
|
||||
return err
|
||||
case "inspect", "cancel", "retry":
|
||||
value, err := one()
|
||||
if err != nil {
|
||||
@@ -1217,16 +1084,6 @@ func (a App) Run(ctx context.Context, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func operatorTransportURL(raw string) (string, error) {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || u.Scheme == "" || u.Host == "" {
|
||||
return "", fmt.Errorf("invalid configured pipeline URL")
|
||||
}
|
||||
u.Path = "/operator-connect"
|
||||
u.RawPath = ""
|
||||
return u.String(), nil
|
||||
}
|
||||
|
||||
func (a App) output(value any) error {
|
||||
encoder := json.NewEncoder(a.Out)
|
||||
encoder.SetIndent("", " ")
|
||||
@@ -1672,7 +1529,7 @@ func (e *waitLimitError) Runs() string {
|
||||
// waitLine is the one-line terminal report `pipelines wait` prints: the run id,
|
||||
// its status and outcome, its stage and how long it took.
|
||||
func waitLine(d *protocol.RunDetail) string {
|
||||
line := fmt.Sprintf("run %d · %s", d.ID, d.StatusLabel())
|
||||
line := fmt.Sprintf("run %d · %s", d.ID, d.Status)
|
||||
if outcome := strings.TrimSpace(d.Outcome); outcome != "" {
|
||||
line += " (" + outcome + ")"
|
||||
}
|
||||
|
||||
+53
-58
@@ -2,67 +2,62 @@ package cli
|
||||
|
||||
// Help is also the public command inventory for agents and parity checks.
|
||||
var usage = map[string]string{
|
||||
"provider-check": "provider-check --model PROVIDER/MODEL [--yes] [--json]",
|
||||
"new": "new [name] [prompt] [--template KEY] [--no-open | --json]",
|
||||
"edit": "edit [name|id] [prompt] [--version N] [--no-open | --json]",
|
||||
"drafts": "drafts [--json]",
|
||||
"draft-delete": "draft-delete <draft.md> [--json]",
|
||||
"validate": "validate <draft.md> [--json]",
|
||||
"submit": "submit <draft.md> [--name NAME] [--create-only] [--yes] [--json]",
|
||||
"list": "list [--json]",
|
||||
"search": "search QUERY [--type KIND] [--limit N] [--json]",
|
||||
"show": "show <name|id> [--version N] [--json]",
|
||||
"versions": "versions <name|id> [--json]",
|
||||
"export": "export [name|id ...] [--output FILE.zip] [--all] [--version N]",
|
||||
"import": "import [FILE.zip] [--all] [--on-conflict ask|keep|both|import] [--yes] [--json]",
|
||||
"templates": "templates [--json]",
|
||||
"template": "template <key> [--source | --json]",
|
||||
"run": "run [name|id] [prompt] [--title TITLE] [--model PROVIDER/MODEL] [--stage-model STAGE=PROVIDER/MODEL ...] [--json [--wait [--wait-timeout DURATION]]]",
|
||||
"runs": "runs [--pipeline NAME] [--json]",
|
||||
"inspect": "inspect <run-id> [--json]",
|
||||
"wait": "wait <run-id> [run-id ...] [--timeout DURATION] [--json]",
|
||||
"cancel": "cancel <run-id> [--json]",
|
||||
"decide": "decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]",
|
||||
"retry": "retry <run-id> [--mode resume|restart] [--stage KEY [--visit N]] [--version N] [--input-file FILE|-] [--json [--wait [--wait-timeout DURATION]]]",
|
||||
"title": "title <run-id> \"<title>\" [--json]",
|
||||
"debug": "debug <run-id> [--json | --no-open]",
|
||||
"chat-adopt": "chat-adopt <session-id> [--json]",
|
||||
"connection": "connection [--json]",
|
||||
"ping": "ping [--json]",
|
||||
"version": "version [--json]",
|
||||
"status": "status [--json]",
|
||||
"stats": "stats [--json]",
|
||||
"resource-inspect": "resource-inspect <resource> [--json]",
|
||||
"resource-reconcile": "resource-reconcile <resource> <run-id> <stage> <sequence> --evidence TEXT [--no-mutating-action] [--yes] [--json]",
|
||||
"resource-pool-inspect": "resource-pool-inspect <pool> [--json]",
|
||||
"resource-pool-configure": "resource-pool-configure <pool.json|-> --yes [--json]",
|
||||
"rpc": "rpc <request.json|->",
|
||||
"retention": "retention report|apply [--older-than DURATION] [--keep N] [--db FILE] [--artifacts DIR] [--json] [--yes]",
|
||||
"tui": "tui",
|
||||
"provider-check": "provider-check --model PROVIDER/MODEL [--yes] [--json]",
|
||||
"new": "new [name] [prompt] [--template KEY] [--no-open | --json]",
|
||||
"edit": "edit [name|id] [prompt] [--version N] [--no-open | --json]",
|
||||
"drafts": "drafts [--json]",
|
||||
"draft-delete": "draft-delete <draft.md> [--json]",
|
||||
"validate": "validate <draft.md> [--json]",
|
||||
"submit": "submit <draft.md> [--name NAME] [--create-only] [--yes] [--json]",
|
||||
"list": "list [--json]",
|
||||
"search": "search QUERY [--type KIND] [--limit N] [--json]",
|
||||
"show": "show <name|id> [--version N] [--json]",
|
||||
"versions": "versions <name|id> [--json]",
|
||||
"export": "export [name|id ...] [--output FILE.zip] [--all] [--version N]",
|
||||
"import": "import [FILE.zip] [--all] [--on-conflict ask|keep|both|import] [--yes] [--json]",
|
||||
"templates": "templates [--json]",
|
||||
"template": "template <key> [--source | --json]",
|
||||
"run": "run [name|id] [prompt] [--title TITLE] [--model PROVIDER/MODEL] [--stage-model STAGE=PROVIDER/MODEL ...] [--json [--wait [--wait-timeout DURATION]]]",
|
||||
"runs": "runs [--pipeline NAME] [--json]",
|
||||
"inspect": "inspect <run-id> [--json]",
|
||||
"wait": "wait <run-id> [run-id ...] [--timeout DURATION] [--json]",
|
||||
"cancel": "cancel <run-id> [--json]",
|
||||
"decide": "decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]",
|
||||
"retry": "retry <run-id> [--mode resume|restart] [--stage KEY [--visit N]] [--version N] [--input-file FILE|-] [--json [--wait [--wait-timeout DURATION]]]",
|
||||
"title": "title <run-id> \"<title>\" [--json]",
|
||||
"debug": "debug <run-id> [--json | --no-open]",
|
||||
"chat-adopt": "chat-adopt <session-id> [--json]",
|
||||
"connection": "connection [--json]",
|
||||
"ping": "ping [--json]",
|
||||
"version": "version [--json]",
|
||||
"status": "status [--json]",
|
||||
"stats": "stats [--json]",
|
||||
"rpc": "rpc <request.json|->",
|
||||
"retention": "retention report|apply [--older-than DURATION] [--keep N] [--db FILE] [--artifacts DIR] [--json] [--yes]",
|
||||
"tui": "tui",
|
||||
}
|
||||
|
||||
var notes = map[string]string{
|
||||
"version": "Identify the local CLI binary without connecting to a server. --version is an alias. Use ping --json to identify the running server separately.",
|
||||
"ping": "Pong includes the running server's compiled build.version, build.commit and build.dirty (null when unknown). This identifies the server, not the client or runtime workspace. Older servers omit build. The client compares its own build/protocol against the server: a same-protocol build mismatch prints one warning to stderr, a server speaking a newer protocol fails the command, and a response carrying a field this client does not model prints a one-time unrecognized-field notice instead of dropping it silently.",
|
||||
"connection": "Prints the server endpoint and phone notification endpoints. Like ping, it reports the client/server build relationship and warns or fails rather than silently proceeding against a mismatched server.",
|
||||
"status": "Reports server, host and ntfy publishing health. The client compares its build/protocol against the server and warns once (or fails on a newer protocol) rather than proceeding silently.",
|
||||
"export": "Without a name, select pipelines with Space and save a ZIP in the current directory. --output writes a portable definition bundle; --all skips selection. One name without --output prints Markdown as before. No runs, credentials or old versions are exported.",
|
||||
"import": "Select definitions from a ZIP. Identical existing definitions are skipped. For differing names, choose keep existing, keep both under a new name, or use imported as a new version (history retained). --on-conflict supplies an explicit automation policy; --yes only approves final publication. Schedules may activate; no run history is imported.",
|
||||
"new": "Create an optional template copy or a one-stage starter. Opens the configured default harness; --harness opencode|pi|nvim overrides it. No submission is performed.",
|
||||
"edit": "Omit the name to open fuzzy selection. Original immutable source is copied to a private draft. The configured harness edits that draft, never the live definition.",
|
||||
"validate": "Uses the same server parser as submission. Does not persist or activate schedules. Structural validity does not prove runtime/tool availability.",
|
||||
"submit": "Confirm publication at the prompt, or use --yes for explicit automation; schedules activate on acceptance. Managed drafts provide the name and base-version guard. --force never bypasses validation.",
|
||||
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. --model PROVIDER/MODEL overrides the model for every stage of that run; --stage-model STAGE=PROVIDER/MODEL overrides one stage and is repeatable, with a repeated key last-wins. Precedence is per-stage > whole-run > the model pinned in the pipeline version; overrides apply to that run only and never change the pinned definition. An unknown stage key or a malformed provider/model is refused before the run starts, naming the valid stage keys. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
|
||||
"retry": "Resume a failed stage by default, restoring the completed prefix handoff; use --mode=restart to start at entry. --stage selects a recorded stage (use --visit for repeated stages). --input-file accepts a bounded JSON object, merged with replacement at the selected stage; '-' reads stdin. Original history and pinned identity are retained; --version selects an explicit published version of the same pipeline only. --wait-timeout leaves an active retry for later read-only inspection or reattachment with the original --request-key, rather than starting a duplicate retry.",
|
||||
"title": "Rename a run's operator-facing title after the fact without touching its input, steps or execution. The title is collapsed to a single bounded line and recorded as explicitly set. `pipelines runs` and `pipelines inspect` then show it.",
|
||||
"wait": "Block until each named run reaches a terminal status - succeeded, failed, canceled or interrupted, and also through a run waiting on a person - then print one line per run with its id, status and outcome, its stage and how long it took, and exit 0 even for a run that finished unsuccessfully. Several ids wait together in one command. The exit is non-zero only for a client-side failure: an unknown run id, a connection that could not be established, or --timeout expiring while a run is still active; a timeout says so plainly, never cancels, retries or otherwise touches a run, and is never reported as its outcome. Read-only: it needs no workspace, is not refused by the workspace guard and writes nothing. --json prints the final run envelope instead of the line (one envelope for one id, an array for several).",
|
||||
"decide": "Choose an option for a run waiting on a human stage. The run must be `waiting`; the option must be one of that stage's non-error edges. The stage is inferred from the run's current stage unless named. The choice and optional --feedback are recorded as the step's evidence (visible in inspect and the TUI) and the run resumes along the chosen edge. An unknown option is refused with the valid list; a repeated decision for an already-decided visit does not double-apply. Works across a server restart because the wait is persisted.",
|
||||
"debug": "Open a fresh configured harness conversation to investigate a run using inspect/show/runs CLI queries. --harness opencode|pi|nvim chooses another configured harness or read-only nvim packet. Does not resume implementation or retry. --json returns the prompt without launching anything.",
|
||||
"rpc": "Accepts a JSON protocol envelope; '-' reads stdin. Every TUI server operation is available here too. Mutating commands are not dry runs. The omni-search command is `search` (fields query, optional kind and limit) and answers with ranked `search_results`.",
|
||||
"chat-adopt": "Adopt a session that already exists on the configured OpenCode endpoint so it appears in chat_sessions and can be continued. The id must be one the endpoint reports; its directory and title are taken from the endpoint's own record (never guessed), and a missing id records nothing. Adopting is always explicit and does not change the server's endpoint choice.",
|
||||
"retention": "Local, explicit run-history retention; never automatic and never a side effect of another command. `report` is strictly read-only and prints what a policy would remove; `apply` removes it after approval (--yes for scripted use). Policy: --older-than DURATION selects runs created before now minus that rolling UTC duration (Go durations plus d/w; not a calendar boundary), --keep N keeps the newest N runs by run id and selects older ranks (--keep 0 keeps none). Both together are the conservative intersection; at least one is required. Protected and never removed: queued/running runs, runs waiting on a person (waiting run or step), runs with an active cleanup agent, and any run referenced by a retained run's retry provenance (transitively). A run that becomes protected before deletion aborts the whole apply. Artifacts at <root>/<run> are removed with their run and kept with protected/retained runs; the database is VACUUMed and its WAL checkpointed, with measured reclaimed bytes. Definitions and versions are immutable and never touched. --db must name an existing database (default: the standard data directory); --artifacts defaults to artifacts/ beside --db. Works without a running server.",
|
||||
"search": "One read-only query over pipelines, versions, runs, steps, field text and artifacts. QUERY is free text, a bare status word (queued, running, succeeded, failed, canceled, interrupted) selecting those runs, or a lone date expression (today, yesterday, this week [Monday 00:00 local], or YYYY-MM-DD) selecting the runs created in that window. --type is the only narrowing mechanism and takes one kind: pipeline, version, run, step, field or artifact; an unknown kind is an error. Ranking: an exact name match (3) beats a name substring (2) beats a body-text match (1); equal tiers break by recency, then a stable kind/target tie-break, and every hit reports score, matched_field and recency so a client can re-rank. Bounds: the result cap defaults to 50 and clamps to 200 (0 = default); every kind scans at most 2000 rows and field-text scanning is additionally capped at 2000 steps / 1 MiB; Truncated plus scanned_steps, scanned_bytes and scan_bound_bytes report when a bound applied. Targets are stable: pipeline:<id>, pipeline:<id>:v<version>, run:<id>, run:<id>:<sequence>, run:<id>:<sequence>:<field>, or an artifact URL.",
|
||||
"resource-pool-configure": "Read a bounded full pool declaration from JSON and replace the declaration only after explicit --yes consent. Use '-' for stdin. This is local operator transport only; it never accepts or returns capabilities or credentials. resource-pool-inspect reads the declared members and status.",
|
||||
"version": "Identify the local CLI binary without connecting to a server. --version is an alias. Use ping --json to identify the running server separately.",
|
||||
"ping": "Pong includes the running server's compiled build.version, build.commit and build.dirty (null when unknown). This identifies the server, not the client or runtime workspace. Older servers omit build. The client compares its own build/protocol against the server: a same-protocol build mismatch prints one warning to stderr, a server speaking a newer protocol fails the command, and a response carrying a field this client does not model prints a one-time unrecognized-field notice instead of dropping it silently.",
|
||||
"connection": "Prints the server endpoint and phone notification endpoints. Like ping, it reports the client/server build relationship and warns or fails rather than silently proceeding against a mismatched server.",
|
||||
"status": "Reports server, host and ntfy publishing health. The client compares its build/protocol against the server and warns once (or fails on a newer protocol) rather than proceeding silently.",
|
||||
"export": "Without a name, select pipelines with Space and save a ZIP in the current directory. --output writes a portable definition bundle; --all skips selection. One name without --output prints Markdown as before. No runs, credentials or old versions are exported.",
|
||||
"import": "Select definitions from a ZIP. Identical existing definitions are skipped. For differing names, choose keep existing, keep both under a new name, or use imported as a new version (history retained). --on-conflict supplies an explicit automation policy; --yes only approves final publication. Schedules may activate; no run history is imported.",
|
||||
"new": "Create an optional template copy or a one-stage starter. Opens the configured default harness; --harness opencode|pi|nvim overrides it. No submission is performed.",
|
||||
"edit": "Omit the name to open fuzzy selection. Original immutable source is copied to a private draft. The configured harness edits that draft, never the live definition.",
|
||||
"validate": "Uses the same server parser as submission. Does not persist or activate schedules. Structural validity does not prove runtime/tool availability.",
|
||||
"submit": "Confirm publication at the prompt, or use --yes for explicit automation; schedules activate on acceptance. Managed drafts provide the name and base-version guard. --force never bypasses validation.",
|
||||
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. --model PROVIDER/MODEL overrides the model for every stage of that run; --stage-model STAGE=PROVIDER/MODEL overrides one stage and is repeatable, with a repeated key last-wins. Precedence is per-stage > whole-run > the model pinned in the pipeline version; overrides apply to that run only and never change the pinned definition. An unknown stage key or a malformed provider/model is refused before the run starts, naming the valid stage keys. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
|
||||
"retry": "Resume a failed stage by default, restoring the completed prefix handoff; use --mode=restart to start at entry. --stage selects a recorded stage (use --visit for repeated stages). --input-file accepts a bounded JSON object, merged with replacement at the selected stage; '-' reads stdin. Original history and pinned identity are retained; --version selects an explicit published version of the same pipeline only. --wait-timeout leaves an active retry for later read-only inspection or reattachment with the original --request-key, rather than starting a duplicate retry.",
|
||||
"title": "Rename a run's operator-facing title after the fact without touching its input, steps or execution. The title is collapsed to a single bounded line and recorded as explicitly set. `pipelines runs` and `pipelines inspect` then show it.",
|
||||
"wait": "Block until each named run reaches a terminal status - succeeded, failed, canceled or interrupted, and also through a run waiting on a person - then print one line per run with its id, status and outcome, its stage and how long it took, and exit 0 even for a run that finished unsuccessfully. Several ids wait together in one command. The exit is non-zero only for a client-side failure: an unknown run id, a connection that could not be established, or --timeout expiring while a run is still active; a timeout says so plainly, never cancels, retries or otherwise touches a run, and is never reported as its outcome. Read-only: it needs no workspace, is not refused by the workspace guard and writes nothing. --json prints the final run envelope instead of the line (one envelope for one id, an array for several).",
|
||||
"decide": "Choose an option for a run waiting on a human stage. The run must be `waiting`; the option must be one of that stage's non-error edges. The stage is inferred from the run's current stage unless named. The choice and optional --feedback are recorded as the step's evidence (visible in inspect and the TUI) and the run resumes along the chosen edge. An unknown option is refused with the valid list; a repeated decision for an already-decided visit does not double-apply. Works across a server restart because the wait is persisted.",
|
||||
"debug": "Open a fresh configured harness conversation to investigate a run using inspect/show/runs CLI queries. --harness opencode|pi|nvim chooses another configured harness or read-only nvim packet. Does not resume implementation or retry. --json returns the prompt without launching anything.",
|
||||
"rpc": "Accepts a JSON protocol envelope; '-' reads stdin. Every TUI server operation is available here too. Mutating commands are not dry runs. The omni-search command is `search` (fields query, optional kind and limit) and answers with ranked `search_results`.",
|
||||
"chat-adopt": "Adopt a session that already exists on the configured OpenCode endpoint so it appears in chat_sessions and can be continued. The id must be one the endpoint reports; its directory and title are taken from the endpoint's own record (never guessed), and a missing id records nothing. Adopting is always explicit and does not change the server's endpoint choice.",
|
||||
"retention": "Local, explicit run-history retention; never automatic and never a side effect of another command. `report` is strictly read-only and prints what a policy would remove; `apply` removes it after approval (--yes for scripted use). Policy: --older-than DURATION selects runs created before now minus that rolling UTC duration (Go durations plus d/w; not a calendar boundary), --keep N keeps the newest N runs by run id and selects older ranks (--keep 0 keeps none). Both together are the conservative intersection; at least one is required. Protected and never removed: queued/running runs, runs waiting on a person (waiting run or step), runs with an active cleanup agent, and any run referenced by a retained run's retry provenance (transitively). A run that becomes protected before deletion aborts the whole apply. Artifacts at <root>/<run> are removed with their run and kept with protected/retained runs; the database is VACUUMed and its WAL checkpointed, with measured reclaimed bytes. Definitions and versions are immutable and never touched. --db must name an existing database (default: the standard data directory); --artifacts defaults to artifacts/ beside --db. Works without a running server.",
|
||||
"search": "One read-only query over pipelines, versions, runs, steps, field text and artifacts. QUERY is free text, a bare status word (queued, running, succeeded, failed, canceled, interrupted) selecting those runs, or a lone date expression (today, yesterday, this week [Monday 00:00 local], or YYYY-MM-DD) selecting the runs created in that window. --type is the only narrowing mechanism and takes one kind: pipeline, version, run, step, field or artifact; an unknown kind is an error. Ranking: an exact name match (3) beats a name substring (2) beats a body-text match (1); equal tiers break by recency, then a stable kind/target tie-break, and every hit reports score, matched_field and recency so a client can re-rank. Bounds: the result cap defaults to 50 and clamps to 200 (0 = default); every kind scans at most 2000 rows and field-text scanning is additionally capped at 2000 steps / 1 MiB; Truncated plus scanned_steps, scanned_bytes and scan_bound_bytes report when a bound applied. Targets are stable: pipeline:<id>, pipeline:<id>:v<version>, run:<id>, run:<id>:<sequence>, run:<id>:<sequence>:<field>, or an artifact URL.",
|
||||
}
|
||||
|
||||
var serviceHelp = map[string]string{
|
||||
|
||||
@@ -44,12 +44,9 @@ func decodeControl(text string, value any) error {
|
||||
}
|
||||
|
||||
func parseControl(s *Stage, fields map[string]string) error {
|
||||
if text, ok := fields["resources"]; ok {
|
||||
if err := decodeControl(text, &s.Resources); err != nil || s.Resources == nil || !validStageResources(*s.Resources) {
|
||||
return fmt.Errorf("resources must contain exactly one exclusive or pool key (1–128 ASCII letters, digits, _ or -; start with a letter)")
|
||||
}
|
||||
delete(fields, "resources")
|
||||
}
|
||||
// A resources block from a definition created before the feature was removed
|
||||
// is ignored: it no longer declares scheduling or validation.
|
||||
delete(fields, "resources")
|
||||
if text, ok := fields["limits"]; ok {
|
||||
if err := decodeControl(text, &s.Limits); err != nil || s.Limits == nil {
|
||||
return fmt.Errorf("limits must be an object with max_visits and optional on outcomes")
|
||||
@@ -81,15 +78,7 @@ func parseControl(s *Stage, fields map[string]string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validStageResources(resources protocol.StageResources) bool {
|
||||
return (ValidResourceKey(resources.Exclusive) && resources.Pool == "") ||
|
||||
(resources.Exclusive == "" && ValidResourceKey(resources.Pool))
|
||||
}
|
||||
|
||||
func validateControl(s Stage, stages map[string]bool) error {
|
||||
if s.Resources != nil && s.Type == "human" {
|
||||
return fmt.Errorf("stage %s: human stages cannot declare resources", s.Key)
|
||||
}
|
||||
if s.Limits != nil {
|
||||
for _, outcome := range s.Limits.On {
|
||||
target, exists := s.Edges[outcome]
|
||||
@@ -109,7 +98,3 @@ func validateControl(s Stage, stages map[string]bool) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidResourceKey(key string) bool {
|
||||
return len(key) <= 128 && identifier.MatchString(key)
|
||||
}
|
||||
|
||||
@@ -6,6 +6,19 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRetiredResourcesBlockIsIgnored(t *testing.T) {
|
||||
source := Template([]string{"worker"}, true) + "```resources\n{\"exclusive\":\"phone\"}\n```\n"
|
||||
graph, err := Parse(source)
|
||||
if err != nil {
|
||||
t.Fatalf("resources block was not ignored: %v", err)
|
||||
}
|
||||
for _, stage := range graph.Stages {
|
||||
if _, exists := stage.Blocks["resources"]; exists {
|
||||
t.Fatalf("retired resources block was retained on stage %s", stage.Key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidation(t *testing.T) {
|
||||
valid := Template([]string{"implementer", "reviewer"}, true)
|
||||
cycle := strings.Replace(valid, "success -> #success", "bad_implementation -> implementer\nsuccess -> #success", 1)
|
||||
|
||||
@@ -1,759 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
const maxRecoveryText = 512
|
||||
|
||||
func boundedRecoveryText(value string) string {
|
||||
value = strings.TrimSpace(value)
|
||||
return value
|
||||
}
|
||||
|
||||
// InspectResource exposes only bounded state; capability tokens never leave this
|
||||
// package or enter a protocol value.
|
||||
func (s *Store) InspectResource(ctx context.Context, resource string) (protocol.ResourceInspection, error) {
|
||||
if !ValidResourceKey(resource) {
|
||||
return protocol.ResourceInspection{}, fmt.Errorf("invalid resource")
|
||||
}
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
out := protocol.ResourceInspection{Resource: resource, Waiters: []protocol.ResourceWaiter{}, Audits: []protocol.ResourceRecoveryAudit{}}
|
||||
var runID int64
|
||||
var stage string
|
||||
var seq, cleanup int
|
||||
err := s.db.QueryRowContext(ctx, `SELECT run_id,stage_key,sequence,cleanup_confirmed FROM resource_owners WHERE resource_key=?`, resource).Scan(&runID, &stage, &seq, &cleanup)
|
||||
if err == nil {
|
||||
var active int
|
||||
if err := s.db.QueryRowContext(ctx, `SELECT COALESCE(MAX(agent_active),0) FROM run_steps WHERE run_id=? AND sequence=?`, runID, seq).Scan(&active); err != nil {
|
||||
return out, err
|
||||
}
|
||||
out.Owner = &protocol.ResourceOwner{RunID: runID, Stage: stage, Sequence: seq, CleanupConfirmed: cleanup != 0, Quarantined: cleanup == 0, AgentActive: active != 0}
|
||||
} else if err != sql.ErrNoRows {
|
||||
return out, err
|
||||
}
|
||||
rows, err := s.db.QueryContext(ctx, `SELECT run_id,stage_key,sequence,queued_at FROM resource_waiters WHERE resource_key=? ORDER BY queue_order LIMIT 32`, resource)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
for rows.Next() {
|
||||
var w protocol.ResourceWaiter
|
||||
if err := rows.Scan(&w.RunID, &w.Stage, &w.Sequence, &w.QueuedAt); err != nil {
|
||||
rows.Close()
|
||||
return out, err
|
||||
}
|
||||
out.Waiters = append(out.Waiters, w)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
rows.Close()
|
||||
return out, err
|
||||
}
|
||||
rows.Close()
|
||||
audits, err := s.db.QueryContext(ctx, `SELECT run_id,stage_key,sequence,actor,source,evidence,result,created_at FROM resource_recovery_audit WHERE resource_key=? ORDER BY rowid DESC LIMIT 16`, resource)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
defer audits.Close()
|
||||
for audits.Next() {
|
||||
var a protocol.ResourceRecoveryAudit
|
||||
a.Resource = resource
|
||||
if err := audits.Scan(&a.RunID, &a.Stage, &a.Sequence, &a.Actor, &a.Source, &a.Evidence, &a.Result, &a.CreatedAt); err != nil {
|
||||
audits.Close()
|
||||
return out, err
|
||||
}
|
||||
out.Audits = append(out.Audits, a)
|
||||
}
|
||||
return out, audits.Err()
|
||||
}
|
||||
|
||||
// ReconcileResource is the sole operator recovery mutation. The persisted
|
||||
// capability is compared only inside this authoritative store operation.
|
||||
func (s *Store) ReconcileResource(ctx context.Context, request protocol.ResourceRecoveryRequest) error {
|
||||
if !ValidResourceKey(request.Resource) || request.RunID <= 0 || request.Sequence <= 0 || !identifier.MatchString(request.Stage) {
|
||||
return fmt.Errorf("invalid resource recovery identity")
|
||||
}
|
||||
if request.Evidence == nil || !request.Evidence.CleanupConfirmed || !validRecoveryEvidence(request.Evidence.Summary) {
|
||||
return fmt.Errorf("positive cleanup/restoration evidence is required")
|
||||
}
|
||||
if strings.TrimSpace(request.Actor) == "" || len(request.Actor) > maxRecoveryText {
|
||||
return fmt.Errorf("bounded actor is required")
|
||||
}
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
var token, stage string
|
||||
var runID int64
|
||||
var seq, cleanup int
|
||||
err = tx.QueryRowContext(ctx, `SELECT run_id,stage_key,sequence,token,cleanup_confirmed FROM resource_owners WHERE resource_key=?`, request.Resource).Scan(&runID, &stage, &seq, &token, &cleanup)
|
||||
if err == sql.ErrNoRows {
|
||||
var count int
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND result='reconciled'`, request.Resource, request.RunID, request.Stage, request.Sequence).Scan(&count); err != nil {
|
||||
return err
|
||||
}
|
||||
if count > 0 {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("resource owner not found")
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if runID != request.RunID || stage != request.Stage || seq != request.Sequence {
|
||||
return fmt.Errorf("resource owner visit is stale")
|
||||
}
|
||||
if cleanup != 0 {
|
||||
return fmt.Errorf("resource owner is not quarantined")
|
||||
}
|
||||
var status string
|
||||
if err := tx.QueryRowContext(ctx, `SELECT status FROM pipeline_runs WHERE id=?`, runID).Scan(&status); err != nil {
|
||||
return err
|
||||
}
|
||||
if status == protocol.RunStatusQueued || status == protocol.RunStatusWaiting || status == protocol.RunStatusRunning {
|
||||
return fmt.Errorf("owning run is still active")
|
||||
}
|
||||
var active int
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COALESCE(MAX(agent_active),0) FROM run_steps WHERE run_id=? AND sequence=?`, runID, seq).Scan(&active); err != nil {
|
||||
return err
|
||||
}
|
||||
if active != 0 {
|
||||
return fmt.Errorf("owning agent is still active")
|
||||
}
|
||||
evidence := boundedRecoveryText(request.Evidence.Summary)
|
||||
if request.Evidence.NoMutatingAction {
|
||||
evidence = boundedRecoveryText("no mutating action occurred; " + evidence)
|
||||
}
|
||||
evidenceHash := sha256.Sum256([]byte(evidence))
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_recovery_audit(resource_key,run_id,stage_key,sequence,actor,source,evidence,result) VALUES(?,?,?,?,?,?,?,?)`, request.Resource, runID, stage, seq, boundedRecoveryText(request.Actor), "loopback-operator", "sha256:"+hex.EncodeToString(evidenceHash[:]), "reconciled"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE resource_owners SET cleanup_confirmed=1,diagnostic='' WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND token=? AND cleanup_confirmed=0`, request.Resource, runID, stage, seq, token); err != nil {
|
||||
return err
|
||||
}
|
||||
owner := ResourceLease{ResourceVisit: ResourceVisit{Resource: request.Resource, RunID: runID, Stage: stage, Sequence: seq}, token: token}
|
||||
if _, err := deleteResourceOwner(ctx, tx, owner); err != nil {
|
||||
return err
|
||||
}
|
||||
next, err := s.handoffResourceTx(ctx, tx, request.Resource)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
if next.token != "" {
|
||||
s.resourceWaiters[next.ResourceVisit] <- next
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validRecoveryEvidence(value string) bool {
|
||||
value = strings.TrimSpace(value)
|
||||
if value == "" || len(value) > maxRecoveryText || !utf8.ValidString(value) {
|
||||
return false
|
||||
}
|
||||
if strings.EqualFold(value, "true") || strings.EqualFold(value, "false") || strings.EqualFold(value, "null") {
|
||||
return false
|
||||
}
|
||||
if strings.HasPrefix(value, "{") || strings.HasPrefix(value, "[") {
|
||||
return false
|
||||
}
|
||||
for _, r := range value {
|
||||
if r == '\x00' || (r < 0x20 && r != '\t' && r != '\n' && r != '\r') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ResourceVisit identifies a pending visit, before it consumes execution accounting.
|
||||
type ResourceVisit struct {
|
||||
Resource string
|
||||
RunID int64
|
||||
Stage string
|
||||
Sequence int
|
||||
}
|
||||
|
||||
// ResourceLease is an opaque capability. Its credential is deliberately not exported
|
||||
// or serializable, and must never be put in model input or public wire metadata.
|
||||
type ResourceLease struct {
|
||||
ResourceVisit
|
||||
Target *protocol.ResourceTarget
|
||||
pool string
|
||||
token string
|
||||
}
|
||||
|
||||
const maxPoolMembers = 32
|
||||
const maxTargetMetadata = 16
|
||||
const maxTargetMetadataKey = 128
|
||||
const maxTargetValue = 128
|
||||
|
||||
func validateTargetMetadata(metadata map[string]string) error {
|
||||
if len(metadata) > maxTargetMetadata {
|
||||
return fmt.Errorf("too many target metadata fields")
|
||||
}
|
||||
for k, v := range metadata {
|
||||
if !identifier.MatchString(k) || len(k) > maxTargetMetadataKey || len(v) > maxTargetValue || !utf8.ValidString(v) {
|
||||
return fmt.Errorf("invalid target metadata")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateResourcePool(name string, members []protocol.ResourcePoolMember) error {
|
||||
if !ValidResourceKey(name) || len(members) == 0 || len(members) > maxPoolMembers {
|
||||
return fmt.Errorf("invalid resource pool")
|
||||
}
|
||||
seen := make(map[string]bool, len(members))
|
||||
for i := range members {
|
||||
member := &members[i]
|
||||
if !ValidResourceKey(member.Member) || seen[member.Member] {
|
||||
return fmt.Errorf("invalid pool member")
|
||||
}
|
||||
if err := validateTargetMetadata(member.Metadata); err != nil {
|
||||
return err
|
||||
}
|
||||
seen[member.Member] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ConfigureResourcePool replaces a pool declaration atomically. Existing
|
||||
// owners keep their concrete member; queued requests are evaluated against the
|
||||
// new declaration on the next release and are never silently transferred.
|
||||
func (s *Store) ConfigureResourcePool(ctx context.Context, pool protocol.ResourcePool) error {
|
||||
if err := validateResourcePool(pool.Name, pool.Members); err != nil {
|
||||
return err
|
||||
}
|
||||
encoded, err := json.Marshal(pool.Members)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
_, err = s.db.ExecContext(ctx, `INSERT INTO resource_pools(name,members_json) VALUES(?,?) ON CONFLICT(name) DO UPDATE SET members_json=excluded.members_json,updated_at=CURRENT_TIMESTAMP`, pool.Name, string(encoded))
|
||||
return err
|
||||
}
|
||||
|
||||
// InspectResourcePool returns the declared pool, including only bounded,
|
||||
// non-secret member metadata and status.
|
||||
func (s *Store) InspectResourcePool(ctx context.Context, name string) (protocol.ResourcePool, error) {
|
||||
if !ValidResourceKey(name) {
|
||||
return protocol.ResourcePool{}, fmt.Errorf("invalid resource pool")
|
||||
}
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
members, err := s.resourcePool(ctx, s.db, name)
|
||||
if err != nil {
|
||||
return protocol.ResourcePool{}, err
|
||||
}
|
||||
return protocol.ResourcePool{Name: name, Members: members}, nil
|
||||
}
|
||||
|
||||
func (s *Store) resourcePool(ctx context.Context, tx interface {
|
||||
QueryRowContext(context.Context, string, ...any) *sql.Row
|
||||
}, name string) ([]protocol.ResourcePoolMember, error) {
|
||||
var encoded string
|
||||
if err := tx.QueryRowContext(ctx, "SELECT members_json FROM resource_pools WHERE name=?", name).Scan(&encoded); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var members []protocol.ResourcePoolMember
|
||||
if err := json.Unmarshal([]byte(encoded), &members); err != nil {
|
||||
return nil, fmt.Errorf("invalid persisted pool: %w", err)
|
||||
}
|
||||
if err := validateResourcePool(name, members); err != nil {
|
||||
return nil, fmt.Errorf("invalid persisted pool: %w", err)
|
||||
}
|
||||
return members, nil
|
||||
}
|
||||
|
||||
func poolTarget(member protocol.ResourcePoolMember) *protocol.ResourceTarget {
|
||||
metadata := make(map[string]string, len(member.Metadata))
|
||||
for k, v := range member.Metadata {
|
||||
metadata[k] = v
|
||||
}
|
||||
return &protocol.ResourceTarget{Member: member.Member, Metadata: metadata}
|
||||
}
|
||||
|
||||
// RequestResourcePool selects one free eligible concrete member while holding
|
||||
// the same authority as direct resource requests.
|
||||
func (s *Store) RequestResourcePool(ctx context.Context, pool string, visit ResourceVisit) (ResourceLease, <-chan ResourceLease, *protocol.ResourceWait, error) {
|
||||
if !ValidResourceKey(pool) || visit.Resource != pool || visit.RunID <= 0 || visit.Sequence <= 0 || !identifier.MatchString(visit.Stage) {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("invalid resource pool visit")
|
||||
}
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
if ch := s.resourceWaiters[visit]; ch != nil {
|
||||
wait, err := s.resourceWait(ctx, visit.RunID)
|
||||
return ResourceLease{}, ch, wait, err
|
||||
}
|
||||
credential := make([]byte, 32)
|
||||
if _, err := rand.Read(credential); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
lease := ResourceLease{ResourceVisit: visit, token: hex.EncodeToString(credential)}
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
var status string
|
||||
if err := tx.QueryRowContext(ctx, "SELECT status FROM pipeline_runs WHERE id=?", visit.RunID).Scan(&status); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if status != protocol.RunStatusQueued && status != protocol.RunStatusRunning {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("run is not active")
|
||||
}
|
||||
var existing ResourceLease
|
||||
var existingPool string
|
||||
err = tx.QueryRowContext(ctx, "SELECT resource_key,run_id,stage_key,sequence,token,pool_name FROM resource_owners WHERE run_id=?", visit.RunID).Scan(&existing.Resource, &existing.RunID, &existing.Stage, &existing.Sequence, &existing.token, &existingPool)
|
||||
if err == nil {
|
||||
if existing.RunID != visit.RunID || existing.Stage != visit.Stage || existing.Sequence != visit.Sequence {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("resource owner visit is stale")
|
||||
}
|
||||
if existingPool != pool {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("resource owner pool visit is stale")
|
||||
}
|
||||
members, poolErr := s.resourcePool(ctx, tx, pool)
|
||||
if poolErr != nil {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("unknown resource pool: %w", poolErr)
|
||||
}
|
||||
for _, member := range members {
|
||||
if member.Member == existing.Resource {
|
||||
existing.ResourceVisit = visit
|
||||
existing.pool = pool
|
||||
existing.Target = poolTarget(member)
|
||||
return existing, nil, nil, nil
|
||||
}
|
||||
}
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("run already owns another resource visit")
|
||||
}
|
||||
if err != sql.ErrNoRows {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
// A queued visit may be re-entered after the process that owned its live
|
||||
// channel was restarted. Reattach to the durable waiter instead of trying
|
||||
// to insert a second reservation.
|
||||
var persistedPool string
|
||||
err = tx.QueryRowContext(ctx, "SELECT pool_name FROM resource_waiters WHERE run_id=? AND stage_key=? AND sequence=?", visit.RunID, visit.Stage, visit.Sequence).Scan(&persistedPool)
|
||||
if err == nil {
|
||||
if persistedPool != pool {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("resource waiter pool visit is stale")
|
||||
}
|
||||
wait, waitErr := s.resourceWait(ctx, visit.RunID)
|
||||
if waitErr != nil {
|
||||
return ResourceLease{}, nil, nil, waitErr
|
||||
}
|
||||
if s.resourceWaiters == nil {
|
||||
s.resourceWaiters = make(map[ResourceVisit]chan ResourceLease)
|
||||
}
|
||||
ch := make(chan ResourceLease, 1)
|
||||
s.resourceWaiters[visit] = ch
|
||||
return ResourceLease{}, ch, wait, nil
|
||||
}
|
||||
if err != sql.ErrNoRows {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
members, err := s.resourcePool(ctx, tx, pool)
|
||||
if err != nil {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("unknown resource pool: %w", err)
|
||||
}
|
||||
for _, member := range members {
|
||||
if !member.Enabled || member.Quarantined {
|
||||
continue
|
||||
}
|
||||
var count int
|
||||
if err := tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", member.Member).Scan(&count); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if count != 0 {
|
||||
continue
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_owners(resource_key,run_id,stage_key,sequence,token,pool_name) VALUES(?,?,?,?,?,?)`, member.Member, visit.RunID, visit.Stage, visit.Sequence, lease.token, pool); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
lease.Resource = member.Member
|
||||
lease.pool = pool
|
||||
lease.Target = poolTarget(member)
|
||||
if err := tx.Commit(); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
return lease, nil, nil, nil
|
||||
}
|
||||
wait := &protocol.ResourceWait{Resource: pool, Stage: visit.Stage, Sequence: visit.Sequence, QueuedSince: time.Now().UTC().Format(time.RFC3339Nano)}
|
||||
encoded, err := json.Marshal(wait)
|
||||
if err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_waiters(resource_key,pool_name,run_id,stage_key,sequence,token,queued_at) VALUES(?,?,?,?,?,?,?)`, pool, pool, visit.RunID, visit.Stage, visit.Sequence, lease.token, wait.QueuedSince); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='queued',current_stage=?,resource_wait=? WHERE id=? AND status IN ('queued','running')", visit.Stage, string(encoded), visit.RunID)); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if s.resourceWaiters == nil {
|
||||
s.resourceWaiters = make(map[ResourceVisit]chan ResourceLease)
|
||||
}
|
||||
ch := make(chan ResourceLease, 1)
|
||||
s.resourceWaiters[visit] = ch
|
||||
return ResourceLease{}, ch, wait, nil
|
||||
}
|
||||
|
||||
// RequestResource acquires immediately or registers one FIFO waiter/channel. The
|
||||
// mutex covers only short DB transactions and channel bookkeeping, never execution.
|
||||
func (s *Store) RequestResource(ctx context.Context, visit ResourceVisit) (ResourceLease, <-chan ResourceLease, *protocol.ResourceWait, error) {
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
if !ValidResourceKey(visit.Resource) || visit.RunID <= 0 || visit.Sequence <= 0 || !identifier.MatchString(visit.Stage) {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("invalid resource visit")
|
||||
}
|
||||
if ch := s.resourceWaiters[visit]; ch != nil {
|
||||
wait, err := s.resourceWait(ctx, visit.RunID)
|
||||
return ResourceLease{}, ch, wait, err
|
||||
}
|
||||
credential := make([]byte, 32)
|
||||
if _, err := rand.Read(credential); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
lease := ResourceLease{ResourceVisit: visit, token: hex.EncodeToString(credential)}
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
var status string
|
||||
if err := tx.QueryRowContext(ctx, "SELECT status FROM pipeline_runs WHERE id=?", visit.RunID).Scan(&status); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if status != protocol.RunStatusQueued && status != protocol.RunStatusRunning {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("run is not active")
|
||||
}
|
||||
var existing ResourceLease
|
||||
var existingPool string
|
||||
err = tx.QueryRowContext(ctx, "SELECT resource_key,run_id,stage_key,sequence,token,pool_name FROM resource_owners WHERE run_id=?", visit.RunID).Scan(&existing.Resource, &existing.RunID, &existing.Stage, &existing.Sequence, &existing.token, &existingPool)
|
||||
if err == nil {
|
||||
if existingPool != "" {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("run already owns a pooled resource visit")
|
||||
}
|
||||
if existing.ResourceVisit != visit {
|
||||
return ResourceLease{}, nil, nil, fmt.Errorf("run already owns another resource visit")
|
||||
}
|
||||
return existing, nil, nil, nil
|
||||
}
|
||||
if err != sql.ErrNoRows {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
result, err := tx.ExecContext(ctx, `INSERT INTO resource_owners(resource_key,run_id,stage_key,sequence,token,pool_name)
|
||||
VALUES(?,?,?,?,?,?) ON CONFLICT(resource_key) DO NOTHING`, visit.Resource, visit.RunID, visit.Stage, visit.Sequence, lease.token, "")
|
||||
if err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
n, err := result.RowsAffected()
|
||||
if err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if n == 1 {
|
||||
return lease, nil, nil, tx.Commit()
|
||||
}
|
||||
wait := &protocol.ResourceWait{Resource: visit.Resource, Stage: visit.Stage, Sequence: visit.Sequence, QueuedSince: time.Now().UTC().Format(time.RFC3339Nano)}
|
||||
encoded, err := json.Marshal(wait)
|
||||
if err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_waiters(resource_key,run_id,stage_key,sequence,token,queued_at) VALUES(?,?,?,?,?,?)`, visit.Resource, visit.RunID, visit.Stage, visit.Sequence, lease.token, wait.QueuedSince); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='queued',current_stage=?,resource_wait=? WHERE id=? AND status IN ('queued','running')", visit.Stage, string(encoded), visit.RunID)); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return ResourceLease{}, nil, nil, err
|
||||
}
|
||||
if s.resourceWaiters == nil {
|
||||
s.resourceWaiters = make(map[ResourceVisit]chan ResourceLease)
|
||||
}
|
||||
ch := make(chan ResourceLease, 1)
|
||||
s.resourceWaiters[visit] = ch
|
||||
return ResourceLease{}, ch, wait, nil
|
||||
}
|
||||
|
||||
func (s *Store) resourceWait(ctx context.Context, id int64) (*protocol.ResourceWait, error) {
|
||||
var wait protocol.ResourceWait
|
||||
err := s.db.QueryRowContext(ctx, "SELECT resource_key,stage_key,sequence,queued_at FROM resource_waiters WHERE run_id=?", id).Scan(&wait.Resource, &wait.Stage, &wait.Sequence, &wait.QueuedSince)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
return &wait, err
|
||||
}
|
||||
|
||||
// CancelResource removes a queued visit, or returns a concurrently reserved grant
|
||||
// to the queue before any execution is allowed. It cannot cancel another visit.
|
||||
func (s *Store) CancelResource(ctx context.Context, visit ResourceVisit) error {
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
defer delete(s.resourceWaiters, visit)
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
deleted, err := tx.ExecContext(ctx, "DELETE FROM resource_waiters WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=?", visit.Resource, visit.RunID, visit.Stage, visit.Sequence)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
n, err := deleted.RowsAffected()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n > 0 {
|
||||
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET resource_wait='' WHERE id=?", visit.RunID); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
ch := s.resourceWaiters[visit]
|
||||
// A grant is buffered while this registry entry exists. Cancellation can
|
||||
// reclaim only that not-yet-accepted capability, never an executing owner.
|
||||
var grant ResourceLease
|
||||
if ch != nil {
|
||||
select {
|
||||
case grant = <-ch:
|
||||
default:
|
||||
}
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
if grant.token != "" {
|
||||
if err := s.SetResourceCleanup(ctx, grant, true); err != nil {
|
||||
return err
|
||||
}
|
||||
return s.releaseResourceLocked(ctx, grant)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// AcceptResource removes the channel exactly once. Cancellation is rechecked by
|
||||
// the runner before BeginStep; a canceled accepted grant is released as unused.
|
||||
func (s *Store) AcceptResource(lease ResourceLease) {
|
||||
s.resourceMu.Lock()
|
||||
delete(s.resourceWaiters, lease.ResourceVisit)
|
||||
s.resourceMu.Unlock()
|
||||
}
|
||||
|
||||
// HasResourceOwner is used only for orphan/restart reconciliation, never for a
|
||||
// resource-free stage visit.
|
||||
func (s *Store) HasResourceOwner(ctx context.Context, runID int64) (bool, error) {
|
||||
var exists bool
|
||||
err := s.db.QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM resource_owners WHERE run_id=?)", runID).Scan(&exists)
|
||||
return exists, err
|
||||
}
|
||||
|
||||
func (s *Store) ConfirmRecoveredSession(ctx context.Context, owner SessionOwner) error {
|
||||
_, err := s.db.ExecContext(ctx, "UPDATE run_steps SET agent_active=0 WHERE run_id=? AND session_id=?", owner.RunID, owner.ID)
|
||||
return err
|
||||
}
|
||||
|
||||
// SetResourceCleanup records positive quiescence durably before release. False
|
||||
// quarantines the owner; neither age nor another run can remove the lease.
|
||||
func (s *Store) SetResourceCleanup(ctx context.Context, lease ResourceLease, confirmed bool) error {
|
||||
if lease.pool != "" && (lease.Target == nil || !ValidResourceKey(lease.Target.Member)) {
|
||||
return fmt.Errorf("invalid pooled resource lease")
|
||||
}
|
||||
diagnostic := ""
|
||||
if !confirmed {
|
||||
diagnostic = "Cleanup/restoration unconfirmed; resource quarantined pending inspected reconciliation"
|
||||
}
|
||||
resource := lease.Resource
|
||||
if lease.pool != "" {
|
||||
resource = lease.Target.Member
|
||||
}
|
||||
return affected(s.db.ExecContext(ctx, `UPDATE resource_owners SET cleanup_confirmed=?,diagnostic=? WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND token=?`, confirmed, diagnostic, resource, lease.RunID, lease.Stage, lease.Sequence, lease.token))
|
||||
}
|
||||
|
||||
// ReleaseResource requires the complete capability AND positive cleanup evidence.
|
||||
// Mismatches and repeated releases are harmless, including after transfer.
|
||||
func (s *Store) ReleaseResource(ctx context.Context, lease ResourceLease) error {
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
return s.releaseResourceLocked(ctx, lease)
|
||||
}
|
||||
|
||||
func (s *Store) releaseResourceLocked(ctx context.Context, lease ResourceLease) error {
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
ownerResource := lease.Resource
|
||||
if lease.pool != "" {
|
||||
if lease.Target == nil || !ValidResourceKey(lease.Target.Member) {
|
||||
return fmt.Errorf("invalid pooled resource lease")
|
||||
}
|
||||
ownerResource = lease.Target.Member
|
||||
}
|
||||
lease.Resource = ownerResource
|
||||
n, err := deleteResourceOwner(ctx, tx, lease)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n == 0 {
|
||||
return nil
|
||||
}
|
||||
next, err := s.handoffResourceTx(ctx, tx, ownerResource)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return err
|
||||
}
|
||||
if next.token != "" {
|
||||
s.resourceWaiters[next.ResourceVisit] <- next
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// handoffResourceTx is shared by ordinary release and inspected recovery. It
|
||||
// validates the in-memory waiter before changing durable ownership so a stale
|
||||
// server cannot grant a capability to a runner that no longer exists.
|
||||
func (s *Store) handoffResourceTx(ctx context.Context, tx *sql.Tx, resource string) (ResourceLease, error) {
|
||||
var next ResourceLease
|
||||
next.Resource = resource
|
||||
rows, err := tx.QueryContext(ctx, `SELECT queue_order,resource_key,pool_name,run_id,stage_key,sequence,token FROM resource_waiters WHERE resource_key=? OR pool_name<>'' ORDER BY queue_order`, resource)
|
||||
if err != nil {
|
||||
return ResourceLease{}, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var selected *protocol.ResourceTarget
|
||||
var queue int64
|
||||
var waiterResource string
|
||||
for rows.Next() {
|
||||
var candidate ResourceLease
|
||||
var candidatePool string
|
||||
if err := rows.Scan(&queue, &waiterResource, &candidatePool, &candidate.RunID, &candidate.Stage, &candidate.Sequence, &candidate.token); err != nil {
|
||||
return ResourceLease{}, err
|
||||
}
|
||||
if candidatePool != "" {
|
||||
candidate.Resource = candidatePool
|
||||
candidate.pool = candidatePool
|
||||
} else {
|
||||
candidate.Resource = resource
|
||||
}
|
||||
if candidatePool == "" {
|
||||
if waiterResource != resource {
|
||||
continue
|
||||
}
|
||||
selected = &protocol.ResourceTarget{Member: resource}
|
||||
} else {
|
||||
members, e := s.resourcePool(ctx, tx, candidatePool)
|
||||
if e != nil {
|
||||
continue
|
||||
}
|
||||
found := false
|
||||
for _, member := range members {
|
||||
if !member.Enabled || member.Quarantined {
|
||||
continue
|
||||
}
|
||||
var count int
|
||||
if e = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", member.Member).Scan(&count); e != nil {
|
||||
return ResourceLease{}, e
|
||||
}
|
||||
if count == 0 {
|
||||
selected, found = poolTarget(member), true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if s.resourceWaiters[candidate.ResourceVisit] == nil {
|
||||
if candidatePool == "" {
|
||||
return ResourceLease{}, fmt.Errorf("resource waiter has no live runner; refusing handoff")
|
||||
}
|
||||
continue
|
||||
}
|
||||
next = candidate
|
||||
next.Target = selected
|
||||
break
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return ResourceLease{}, err
|
||||
}
|
||||
if next.token == "" {
|
||||
return ResourceLease{}, nil
|
||||
}
|
||||
ownerResource := next.Resource
|
||||
if next.pool != "" {
|
||||
ownerResource = next.Target.Member
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO resource_owners(resource_key,run_id,stage_key,sequence,token,pool_name) VALUES(?,?,?,?,?,?)`, ownerResource, next.RunID, next.Stage, next.Sequence, next.token, next.pool); err != nil {
|
||||
return ResourceLease{}, err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `DELETE FROM resource_waiters WHERE queue_order=?`, queue); err != nil {
|
||||
return ResourceLease{}, err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET resource_wait='' WHERE id=?", next.RunID); err != nil {
|
||||
return ResourceLease{}, err
|
||||
}
|
||||
if next.pool == "" {
|
||||
next.Resource = ownerResource
|
||||
}
|
||||
return next, nil
|
||||
}
|
||||
|
||||
func deleteResourceOwner(ctx context.Context, tx *sql.Tx, lease ResourceLease) (int64, error) {
|
||||
result, err := tx.ExecContext(ctx, `DELETE FROM resource_owners WHERE resource_key=? AND run_id=? AND stage_key=? AND sequence=? AND token=? AND cleanup_confirmed=1`, lease.Resource, lease.RunID, lease.Stage, lease.Sequence, lease.token)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return result.RowsAffected()
|
||||
}
|
||||
|
||||
// recoverResourceOwners runs only at startup, after pending waiters have been
|
||||
// interrupted and removed. Every reclamation uses the persisted capability;
|
||||
// absent cleanup evidence is never replaced with a timestamp-based expiry.
|
||||
func recoverResourceOwners(ctx context.Context, tx *sql.Tx) error {
|
||||
rows, err := tx.QueryContext(ctx, "SELECT resource_key,run_id,stage_key,sequence,token FROM resource_owners WHERE cleanup_confirmed=1")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var leases []ResourceLease
|
||||
for rows.Next() {
|
||||
var lease ResourceLease
|
||||
if err := rows.Scan(&lease.Resource, &lease.RunID, &lease.Stage, &lease.Sequence, &lease.token); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
leases = append(leases, lease)
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, lease := range leases {
|
||||
if _, err := deleteResourceOwner(ctx, tx, lease); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
_, err = tx.ExecContext(ctx, "UPDATE resource_owners SET diagnostic='Server stopped; resource quarantined pending inspected cleanup/restoration'")
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestOpenDropsRetiredResourceSchema proves an existing database that still
|
||||
// carries the removed pool/ownership/wait-queue tables and the run queue
|
||||
// snapshot column opens cleanly, losing only that retired state. A second open
|
||||
// proves the removal is idempotent.
|
||||
func TestOpenDropsRetiredResourceSchema(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "legacy-resources.db")
|
||||
store, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := store.db.Exec(`ALTER TABLE pipeline_runs ADD COLUMN resource_wait TEXT NOT NULL DEFAULT '';
|
||||
CREATE TABLE resource_owners (
|
||||
resource_key TEXT PRIMARY KEY, run_id INTEGER NOT NULL REFERENCES pipeline_runs(id),
|
||||
stage_key TEXT NOT NULL, sequence INTEGER NOT NULL, token TEXT NOT NULL,
|
||||
cleanup_confirmed INTEGER NOT NULL DEFAULT 0, diagnostic TEXT NOT NULL DEFAULT '');
|
||||
CREATE TABLE resource_waiters (
|
||||
queue_order INTEGER PRIMARY KEY AUTOINCREMENT, resource_key TEXT NOT NULL,
|
||||
run_id INTEGER NOT NULL UNIQUE REFERENCES pipeline_runs(id), stage_key TEXT NOT NULL,
|
||||
sequence INTEGER NOT NULL, token TEXT NOT NULL, queued_at TEXT NOT NULL);
|
||||
CREATE TABLE resource_pools (
|
||||
name TEXT PRIMARY KEY, members_json TEXT NOT NULL, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);
|
||||
CREATE TABLE resource_recovery_audit (
|
||||
resource_key TEXT NOT NULL, run_id INTEGER NOT NULL, stage_key TEXT NOT NULL,
|
||||
sequence INTEGER NOT NULL, actor TEXT NOT NULL, source TEXT NOT NULL,
|
||||
evidence TEXT NOT NULL, result TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := store.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
for attempt := 0; attempt < 2; attempt++ {
|
||||
store, err = Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open existing database (attempt %d): %v", attempt+1, err)
|
||||
}
|
||||
if runs, err := store.ListRuns(context.Background()); err != nil || len(runs) != 0 {
|
||||
t.Fatalf("list runs after migration (attempt %d): %+v %v", attempt+1, runs, err)
|
||||
}
|
||||
for _, table := range []string{"resource_owners", "resource_waiters", "resource_pools", "resource_recovery_audit"} {
|
||||
var name string
|
||||
err := store.db.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name=?", table).Scan(&name)
|
||||
if err == nil {
|
||||
t.Fatalf("retired table %s survived migration", table)
|
||||
}
|
||||
}
|
||||
var count int
|
||||
if err := store.db.QueryRow("SELECT COUNT(*) FROM pragma_table_info('pipeline_runs') WHERE name='resource_wait'").Scan(&count); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if count != 0 {
|
||||
t.Fatal("retired resource_wait column survived migration")
|
||||
}
|
||||
if err := store.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,779 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
func TestResourceDeclaration(t *testing.T) {
|
||||
base := Template([]string{"work"}, true)
|
||||
for _, value := range []string{`{"exclusive":"phone-X"}`, `{ "exclusive" : "A_1" }`} {
|
||||
graph, err := Parse(base + "```resources\n" + value + "\n```\n")
|
||||
if err != nil || graph.Stages[0].Resources == nil {
|
||||
t.Fatalf("%s: %v", value, err)
|
||||
}
|
||||
if _, ok := StageDefinition(graph.Stages[0]).Blocks["resources"]; ok {
|
||||
t.Fatal("runner control reached executor")
|
||||
}
|
||||
encoded, _ := json.Marshal(graph)
|
||||
if !strings.Contains(string(encoded), `"resources"`) {
|
||||
t.Fatal("resource lost in persisted graph")
|
||||
}
|
||||
}
|
||||
for _, value := range []string{`{}`, `null`, `[]`, `{"exclusive":""}`, `{"exclusive":"a b"}`, `{"exclusive":"a","exclusive":"b"}`, `{"exclusive":"a","extra":1}`, `{"exclusive":["a","b"]}`, fmt.Sprintf(`{"exclusive":%q}`, strings.Repeat("a", 129))} {
|
||||
if _, err := Parse(base + "```resources\n" + value + "\n```\n"); err == nil {
|
||||
t.Fatalf("accepted %s", value)
|
||||
}
|
||||
}
|
||||
human := "# stage gate\n```prompt\nChoose\n```\n```configs\ntype=human\n```\n```edges\ngo -> #success\n```\n```resources\n{\"exclusive\":\"phone\"}\n```\n"
|
||||
if _, err := Parse(human); err == nil {
|
||||
t.Fatal("human acquired a resource")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePoolDeclaration(t *testing.T) {
|
||||
base := Template([]string{"work"}, true)
|
||||
graph, err := Parse(base + "```resources\n{\"pool\":\"android-targets\"}\n```\n")
|
||||
if err != nil || graph.Stages[0].Resources == nil || graph.Stages[0].Resources.Pool != "android-targets" {
|
||||
t.Fatalf("pool declaration: %#v, %v", graph.Stages[0].Resources, err)
|
||||
}
|
||||
for _, value := range []string{`{"exclusive":"phone","pool":"targets"}`, `{"pool":""}`, `{"pool":"a b"}`} {
|
||||
if _, err := Parse(base + "```resources\n" + value + "\n```\n"); err == nil {
|
||||
t.Fatalf("accepted invalid pool declaration %s", value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePoolAllocatesConcreteMembersAndExcludesDirectOwner(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "android-targets", Members: []protocol.ResourcePoolMember{
|
||||
{Member: "phone", Enabled: true, Metadata: map[string]string{"kind": "phone"}},
|
||||
{Member: "emulator", Enabled: true, Metadata: map[string]string{"kind": "emulator"}},
|
||||
}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
phone := acquireTestResource(t, s, create("phone"))
|
||||
visit := create("android-targets")
|
||||
lease, ch, _, err := s.RequestResourcePool(ctx, "android-targets", visit)
|
||||
if err != nil || ch != nil || lease.Target == nil || lease.Target.Member != "emulator" {
|
||||
t.Fatalf("pool selected %#v, channel=%v, err=%v", lease.Target, ch, err)
|
||||
}
|
||||
if lease.Target.Metadata["kind"] != "emulator" {
|
||||
t.Fatal("missing bounded target metadata")
|
||||
}
|
||||
releaseTestResource(t, s, lease)
|
||||
releaseTestResource(t, s, phone)
|
||||
}
|
||||
|
||||
func TestResourcePoolFIFOWakeAndQuarantineIsolation(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
|
||||
{Member: "held", Enabled: true}, {Member: "free", Enabled: true}, {Member: "bad", Enabled: false, Quarantined: true},
|
||||
}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
held := acquireTestResource(t, s, create("held"))
|
||||
first := create("targets")
|
||||
firstLease, firstCh, _, err := s.RequestResourcePool(ctx, "targets", first)
|
||||
if err != nil || firstCh != nil || firstLease.Target == nil {
|
||||
t.Fatalf("expected immediate free member: %v", err)
|
||||
}
|
||||
// The second request waits once both concrete members are held.
|
||||
second := create("targets")
|
||||
_, secondCh, _, err := s.RequestResourcePool(ctx, "targets", second)
|
||||
if err != nil || secondCh == nil {
|
||||
t.Fatalf("expected pool waiter: %v", err)
|
||||
}
|
||||
releaseTestResource(t, s, firstLease)
|
||||
select {
|
||||
case next := <-secondCh:
|
||||
if next.Target == nil || next.Target.Member != "free" {
|
||||
t.Fatalf("wrong handoff: %#v", next.Target)
|
||||
}
|
||||
s.AcceptResource(next)
|
||||
releaseTestResource(t, s, next)
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("pool waiter not woken")
|
||||
}
|
||||
releaseTestResource(t, s, held)
|
||||
}
|
||||
|
||||
func TestResourcePoolHandoffUsesAnotherMemberWhenReleasedTargetQuarantined(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
pool := protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
|
||||
{Member: "phone", Enabled: true}, {Member: "emulator", Enabled: false},
|
||||
}}
|
||||
if err := s.ConfigureResourcePool(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
owner, ready, _, err := s.RequestResourcePool(ctx, pool.Name, create(pool.Name))
|
||||
if err != nil || ready != nil || owner.Target.Member != "phone" {
|
||||
t.Fatalf("owner: %+v %v", owner, err)
|
||||
}
|
||||
_, waiter, _, err := s.RequestResourcePool(ctx, pool.Name, create(pool.Name))
|
||||
if err != nil || waiter == nil {
|
||||
t.Fatalf("waiter: %v", err)
|
||||
}
|
||||
pool.Members[0].Quarantined = true
|
||||
pool.Members[1].Enabled = true
|
||||
if err := s.ConfigureResourcePool(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
releaseTestResource(t, s, owner)
|
||||
select {
|
||||
case next := <-waiter:
|
||||
if next.Target == nil || next.Target.Member != "emulator" {
|
||||
t.Fatalf("wrong target: %+v", next.Target)
|
||||
}
|
||||
s.AcceptResource(next)
|
||||
releaseTestResource(t, s, next)
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("quarantined released phone blocked the free emulator")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePoolRejectsStaleVisitForExistingOwner(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
|
||||
{Member: "phone", Enabled: true},
|
||||
}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
owner := acquireTestResource(t, s, create("phone"))
|
||||
stale := owner.ResourceVisit
|
||||
stale.Resource = "targets"
|
||||
stale.Sequence++
|
||||
if _, ch, _, err := s.RequestResourcePool(ctx, "targets", stale); err == nil || ch != nil {
|
||||
t.Fatalf("stale pool visit was accepted: channel=%v err=%v", ch, err)
|
||||
}
|
||||
if err := s.SetResourceCleanup(ctx, owner, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.ReleaseResource(ctx, owner); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePoolAndDirectVisitsCannotBeReinterpreted(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{{Member: "phone", Enabled: true}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pooled := create("targets")
|
||||
lease, ch, _, err := s.RequestResourcePool(ctx, "targets", pooled)
|
||||
if err != nil || ch != nil {
|
||||
t.Fatalf("pool acquire: %v %v", ch, err)
|
||||
}
|
||||
if _, ch, _, err := s.RequestResource(ctx, ResourceVisit{Resource: "phone", RunID: pooled.RunID, Stage: pooled.Stage, Sequence: pooled.Sequence}); err == nil || ch != nil {
|
||||
t.Fatalf("pooled owner was accepted as direct lease: channel=%v err=%v", ch, err)
|
||||
}
|
||||
releaseTestResource(t, s, lease)
|
||||
}
|
||||
|
||||
func TestResourcePoolConfigurationAndOwnerSurviveReopen(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
|
||||
{Member: "phone", Enabled: true, Metadata: map[string]string{"kind": "phone"}},
|
||||
}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
visit := create("targets")
|
||||
owner, ch, _, err := s.RequestResourcePool(ctx, "targets", visit)
|
||||
if err != nil || ch != nil {
|
||||
t.Fatalf("pool acquire: %v %v", ch, err)
|
||||
}
|
||||
path := s.path
|
||||
if err := s.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reopened, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reopened.Close()
|
||||
recovered, ch, _, err := reopened.RequestResourcePool(ctx, "targets", visit)
|
||||
if err != nil || ch != nil || recovered.Target == nil || recovered.Target.Member != "phone" || recovered.Target.Metadata["kind"] != "phone" {
|
||||
t.Fatalf("reopened pool owner not recovered: target=%#v channel=%v err=%v", recovered.Target, ch, err)
|
||||
}
|
||||
if err := reopened.SetResourceCleanup(ctx, recovered, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := reopened.ReleaseResource(ctx, recovered); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = owner
|
||||
}
|
||||
|
||||
func TestResourcePoolAllocatesDistinctMembersConcurrently(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
if err := s.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
|
||||
{Member: "phone", Enabled: true}, {Member: "emulator", Enabled: true},
|
||||
}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
visits := []ResourceVisit{create("targets"), create("targets")}
|
||||
leases := make(chan ResourceLease, len(visits))
|
||||
errs := make(chan error, len(visits))
|
||||
var wg sync.WaitGroup
|
||||
for _, visit := range visits {
|
||||
wg.Add(1)
|
||||
go func(v ResourceVisit) {
|
||||
defer wg.Done()
|
||||
lease, ch, _, err := s.RequestResourcePool(ctx, "targets", v)
|
||||
if err != nil {
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
if ch != nil {
|
||||
select {
|
||||
case lease = <-ch:
|
||||
case <-time.After(time.Second):
|
||||
errs <- fmt.Errorf("pool request remained queued")
|
||||
return
|
||||
}
|
||||
s.AcceptResource(lease)
|
||||
}
|
||||
leases <- lease
|
||||
}(visit)
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
for err := range errs {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := map[string]bool{}
|
||||
for range visits {
|
||||
lease := <-leases
|
||||
if lease.Target == nil || got[lease.Target.Member] {
|
||||
t.Fatalf("members were not distinct: %#v", lease.Target)
|
||||
}
|
||||
got[lease.Target.Member] = true
|
||||
releaseTestResource(t, s, lease)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePoolRejectsInvalidDeclarations(t *testing.T) {
|
||||
s, _ := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
cases := []protocol.ResourcePool{
|
||||
{Name: "missing-members"},
|
||||
{Name: "duplicate", Members: []protocol.ResourcePoolMember{{Member: "phone"}, {Member: "phone"}}},
|
||||
{Name: "bad-member", Members: []protocol.ResourcePoolMember{{Member: "not valid"}}},
|
||||
{Name: "bad-metadata", Members: []protocol.ResourcePoolMember{{Member: "phone", Metadata: map[string]string{"bad key": "x"}}}},
|
||||
{Name: "oversized-metadata-key", Members: []protocol.ResourcePoolMember{{Member: "phone", Metadata: map[string]string{strings.Repeat("k", maxTargetMetadataKey+1): "x"}}}},
|
||||
}
|
||||
for _, pool := range cases {
|
||||
if err := s.ConfigureResourcePool(ctx, pool); err == nil {
|
||||
t.Fatalf("accepted invalid pool %#v", pool)
|
||||
}
|
||||
}
|
||||
visit := ResourceVisit{Resource: "unknown", RunID: 1, Stage: "work", Sequence: 1}
|
||||
if _, _, _, err := s.RequestResourcePool(ctx, "unknown", visit); err == nil {
|
||||
t.Fatal("accepted unknown pool")
|
||||
}
|
||||
}
|
||||
|
||||
func resourceStore(t *testing.T) (*Store, func(string) ResourceVisit) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
s, err := Open(filepath.Join(t.TempDir(), "resources.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
if _, err := s.Submit(ctx, "resource", Template([]string{"work"}, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := s.List(ctx)
|
||||
d, _ := s.Get(ctx, items[0].ID, 1)
|
||||
count := 0
|
||||
return s, func(resource string) ResourceVisit {
|
||||
count++
|
||||
id, _, err := s.CreateRun(ctx, d, fmt.Sprint(count), "workspace", false, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return ResourceVisit{Resource: resource, RunID: id, Stage: "work", Sequence: 1}
|
||||
}
|
||||
}
|
||||
|
||||
func acquireTestResource(t *testing.T, s *Store, visit ResourceVisit) ResourceLease {
|
||||
t.Helper()
|
||||
l, ch, _, err := s.RequestResource(context.Background(), visit)
|
||||
if err != nil || ch != nil {
|
||||
t.Fatalf("acquire: %v %v", ch, err)
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
func releaseTestResource(t *testing.T, s *Store, lease ResourceLease) {
|
||||
t.Helper()
|
||||
if err := s.SetResourceCleanup(context.Background(), lease, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.ReleaseResource(context.Background(), lease); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceFIFOOwnershipCancellationAndRegistry(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
a := acquireTestResource(t, s, create("X"))
|
||||
duplicate := acquireTestResource(t, s, a.ResourceVisit)
|
||||
if duplicate != a {
|
||||
t.Fatal("duplicate acquisition changed capability")
|
||||
}
|
||||
var visits []ResourceVisit
|
||||
var channels []<-chan ResourceLease
|
||||
for i := 0; i < 4; i++ {
|
||||
v := create("X")
|
||||
_, ch, wait, err := s.RequestResource(ctx, v)
|
||||
if err != nil || ch == nil || wait == nil {
|
||||
t.Fatal("missing waiter", err)
|
||||
}
|
||||
_, again, _, err := s.RequestResource(ctx, v)
|
||||
if err != nil || again != ch {
|
||||
t.Fatal("duplicate waiter registered", err)
|
||||
}
|
||||
visits = append(visits, v)
|
||||
channels = append(channels, ch)
|
||||
}
|
||||
if err := s.ReleaseResource(ctx, a); err != nil {
|
||||
t.Fatal(err)
|
||||
} // no cleanup receipt
|
||||
for _, ch := range channels {
|
||||
select {
|
||||
case <-ch:
|
||||
t.Fatal("unconfirmed release granted")
|
||||
default:
|
||||
}
|
||||
}
|
||||
if err := s.SetResourceCleanup(ctx, a, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, corrupt := range []func(*ResourceLease){func(l *ResourceLease) { l.RunID++ }, func(l *ResourceLease) { l.Stage = "other" }, func(l *ResourceLease) { l.Sequence++ }, func(l *ResourceLease) { l.token = "wrong" }} {
|
||||
bad := a
|
||||
corrupt(&bad)
|
||||
if err := s.ReleaseResource(ctx, bad); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, ch := range channels {
|
||||
select {
|
||||
case <-ch:
|
||||
t.Fatal("unconfirmed/mismatched release granted")
|
||||
default:
|
||||
}
|
||||
}
|
||||
if err := s.CancelResource(ctx, visits[1]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
releaseTestResource(t, s, a)
|
||||
for _, i := range []int{0, 2, 3} {
|
||||
var grant ResourceLease
|
||||
select {
|
||||
case grant = <-channels[i]:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("FIFO grant not delivered")
|
||||
}
|
||||
s.AcceptResource(grant)
|
||||
if grant.ResourceVisit != visits[i] {
|
||||
t.Fatal("wrong FIFO owner")
|
||||
}
|
||||
// An old owner's repeated release cannot release the next owner.
|
||||
if err := s.ReleaseResource(ctx, a); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, j := range []int{2, 3} {
|
||||
if j > i {
|
||||
select {
|
||||
case <-channels[j]:
|
||||
t.Fatal("multiple grants")
|
||||
default:
|
||||
}
|
||||
}
|
||||
}
|
||||
releaseTestResource(t, s, grant)
|
||||
}
|
||||
if len(s.resourceWaiters) != 0 {
|
||||
t.Fatal("channel registry leaked")
|
||||
}
|
||||
for i := 0; i < 30; i++ {
|
||||
owner := acquireTestResource(t, s, create("X"))
|
||||
v := create("X")
|
||||
_, _, _, err := s.RequestResource(ctx, v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
releaseTestResource(t, s, owner) // race window: granted but not accepted
|
||||
if err := s.CancelResource(ctx, v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(s.resourceWaiters) != 0 {
|
||||
t.Fatal("grant/cancel leaked channel")
|
||||
}
|
||||
}
|
||||
for i := 0; i < 30; i++ {
|
||||
owner := acquireTestResource(t, s, create("X"))
|
||||
v := create("X")
|
||||
if _, _, _, err := s.RequestResource(ctx, v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.SetResourceCleanup(ctx, owner, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(2)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if err := s.ReleaseResource(ctx, owner); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if err := s.CancelResource(ctx, v); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
wg.Wait()
|
||||
if len(s.resourceWaiters) != 0 {
|
||||
t.Fatal("concurrent grant/cancel leaked channel")
|
||||
}
|
||||
}
|
||||
var owners, waiters int
|
||||
s.db.QueryRow("SELECT COUNT(*) FROM resource_owners").Scan(&owners)
|
||||
s.db.QueryRow("SELECT COUNT(*) FROM resource_waiters").Scan(&waiters)
|
||||
if owners != 0 || waiters != 0 {
|
||||
t.Fatalf("stranded owners/waiters: %d/%d", owners, waiters)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceCancellationPersistenceFailureRemovesChannel(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
acquireTestResource(t, s, create("X"))
|
||||
v := create("X")
|
||||
if _, _, _, err := s.RequestResource(context.Background(), v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.CancelResource(context.Background(), v); err == nil {
|
||||
t.Fatal("closed DB cancellation claimed success")
|
||||
}
|
||||
if len(s.resourceWaiters) != 0 {
|
||||
t.Fatal("persistence failure leaked channel")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceRestartQuarantineAndRetention(t *testing.T) {
|
||||
for _, confirmed := range []bool{false, true} {
|
||||
t.Run(fmt.Sprint(confirmed), func(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
a := acquireTestResource(t, s, create("X"))
|
||||
b := create("X")
|
||||
if _, _, _, err := s.RequestResource(ctx, b); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if confirmed {
|
||||
if err := s.SetResourceCleanup(ctx, a, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
path := s.path
|
||||
if err := s.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reopened, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reopened.Close()
|
||||
if err := reopened.RecoverRuns(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := reopened.RecoverRuns(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
detail, err := reopened.GetRun(ctx, b.RunID)
|
||||
if err != nil || detail.Status != "interrupted" || detail.ResourceWait != nil || len(detail.Steps) != 0 || !strings.Contains(detail.Diagnostic, "resource") {
|
||||
t.Fatal("queued visit replayed/lost diagnostic", detail, err)
|
||||
}
|
||||
exists, err := reopened.HasResourceOwner(ctx, a.RunID)
|
||||
if err != nil || exists == confirmed {
|
||||
t.Fatal("incorrect recovery disposition", exists, err)
|
||||
}
|
||||
report, err := reopened.RetentionReport(ctx, RetentionPolicy{Keep: 0}, "", time.Now())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
protected := false
|
||||
for _, p := range report.Protected {
|
||||
if p.ID == a.RunID {
|
||||
protected = true
|
||||
}
|
||||
}
|
||||
if protected == confirmed {
|
||||
t.Fatal("quarantined evidence not protected")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceReconcileMissingWaiterPreservesQuarantine(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
owner := acquireTestResource(t, s, create("X"))
|
||||
if _, err := s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
waiter := create("X")
|
||||
if _, ch, _, err := s.RequestResource(ctx, waiter); err != nil || ch == nil {
|
||||
t.Fatalf("register waiter: %v", err)
|
||||
}
|
||||
// Simulate a runner disappearing while its durable waiter remains. The
|
||||
// authoritative operation must reject the handoff before audit or cleanup.
|
||||
delete(s.resourceWaiters, waiter)
|
||||
err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
|
||||
Resource: owner.Resource,
|
||||
RunID: owner.RunID,
|
||||
Stage: owner.Stage,
|
||||
Sequence: owner.Sequence,
|
||||
Actor: "operator",
|
||||
Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, NoMutatingAction: true, Summary: "inspection confirmed no mutating action occurred"},
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("reconciled without a live waiter channel")
|
||||
}
|
||||
var cleanup, audits, owners int
|
||||
if err := s.db.QueryRow("SELECT cleanup_confirmed FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&cleanup); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=?", owner.Resource).Scan(&audits); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&owners); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cleanup != 0 || audits != 0 || owners != 1 {
|
||||
t.Fatalf("failed recovery changed state: cleanup=%d audits=%d owners=%d", cleanup, audits, owners)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceReconcileTransfersAndAudits(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
owner := acquireTestResource(t, s, create("X"))
|
||||
if _, err := s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
waiter := create("X")
|
||||
_, ch, _, err := s.RequestResource(ctx, waiter)
|
||||
if err != nil || ch == nil {
|
||||
t.Fatalf("register waiter: %v", err)
|
||||
}
|
||||
if err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
|
||||
Resource: owner.Resource,
|
||||
RunID: owner.RunID,
|
||||
Stage: owner.Stage,
|
||||
Sequence: owner.Sequence,
|
||||
Actor: "operator",
|
||||
Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, NoMutatingAction: true, Summary: "inspection confirmed no mutating action occurred"},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case next := <-ch:
|
||||
s.AcceptResource(next)
|
||||
if next.ResourceVisit != waiter {
|
||||
t.Fatalf("wrong handoff: %#v", next.ResourceVisit)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("reconciliation did not wake waiter")
|
||||
}
|
||||
var audits int
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=? AND result='reconciled'", owner.Resource).Scan(&audits); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if audits != 1 {
|
||||
t.Fatalf("recovery audit count=%d", audits)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceReconcileReplayAfterSuccessorHandoff(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
ctx := context.Background()
|
||||
owner := acquireTestResource(t, s, create("X"))
|
||||
if _, err := s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
successor := create("X")
|
||||
_, ch, _, err := s.RequestResource(ctx, successor)
|
||||
if err != nil || ch == nil {
|
||||
t.Fatalf("register successor: %v", err)
|
||||
}
|
||||
request := protocol.ResourceRecoveryRequest{
|
||||
Resource: owner.Resource,
|
||||
RunID: owner.RunID,
|
||||
Stage: owner.Stage,
|
||||
Sequence: owner.Sequence,
|
||||
Actor: "operator",
|
||||
Evidence: &protocol.ResourceRecoveryEvidence{
|
||||
CleanupConfirmed: true,
|
||||
NoMutatingAction: true,
|
||||
Summary: "inspection confirmed no mutating action occurred",
|
||||
},
|
||||
}
|
||||
if err := s.ReconcileResource(ctx, request); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var successorLease ResourceLease
|
||||
select {
|
||||
case successorLease = <-ch:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("reconciliation did not hand off to successor")
|
||||
}
|
||||
if successorLease.ResourceVisit != successor {
|
||||
t.Fatalf("wrong successor visit: %#v", successorLease.ResourceVisit)
|
||||
}
|
||||
if successorLease.token == "" {
|
||||
t.Fatal("successor capability was empty")
|
||||
}
|
||||
s.AcceptResource(successorLease)
|
||||
|
||||
// The old request is an idempotent replay after the handoff. It must not
|
||||
// match or release the successor's private capability.
|
||||
if err := s.ReconcileResource(ctx, request); err == nil {
|
||||
t.Fatal("old recovery replay was accepted against successor owner")
|
||||
}
|
||||
inspection, err := s.InspectResource(ctx, owner.Resource)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if inspection.Owner == nil || inspection.Owner.RunID != successor.RunID || inspection.Owner.Stage != successor.Stage || inspection.Owner.Sequence != successor.Sequence {
|
||||
t.Fatalf("successor was not preserved: %#v", inspection.Owner)
|
||||
}
|
||||
if len(inspection.Waiters) != 0 || len(inspection.Audits) != 1 || inspection.Audits[0].RunID != owner.RunID {
|
||||
t.Fatalf("replay changed handoff state: waiters=%d audits=%#v", len(inspection.Waiters), inspection.Audits)
|
||||
}
|
||||
if len(s.resourceWaiters) != 0 {
|
||||
t.Fatal("successor channel registry leaked")
|
||||
}
|
||||
var owners, waiters, audits int
|
||||
if err := s.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&owners); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_waiters WHERE resource_key=?", owner.Resource).Scan(&waiters); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM resource_recovery_audit WHERE resource_key=?", owner.Resource).Scan(&audits); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if owners != 1 || waiters != 0 || audits != 1 {
|
||||
t.Fatalf("unexpected post-replay counts: owners=%d waiters=%d audits=%d", owners, waiters, audits)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceReconcileRequiresQuarantinedInactiveOwner(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
test := func(t *testing.T, setup func(*Store, ResourceLease) int) {
|
||||
s, create := resourceStore(t)
|
||||
owner := acquireTestResource(t, s, create("X"))
|
||||
expectedCleanup := setup(s, owner)
|
||||
err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
|
||||
Resource: owner.Resource, RunID: owner.RunID, Stage: owner.Stage, Sequence: owner.Sequence,
|
||||
Actor: "operator", Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, Summary: "verified"},
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("reconciled unsafe owner")
|
||||
}
|
||||
var owners, audits, cleanup int
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_owners").Scan(&owners); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit").Scan(&audits); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRow("SELECT cleanup_confirmed FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&cleanup); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if owners != 1 || audits != 0 || cleanup != expectedCleanup {
|
||||
t.Fatalf("unsafe recovery changed state: owners=%d audits=%d cleanup=%d", owners, audits, cleanup)
|
||||
}
|
||||
}
|
||||
t.Run("active run", func(t *testing.T) {
|
||||
test(t, func(s *Store, owner ResourceLease) int {
|
||||
if _, err := s.db.Exec("UPDATE pipeline_runs SET status=? WHERE id=?", protocol.RunStatusWaiting, owner.RunID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return 0
|
||||
})
|
||||
})
|
||||
t.Run("active agent", func(t *testing.T) {
|
||||
test(t, func(s *Store, owner ResourceLease) int {
|
||||
if _, err := s.db.Exec("UPDATE pipeline_runs SET status='interrupted' WHERE id=?", owner.RunID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.db.Exec("INSERT INTO run_steps(run_id,sequence,stage_key,status,agent_active,input_json) VALUES(?,?,?,?,?, '{}')", owner.RunID, owner.Sequence, owner.Stage, "running", 1); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return 0
|
||||
})
|
||||
})
|
||||
t.Run("already confirmed", func(t *testing.T) {
|
||||
test(t, func(s *Store, owner ResourceLease) int {
|
||||
if err := s.SetResourceCleanup(ctx, owner, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return 1
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
func TestResourceReconcileRejectsMalformedEvidenceBeforeMutation(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
for _, summary := range []string{"", "true", "FALSE", "null", "verified\x00cleanup", strings.Repeat("x", maxRecoveryText+1)} {
|
||||
t.Run(fmt.Sprintf("%q", summary), func(t *testing.T) {
|
||||
s, create := resourceStore(t)
|
||||
owner := acquireTestResource(t, s, create("X"))
|
||||
err := s.ReconcileResource(ctx, protocol.ResourceRecoveryRequest{
|
||||
Resource: owner.Resource, RunID: owner.RunID, Stage: owner.Stage, Sequence: owner.Sequence,
|
||||
Actor: "operator", Evidence: &protocol.ResourceRecoveryEvidence{CleanupConfirmed: true, Summary: summary},
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("accepted malformed evidence")
|
||||
}
|
||||
var owners, audits, cleanup int
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_owners").Scan(&owners); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM resource_recovery_audit").Scan(&audits); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.db.QueryRow("SELECT cleanup_confirmed FROM resource_owners WHERE resource_key=?", owner.Resource).Scan(&cleanup); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if owners != 1 || audits != 0 || cleanup != 0 {
|
||||
t.Fatalf("invalid evidence changed state: owners=%d audits=%d cleanup=%d", owners, audits, cleanup)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -130,12 +130,11 @@ type retentionQuerier interface {
|
||||
// retentionRunInfo is one pipeline_runs row plus the step aggregates protection
|
||||
// needs, read in the same statement so the report never issues a per-run query.
|
||||
type retentionRunInfo struct {
|
||||
id int64
|
||||
status string
|
||||
createdAt time.Time
|
||||
waitingSteps int
|
||||
activeSteps int
|
||||
resourceOwners int
|
||||
id int64
|
||||
status string
|
||||
createdAt time.Time
|
||||
waitingSteps int
|
||||
activeSteps int
|
||||
}
|
||||
|
||||
// protectionReasons lists why a run must not be removed for a reason intrinsic
|
||||
@@ -156,9 +155,6 @@ func (r retentionRunInfo) protectionReasons() []string {
|
||||
if r.activeSteps > 0 {
|
||||
reasons = append(reasons, "cleanup agent still active (agent_active=1)")
|
||||
}
|
||||
if r.resourceOwners > 0 {
|
||||
reasons = append(reasons, "resource ownership retained (possibly quarantined)")
|
||||
}
|
||||
return reasons
|
||||
}
|
||||
|
||||
@@ -175,20 +171,9 @@ type retentionSelection struct {
|
||||
// report is a bounded number of round trips rather than one query per run.
|
||||
func selectRetentionRuns(ctx context.Context, q retentionQuerier, policy RetentionPolicy, now time.Time) (retentionSelection, error) {
|
||||
var sel retentionSelection
|
||||
// Maintenance is deliberately non-migrating and may inspect a database
|
||||
// whose server predates resource support.
|
||||
var resources bool
|
||||
if err := q.QueryRowContext(ctx, "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE type='table' AND name='resource_owners')").Scan(&resources); err != nil {
|
||||
return sel, err
|
||||
}
|
||||
owners := "0"
|
||||
if resources {
|
||||
owners = "(SELECT COUNT(*) FROM resource_owners o WHERE o.run_id=r.id)"
|
||||
}
|
||||
rows, err := q.QueryContext(ctx, `SELECT r.id,r.status,strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),
|
||||
COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.status=?),0),
|
||||
COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.agent_active=1),0),
|
||||
`+owners+` `+
|
||||
COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.agent_active=1),0) `+
|
||||
runFrom+` ORDER BY r.id DESC`, protocol.StepStatusWaiting)
|
||||
if err != nil {
|
||||
return sel, err
|
||||
@@ -197,7 +182,7 @@ COALESCE((SELECT COUNT(*) FROM run_steps s WHERE s.run_id=r.id AND s.agent_activ
|
||||
for rows.Next() {
|
||||
var run retentionRunInfo
|
||||
var created string
|
||||
if err := rows.Scan(&run.id, &run.status, &created, &run.waitingSteps, &run.activeSteps, &run.resourceOwners); err != nil {
|
||||
if err := rows.Scan(&run.id, &run.status, &created, &run.waitingSteps, &run.activeSteps); err != nil {
|
||||
return sel, err
|
||||
}
|
||||
run.createdAt = parseRetentionTime(created)
|
||||
|
||||
@@ -21,19 +21,14 @@ const runStamps = `strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),COALESCE(strftime
|
||||
const runSelect = `SELECT r.id,r.pipeline_id,p.name,r.version,v.content_hash,r.status,r.outcome,r.current_stage,
|
||||
` + runStamps + `,r.diagnostic,
|
||||
CASE WHEN r.status='succeeded' THEN r.output_json ELSE COALESCE((SELECT output_json FROM run_steps WHERE run_id=r.id AND status='completed' ORDER BY sequence DESC LIMIT 1),'{}') END,
|
||||
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title,r.model,r.stage_models,r.resource_wait
|
||||
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title,r.model,r.stage_models
|
||||
` + runFrom + ` JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version`
|
||||
|
||||
func scanRun(row interface{ Scan(...any) error }, r *protocol.RunSummary) error {
|
||||
var output, stageModels, resourceWait string
|
||||
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title, &r.Model, &stageModels, &resourceWait); err != nil {
|
||||
var output, stageModels string
|
||||
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title, &r.Model, &stageModels); err != nil {
|
||||
return err
|
||||
}
|
||||
if resourceWait != "" {
|
||||
if err := json.Unmarshal([]byte(resourceWait), &r.ResourceWait); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if stageModels != "" && stageModels != "{}" {
|
||||
if err := json.Unmarshal([]byte(stageModels), &r.StageModels); err != nil {
|
||||
return err
|
||||
|
||||
@@ -44,7 +44,7 @@ func (s *Store) BeginStep(ctx context.Context, id int64, sequence int, key strin
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='running',current_stage=?,resource_wait='' WHERE id=? AND status IN ('queued','running')", key, id)); err != nil {
|
||||
if err := affected(tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='running',current_stage=? WHERE id=? AND status IN ('queued','running')", key, id)); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, "INSERT INTO run_steps(run_id,sequence,stage_key,input_json) VALUES (?,?,?,?)", id, sequence, key, string(encoded)); err != nil {
|
||||
@@ -310,5 +310,5 @@ func (s *Store) FinishRun(ctx context.Context, id int64, status, outcome, diagno
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return affected(s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status=?,outcome=?,diagnostic=?,output_json=?,finished_at=CURRENT_TIMESTAMP,resource_wait='' WHERE id=? AND status IN ('queued','running')", status, outcome, diagnostic, string(data), id))
|
||||
return affected(s.db.ExecContext(ctx, "UPDATE pipeline_runs SET status=?,outcome=?,diagnostic=?,output_json=?,finished_at=CURRENT_TIMESTAMP WHERE id=? AND status IN ('queued','running')", status, outcome, diagnostic, string(data), id))
|
||||
}
|
||||
|
||||
@@ -55,48 +55,64 @@ run_id INTEGER PRIMARY KEY REFERENCES pipeline_runs(id), original_run_id INTEGER
|
||||
CREATE TABLE IF NOT EXISTS retry_checkpoints (
|
||||
run_id INTEGER PRIMARY KEY REFERENCES pipeline_runs(id), source_run_id INTEGER NOT NULL REFERENCES pipeline_runs(id),
|
||||
mode TEXT NOT NULL, failed_sequence INTEGER NOT NULL, handoff_json TEXT NOT NULL,
|
||||
source_version INTEGER NOT NULL, source_hash TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);
|
||||
CREATE TABLE IF NOT EXISTS resource_owners (
|
||||
resource_key TEXT PRIMARY KEY, run_id INTEGER NOT NULL REFERENCES pipeline_runs(id),
|
||||
stage_key TEXT NOT NULL, sequence INTEGER NOT NULL, token TEXT NOT NULL, pool_name TEXT NOT NULL DEFAULT '',
|
||||
cleanup_confirmed INTEGER NOT NULL DEFAULT 0, diagnostic TEXT NOT NULL DEFAULT '',
|
||||
acquired_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);
|
||||
CREATE INDEX IF NOT EXISTS resource_owner_run ON resource_owners(run_id);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS resource_owner_visit ON resource_owners(run_id);
|
||||
CREATE TABLE IF NOT EXISTS resource_waiters (
|
||||
queue_order INTEGER PRIMARY KEY AUTOINCREMENT, resource_key TEXT NOT NULL,
|
||||
run_id INTEGER NOT NULL UNIQUE REFERENCES pipeline_runs(id), stage_key TEXT NOT NULL,
|
||||
sequence INTEGER NOT NULL, token TEXT NOT NULL, queued_at TEXT NOT NULL);
|
||||
CREATE INDEX IF NOT EXISTS resource_wait_fifo ON resource_waiters(resource_key,queue_order);`)
|
||||
source_version INTEGER NOT NULL, source_hash TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS resource_pools (
|
||||
name TEXT PRIMARY KEY, members_json TEXT NOT NULL, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);`); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.addColumnIfMissing("resource_waiters", "pool_name", "TEXT NOT NULL DEFAULT ''"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.addColumnIfMissing("resource_owners", "pool_name", "TEXT NOT NULL DEFAULT ''"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.migrateRunTitles(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.addColumnIfMissing("pipeline_runs", "resource_wait", "TEXT NOT NULL DEFAULT ''"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS resource_recovery_audit (
|
||||
resource_key TEXT NOT NULL, run_id INTEGER NOT NULL, stage_key TEXT NOT NULL,
|
||||
sequence INTEGER NOT NULL, actor TEXT NOT NULL, source TEXT NOT NULL,
|
||||
evidence TEXT NOT NULL, result TEXT NOT NULL, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP)`); err != nil {
|
||||
if err := s.dropResourceSchema(); err != nil {
|
||||
return err
|
||||
}
|
||||
return s.migrateRunModels()
|
||||
}
|
||||
|
||||
// dropResourceSchema removes the retired resource pool, ownership, wait queue
|
||||
// and recovery tables, plus the queue snapshot column. It is idempotent: a
|
||||
// database that predates the feature or was already migrated opens cleanly,
|
||||
// while an older database loses only the removed feature's state.
|
||||
func (s *Store) dropResourceSchema() error {
|
||||
for _, table := range []string{"resource_owners", "resource_waiters", "resource_pools", "resource_recovery_audit"} {
|
||||
if _, err := s.db.Exec("DROP TABLE IF EXISTS " + table); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return s.dropColumnIfMissing("pipeline_runs", "resource_wait")
|
||||
}
|
||||
|
||||
// dropColumnIfMissing removes a retired column only when it is still present, so
|
||||
// repeated Open calls and databases created after the removal stay idempotent.
|
||||
func (s *Store) dropColumnIfMissing(table, column string) error {
|
||||
rows, err := s.db.Query("PRAGMA table_info(" + table + ")")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
found := false
|
||||
for rows.Next() {
|
||||
var cid, notnull, pk int
|
||||
var name, ctype string
|
||||
var dflt sql.NullString
|
||||
if err := rows.Scan(&cid, &name, &ctype, ¬null, &dflt, &pk); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
if name == column {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
rows.Close()
|
||||
if !found {
|
||||
return nil
|
||||
}
|
||||
_, err = s.db.Exec("ALTER TABLE " + table + " DROP COLUMN " + column)
|
||||
return err
|
||||
}
|
||||
|
||||
// migrateRunModels adds the launch-time model-override columns to a database
|
||||
// created before per-run overrides existed. Each column is added only when
|
||||
// missing, so an upgrade is idempotent across repeated Open calls and existing
|
||||
@@ -158,29 +174,18 @@ func (s *Store) addColumnIfMissing(table, column, definition string) error {
|
||||
}
|
||||
|
||||
func (s *Store) RecoverRuns(ctx context.Context) error {
|
||||
s.resourceMu.Lock()
|
||||
defer s.resourceMu.Unlock()
|
||||
tx, err := s.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
// No replay: pending visits never began. Uncertain owners survive recovery,
|
||||
// including the crash window before BeginStep/session registration.
|
||||
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET diagnostic='Server stopped while queued for a resource; pending visit was not resumed',status='interrupted',finished_at=CURRENT_TIMESTAMP,resource_wait='' WHERE id IN (SELECT run_id FROM resource_waiters); DELETE FROM resource_waiters"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := recoverResourceOwners(ctx, tx); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, "UPDATE run_steps SET status='interrupted',agent_active=CASE WHEN EXISTS(SELECT 1 FROM resource_owners o WHERE o.run_id=run_steps.run_id AND o.sequence=run_steps.sequence) THEN agent_active ELSE 0 END,diagnostic='Server stopped; stage was not resumed',finished_at=CURRENT_TIMESTAMP WHERE (status='running' OR agent_active=1) AND status<>?", protocol.StepStatusWaiting); err != nil {
|
||||
// No replay: pending visits never began. A cleanup agent that was still
|
||||
// active is marked interrupted, and its run is interrupted below.
|
||||
if _, err := tx.ExecContext(ctx, "UPDATE run_steps SET status='interrupted',agent_active=0,diagnostic='Server stopped; stage was not resumed',finished_at=CURRENT_TIMESTAMP WHERE (status='running' OR agent_active=1) AND status<>?", protocol.StepStatusWaiting); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET status='interrupted',diagnostic='Server stopped; run was not resumed',finished_at=CURRENT_TIMESTAMP WHERE status IN ('queued','running') AND status<>?", protocol.RunStatusWaiting); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, "UPDATE pipeline_runs SET diagnostic='Server stopped; resource ownership quarantined: inspect retained session, tool termination and restoration before reconciliation' WHERE id IN (SELECT run_id FROM resource_owners)"); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
@@ -29,6 +29,13 @@ func (s *Store) RunSession(ctx context.Context, id int64, sequence int, session
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// ConfirmRecoveredSession clears the agent-active marker after startup has
|
||||
// interrupted a previously active session, so it no longer blocks new work.
|
||||
func (s *Store) ConfirmRecoveredSession(ctx context.Context, owner SessionOwner) error {
|
||||
_, err := s.db.ExecContext(ctx, "UPDATE run_steps SET agent_active=0 WHERE run_id=? AND session_id=?", owner.RunID, owner.ID)
|
||||
return err
|
||||
}
|
||||
|
||||
type SessionOwner struct {
|
||||
ID string
|
||||
Backend string
|
||||
|
||||
@@ -18,12 +18,10 @@ import (
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
db *sql.DB
|
||||
dir string
|
||||
path string
|
||||
mu sync.Mutex
|
||||
resourceMu sync.Mutex
|
||||
resourceWaiters map[ResourceVisit]chan ResourceLease
|
||||
db *sql.DB
|
||||
dir string
|
||||
path string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// openDB opens one SQLite connection pool for a DSN. Every open path shares it
|
||||
|
||||
@@ -3,31 +3,22 @@ package protocol
|
||||
import "git.a-wan.net/alan/pipelines/internal/buildinfo"
|
||||
|
||||
const (
|
||||
Ping = "ping"
|
||||
Pong = "pong"
|
||||
Error = "error"
|
||||
Submit = "submit"
|
||||
Submitted = "submitted"
|
||||
List = "list_pipelines"
|
||||
PipelineList = "pipelines"
|
||||
GetPipeline = "get_pipeline"
|
||||
Pipeline = "pipeline"
|
||||
ListTemplates = "list_templates"
|
||||
Templates = "templates"
|
||||
Validate = "validate"
|
||||
Validated = "validated"
|
||||
InspectResource = "inspect_resource"
|
||||
ResourceInspectionResult = "resource_inspection"
|
||||
ReconcileResource = "reconcile_resource"
|
||||
ResourceReconciled = "resource_reconciled"
|
||||
InspectResourcePool = "inspect_resource_pool"
|
||||
ResourcePoolInspection = "resource_pool_inspection"
|
||||
ConfigureResourcePool = "configure_resource_pool"
|
||||
ResourcePoolConfigured = "resource_pool_configured"
|
||||
Ping = "ping"
|
||||
Pong = "pong"
|
||||
Error = "error"
|
||||
Submit = "submit"
|
||||
Submitted = "submitted"
|
||||
List = "list_pipelines"
|
||||
PipelineList = "pipelines"
|
||||
GetPipeline = "get_pipeline"
|
||||
Pipeline = "pipeline"
|
||||
ListTemplates = "list_templates"
|
||||
Templates = "templates"
|
||||
Validate = "validate"
|
||||
Validated = "validated"
|
||||
)
|
||||
|
||||
type Stage struct {
|
||||
Resources *StageResources `json:"resources,omitempty"`
|
||||
Limits *StageLimits `json:"limits,omitempty"`
|
||||
Memory map[string]string `json:"memory,omitempty"`
|
||||
Notify *NotificationPolicy `json:"notify,omitempty"`
|
||||
@@ -42,12 +33,6 @@ type Stage struct {
|
||||
Blocks map[string]string `json:"blocks,omitempty"`
|
||||
}
|
||||
|
||||
// StageResources declares one capacity-one resource, local to this store authority.
|
||||
type StageResources struct {
|
||||
Exclusive string `json:"exclusive"`
|
||||
Pool string `json:"pool"`
|
||||
}
|
||||
|
||||
// Limits are enforced by the runner and never included in the model prompt.
|
||||
type StageLimits struct {
|
||||
MaxVisits int `json:"max_visits"`
|
||||
@@ -211,10 +196,6 @@ type Message struct {
|
||||
// RunEvent is the server-pushed run state delta (see runevents.go) and
|
||||
// RunIDs the optional subscribe scope, so a client asks for the runs it
|
||||
// displays. Both are optional so older peers ignore them.
|
||||
RunEvent *RunEventFrame `json:"run_event,omitempty"`
|
||||
RunIDs []int64 `json:"run_ids,omitempty"`
|
||||
Resource string `json:"resource,omitempty"`
|
||||
ResourceInspection *ResourceInspection `json:"resource_inspection,omitempty"`
|
||||
ResourceRecovery *ResourceRecoveryRequest `json:"resource_recovery,omitempty"`
|
||||
ResourcePool *ResourcePool `json:"resource_pool,omitempty"`
|
||||
RunEvent *RunEventFrame `json:"run_event,omitempty"`
|
||||
RunIDs []int64 `json:"run_ids,omitempty"`
|
||||
}
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
package protocol
|
||||
|
||||
type ResourceInspection struct {
|
||||
Resource string `json:"resource"`
|
||||
Owner *ResourceOwner `json:"owner,omitempty"`
|
||||
Waiters []ResourceWaiter `json:"waiters,omitempty"`
|
||||
Audits []ResourceRecoveryAudit `json:"audits,omitempty"`
|
||||
Pool *ResourcePool `json:"pool,omitempty"`
|
||||
}
|
||||
|
||||
// ResourceTarget is bounded, non-secret metadata for the concrete member that
|
||||
// was selected. Credentials and capabilities are intentionally absent.
|
||||
type ResourceTarget struct {
|
||||
Member string `json:"member"`
|
||||
Metadata map[string]string `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ResourcePoolMember struct {
|
||||
Member string `json:"member"`
|
||||
Metadata map[string]string `json:"metadata,omitempty"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Quarantined bool `json:"quarantined,omitempty"`
|
||||
}
|
||||
|
||||
type ResourcePool struct {
|
||||
Name string `json:"name"`
|
||||
Members []ResourcePoolMember `json:"members"`
|
||||
}
|
||||
type ResourceOwner struct {
|
||||
RunID int64 `json:"run_id"`
|
||||
Stage string `json:"stage"`
|
||||
Sequence int `json:"sequence"`
|
||||
CleanupConfirmed bool `json:"cleanup_confirmed"`
|
||||
Quarantined bool `json:"quarantined"`
|
||||
AgentActive bool `json:"agent_active"`
|
||||
}
|
||||
type ResourceWaiter struct {
|
||||
RunID int64 `json:"run_id"`
|
||||
Stage string `json:"stage"`
|
||||
Sequence int `json:"sequence"`
|
||||
QueuedAt string `json:"queued_at"`
|
||||
}
|
||||
type ResourceRecoveryAudit struct {
|
||||
Resource string `json:"resource"`
|
||||
RunID int64 `json:"run_id"`
|
||||
Stage string `json:"stage"`
|
||||
Sequence int `json:"sequence"`
|
||||
Actor string `json:"actor"`
|
||||
Source string `json:"source"`
|
||||
Evidence string `json:"evidence"`
|
||||
Result string `json:"result"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type ResourceRecoveryEvidence struct {
|
||||
CleanupConfirmed bool `json:"cleanup_confirmed"`
|
||||
NoMutatingAction bool `json:"no_mutating_action"`
|
||||
Summary string `json:"summary"`
|
||||
}
|
||||
type ResourceRecoveryRequest struct {
|
||||
Resource string `json:"resource"`
|
||||
RunID int64 `json:"run_id"`
|
||||
Stage string `json:"stage"`
|
||||
Sequence int `json:"sequence"`
|
||||
Evidence *ResourceRecoveryEvidence `json:"evidence,omitempty"`
|
||||
Actor string `json:"actor"`
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
package protocol
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestResourceWaitOlderClientCompatibility(t *testing.T) {
|
||||
data, err := json.Marshal(RunSummary{ID: 1, Status: RunStatusQueued, ResourceWait: &ResourceWait{Resource: "X", Stage: "review", Sequence: 2, QueuedSince: "now"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var old struct {
|
||||
ID int64 `json:"id"`
|
||||
Status string `json:"status"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &old); err != nil || old.Status != "queued" {
|
||||
t.Fatal("old client no longer waits", old, err)
|
||||
}
|
||||
clear, err := json.Marshal(RunEventFrame{RunID: 1, ResourceWaitSet: true})
|
||||
if err != nil || !strings.Contains(string(clear), `"resource_wait_set":true`) {
|
||||
t.Fatal("no explicit clear", string(clear), err)
|
||||
}
|
||||
var current RunSummary
|
||||
if err := json.Unmarshal([]byte(`{"id":2,"status":"running"}`), ¤t); err != nil || current.ResourceWait != nil {
|
||||
t.Fatal("missing field changed old payload", err)
|
||||
}
|
||||
}
|
||||
@@ -24,11 +24,8 @@ const (
|
||||
// from the frame instead of re-reading the whole list. They repeat on every
|
||||
// frame (pipeline identity never changes mid-run).
|
||||
type RunEventFrame struct {
|
||||
// ResourceWaitSet makes a nil ResourceWait an explicit cache clear.
|
||||
ResourceWaitSet bool `json:"resource_wait_set,omitempty"`
|
||||
ResourceWait *ResourceWait `json:"resource_wait,omitempty"`
|
||||
RunID int64 `json:"run_id"`
|
||||
Workspace string `json:"workspace,omitempty"`
|
||||
RunID int64 `json:"run_id"`
|
||||
Workspace string `json:"workspace,omitempty"`
|
||||
// PipelineID, Name, Title and Version identify the run's pipeline and its
|
||||
// operator-facing title, so a new run's row can be built without a fetch.
|
||||
PipelineID int64 `json:"pipeline_id,omitempty"`
|
||||
|
||||
@@ -118,10 +118,9 @@ type SessionExport struct {
|
||||
}
|
||||
|
||||
type RunSummary struct {
|
||||
ResourceWait *ResourceWait `json:"resource_wait,omitempty"`
|
||||
ID int64 `json:"id"`
|
||||
PipelineID int64 `json:"pipeline_id"`
|
||||
Name string `json:"name"`
|
||||
ID int64 `json:"id"`
|
||||
PipelineID int64 `json:"pipeline_id"`
|
||||
Name string `json:"name"`
|
||||
// Title is the operator-facing, bounded single-line description of what the
|
||||
// run is for, generated from its prompt. It is never empty for a new run;
|
||||
// historical rows predating the title column may omit it.
|
||||
@@ -146,19 +145,6 @@ type RunSummary struct {
|
||||
StageModels map[string]string `json:"stage_models,omitempty"`
|
||||
}
|
||||
|
||||
// ResourceWait is bounded public queue metadata. Ownership credentials stay private.
|
||||
type ResourceWait struct {
|
||||
Resource string `json:"resource"`
|
||||
Stage string `json:"stage"`
|
||||
Sequence int `json:"sequence"`
|
||||
QueuedSince string `json:"queued_since"`
|
||||
}
|
||||
|
||||
// ResourceCleanupConfirmedKey lets a stage report failed restoration without
|
||||
// pretending that tool idleness also proves the resource is ready. An explicit
|
||||
// false (or malformed value) vetoes release; it never authorizes release alone.
|
||||
const ResourceCleanupConfirmedKey = "resource_cleanup_confirmed"
|
||||
|
||||
// Label is the operator-facing name of a run: the generated Title when the
|
||||
// server stored one, otherwise the pipeline Name (historical rows predate
|
||||
// titles). It is the single definition shared by the CLI and TUI.
|
||||
@@ -169,14 +155,6 @@ func (r RunSummary) Label() string {
|
||||
return r.Name
|
||||
}
|
||||
|
||||
// StatusLabel separates resource contention from a human decision everywhere.
|
||||
func (r RunSummary) StatusLabel() string {
|
||||
if r.ResourceWait != nil {
|
||||
return "Waiting for resource " + r.ResourceWait.Resource
|
||||
}
|
||||
return r.Status
|
||||
}
|
||||
|
||||
type RunStep struct {
|
||||
Control *ControlEvidence `json:"control,omitempty"`
|
||||
FileChanges *StageFileChanges `json:"file_changes,omitempty"`
|
||||
@@ -331,12 +309,11 @@ type MemoryRead struct {
|
||||
}
|
||||
|
||||
type ControlEvidence struct {
|
||||
Visit int `json:"visit"`
|
||||
MaxVisits int `json:"max_visits"`
|
||||
LimitReached bool `json:"limit_reached,omitempty"`
|
||||
ModelTarget string `json:"model_target,omitempty"`
|
||||
Memory map[string]MemoryRead `json:"memory,omitempty"`
|
||||
ResourceTarget *ResourceTarget `json:"resource_target,omitempty"`
|
||||
Visit int `json:"visit"`
|
||||
MaxVisits int `json:"max_visits"`
|
||||
LimitReached bool `json:"limit_reached,omitempty"`
|
||||
ModelTarget string `json:"model_target,omitempty"`
|
||||
Memory map[string]MemoryRead `json:"memory,omitempty"`
|
||||
}
|
||||
|
||||
// StageFileChanges is observed workspace evidence, not a model's claim of authorship.
|
||||
|
||||
@@ -1,511 +0,0 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/cli"
|
||||
"git.a-wan.net/alan/pipelines/internal/client"
|
||||
"git.a-wan.net/alan/pipelines/internal/pipeline"
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/stage"
|
||||
"github.com/coder/websocket"
|
||||
"github.com/coder/websocket/wsjson"
|
||||
)
|
||||
|
||||
type fixedRemoteListener struct{ net.Listener }
|
||||
|
||||
func (l fixedRemoteListener) Accept() (net.Conn, error) {
|
||||
conn, err := l.Listener.Accept()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return fixedRemoteConn{Conn: conn}, nil
|
||||
}
|
||||
|
||||
type fixedRemoteConn struct{ net.Conn }
|
||||
|
||||
func (fixedRemoteConn) RemoteAddr() net.Addr {
|
||||
return &net.TCPAddr{IP: net.ParseIP("203.0.113.9"), Port: 443}
|
||||
}
|
||||
|
||||
// This is a real CLI/WebSocket/scratch-SQLite exercise with an explicitly FAKE
|
||||
// executor. It never calls inference, a phone, or a deployed server.
|
||||
func TestResourceScratchCLIWaitingThenProgress(t *testing.T) {
|
||||
t.Setenv("PIPELINES_HARNESS_CONFIG", filepath.Join(t.TempDir(), "harnesses.json"))
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "scratch.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
source := pipeline.Template([]string{"work"}, true) + "```resources\n{\"exclusive\":\"fake-phone\"}\n```\n"
|
||||
if _, err := store.Submit(ctx, "scratch-resource", source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
handler := Handler(ctx, store).(*runHandler)
|
||||
started, release := make(chan struct{}), make(chan struct{})
|
||||
handler.runner.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
|
||||
if input["prompt"] == "A" {
|
||||
close(started)
|
||||
select {
|
||||
case <-release:
|
||||
case <-ctx.Done():
|
||||
return stage.Result{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{"execution": "FAKE executor; no device or inference"}}, nil
|
||||
}
|
||||
handler.runner.events.interval = time.Millisecond
|
||||
events := make(chan protocol.RunEventFrame, 64)
|
||||
sub := handler.runner.events.subscribe("cli-evidence", "", nil, func(m protocol.Message) error {
|
||||
if m.RunEvent != nil {
|
||||
events <- *m.RunEvent
|
||||
}
|
||||
return nil
|
||||
})
|
||||
sub.ready()
|
||||
defer sub.close()
|
||||
srv := httptest.NewServer(handler)
|
||||
defer srv.Close()
|
||||
defer func() { cancel(); WaitRuns(handler) }()
|
||||
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
|
||||
run := func(args ...string) string {
|
||||
var out bytes.Buffer
|
||||
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
|
||||
if err := app.Run(ctx, append([]string{"--url", endpoint}, args...)); err != nil {
|
||||
t.Fatalf("CLI %v: %v\n%s", args, err, out.String())
|
||||
}
|
||||
t.Logf("pipelines %s\n%s", strings.Join(args, " "), out.String())
|
||||
return out.String()
|
||||
}
|
||||
parseRun := func(output string) int64 {
|
||||
var m protocol.Message
|
||||
if err := json.Unmarshal([]byte(output), &m); err != nil || m.RunDetail == nil {
|
||||
t.Fatal("CLI run response", output, err)
|
||||
}
|
||||
return m.RunDetail.ID
|
||||
}
|
||||
run("validate", "testdata/android-resource-draft.md", "--json")
|
||||
a := parseRun(run("run", "scratch-resource", "A", "--workspace", t.TempDir(), "--yes", "--json", "--request-key", "a"))
|
||||
<-started
|
||||
b := parseRun(run("run", "scratch-resource", "B", "--workspace", t.TempDir(), "--yes", "--json", "--request-key", "b"))
|
||||
awaitResourceEvent(t, events, b, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
|
||||
var detail protocol.RunDetail
|
||||
if err := json.Unmarshal([]byte(run("inspect", fmt.Sprint(b), "--json")), &detail); err != nil || detail.ResourceWait == nil || detail.Status != "queued" || len(detail.Steps) != 0 {
|
||||
t.Fatal("CLI did not observe queue", err)
|
||||
}
|
||||
if output := run("runs"); !strings.Contains(output, "Waiting for resource fake-phone") {
|
||||
t.Fatal("list omitted resource status", output)
|
||||
}
|
||||
var snapshot *protocol.RunEventFrame
|
||||
handleRunEvents(protocol.Message{Type: protocol.Subscribe, RunIDs: []int64{b}}, handler.runner.events, handler.runner, "snapshot-evidence", func(m protocol.Message) error {
|
||||
if m.RunEvent != nil {
|
||||
snapshot = m.RunEvent
|
||||
}
|
||||
return nil
|
||||
})
|
||||
handler.runner.events.unsubscribeClient("snapshot-evidence")
|
||||
if snapshot == nil || snapshot.ResourceWait == nil || !snapshot.ResourceWaitSet {
|
||||
t.Fatal("subscription snapshot omitted resource wait")
|
||||
}
|
||||
close(release)
|
||||
run("wait", fmt.Sprint(a), fmt.Sprint(b), "--json")
|
||||
detail = protocol.RunDetail{}
|
||||
if err := json.Unmarshal([]byte(run("inspect", fmt.Sprint(b), "--json")), &detail); err != nil || detail.Status != "succeeded" || detail.ResourceWait != nil || len(detail.Steps) != 1 {
|
||||
t.Fatal("automatic progress not observed", err)
|
||||
}
|
||||
t.Log("Observed queued resource wait followed by automatic progress; FAKE executor only, no phone/inference/deployment.")
|
||||
}
|
||||
|
||||
func TestResourceOperatorTransportCLIRecoveryBoundary(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "operator.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
source := pipeline.Template([]string{"work"}, true) + "```resources\n{\"exclusive\":\"scratch-resource\"}\n```\n"
|
||||
if _, err := store.Submit(ctx, "operator-resource", source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, err := store.List(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
detail, err := store.Get(ctx, items[0].ID, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runID, _, err := store.CreateRun(ctx, detail, "operator-recovery", t.TempDir(), false, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
visit := pipeline.ResourceVisit{Resource: "scratch-resource", RunID: runID, Stage: "work", Sequence: 1}
|
||||
if _, ch, _, err := store.RequestResource(ctx, visit); err != nil || ch != nil {
|
||||
t.Fatalf("acquire resource: channel=%v err=%v", ch, err)
|
||||
}
|
||||
if err := store.RecoverRuns(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
handler := Handler(ctx, store)
|
||||
srv := httptest.NewServer(handler)
|
||||
defer srv.Close()
|
||||
normal, err := client.Open(ctx, "ws"+strings.TrimPrefix(srv.URL, "http")+"/connect")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := normal.Call(ctx, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource}); err == nil || !strings.Contains(err.Error(), "operator transport") {
|
||||
t.Fatalf("ordinary connect accepted recovery: %v", err)
|
||||
}
|
||||
normal.Close()
|
||||
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
|
||||
runCLI := func(args ...string) string {
|
||||
var out bytes.Buffer
|
||||
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
|
||||
if err := app.Run(ctx, append([]string{"--url", endpoint}, args...)); err != nil {
|
||||
t.Fatalf("CLI %v: %v\n%s", args, err, out.String())
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
inspection := runCLI("resource-inspect", visit.Resource, "--json")
|
||||
if strings.Contains(inspection, "token") || strings.Contains(inspection, "credential") {
|
||||
t.Fatalf("inspection leaked capability: %s", inspection)
|
||||
}
|
||||
runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), visit.Stage, fmt.Sprint(visit.Sequence), "--evidence", "verified no mutating action", "--yes", "--json")
|
||||
inspection = runCLI("resource-inspect", visit.Resource, "--json")
|
||||
var recovered protocol.ResourceInspection
|
||||
if err := json.Unmarshal([]byte(inspection), &recovered); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if recovered.Owner != nil || len(recovered.Audits) != 1 || recovered.Audits[0].RunID != runID || recovered.Audits[0].Result != "reconciled" {
|
||||
t.Fatalf("CLI recovery state was not persisted: %s", inspection)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceOperatorCLIRejectsStaleAndNonQuarantinedRecovery(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "operator-rejections.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
if _, err := store.Submit(ctx, "operator-rejections", pipeline.Template([]string{"work"}, true)+"```resources\n{\"exclusive\":\"scratch-resource\"}\n```\n"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, err := store.List(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
detail, err := store.Get(ctx, items[0].ID, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runID, _, err := store.CreateRun(ctx, detail, "operator-rejections", t.TempDir(), false, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
visit := pipeline.ResourceVisit{Resource: "scratch-resource", RunID: runID, Stage: "work", Sequence: 1}
|
||||
lease, ch, _, err := store.RequestResource(ctx, visit)
|
||||
if err != nil || ch != nil {
|
||||
t.Fatalf("acquire resource: channel=%v err=%v", ch, err)
|
||||
}
|
||||
if err := store.RecoverRuns(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
handler := Handler(ctx, store)
|
||||
srv := httptest.NewServer(handler)
|
||||
defer srv.Close()
|
||||
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
|
||||
runCLI := func(args ...string) error {
|
||||
var out bytes.Buffer
|
||||
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
|
||||
err := app.Run(ctx, append([]string{"--url", endpoint}, args...))
|
||||
if strings.Contains(out.String(), "token") || strings.Contains(out.String(), "credential") {
|
||||
t.Fatalf("private capability appeared in CLI output: %s", out.String())
|
||||
}
|
||||
return err
|
||||
}
|
||||
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID+1), visit.Stage, fmt.Sprint(visit.Sequence), "--evidence", "verified", "--yes", "--json"); err == nil {
|
||||
t.Fatal("accepted wrong owner")
|
||||
}
|
||||
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), "stale-stage", fmt.Sprint(visit.Sequence), "--evidence", "verified", "--yes", "--json"); err == nil {
|
||||
t.Fatal("accepted stale stage")
|
||||
}
|
||||
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), visit.Stage, "2", "--evidence", "verified", "--yes", "--json"); err == nil {
|
||||
t.Fatal("accepted stale sequence")
|
||||
}
|
||||
if err := store.SetResourceCleanup(ctx, lease, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := runCLI("resource-reconcile", visit.Resource, fmt.Sprint(runID), visit.Stage, fmt.Sprint(visit.Sequence), "--evidence", "verified", "--yes", "--json"); err == nil {
|
||||
t.Fatal("accepted non-quarantined owner")
|
||||
}
|
||||
inspection, err := store.InspectResource(ctx, visit.Resource)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if inspection.Owner == nil || !inspection.Owner.CleanupConfirmed || len(inspection.Audits) != 0 {
|
||||
t.Fatalf("rejected recovery changed state: owner=%#v audits=%d", inspection.Owner, len(inspection.Audits))
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceOperatorTransportDoesNotTrustRemoteOrForwardedHeaders(t *testing.T) {
|
||||
for _, remote := range []string{"203.0.113.9:443", "198.51.100.7:8080", "not-an-address"} {
|
||||
if isLoopbackPeer(remote) {
|
||||
t.Fatalf("accepted non-loopback peer %q", remote)
|
||||
}
|
||||
}
|
||||
for _, remote := range []string{"127.0.0.1:443", "[::1]:443"} {
|
||||
if !isLoopbackPeer(remote) {
|
||||
t.Fatalf("rejected loopback peer %q", remote)
|
||||
}
|
||||
}
|
||||
// X-Forwarded-For is deliberately not consulted: a proxy-supplied header
|
||||
// cannot turn a remote WebSocket peer into the local operator transport.
|
||||
}
|
||||
|
||||
func TestResourceOperatorTransportRejectsForgedForwardedPeer(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "operator-transport.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
if _, err := store.Submit(ctx, "operator-transport", pipeline.Template([]string{"work"}, true)+"```resources\n{\"exclusive\":\"scratch-resource\"}\n```\n"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, err := store.List(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
detail, err := store.Get(ctx, items[0].ID, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runID, _, err := store.CreateRun(ctx, detail, "operator-transport", t.TempDir(), false, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
visit := pipeline.ResourceVisit{Resource: "scratch-resource", RunID: runID, Stage: "work", Sequence: 1}
|
||||
if _, ch, _, err := store.RequestResource(ctx, visit); err != nil || ch != nil {
|
||||
t.Fatalf("acquire resource: channel=%v err=%v", ch, err)
|
||||
}
|
||||
if err := store.RecoverRuns(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
before, err := store.InspectResource(ctx, visit.Resource)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
server := httptest.NewUnstartedServer(Handler(ctx, store))
|
||||
server.Listener = fixedRemoteListener{Listener: server.Listener}
|
||||
server.Start()
|
||||
defer server.Close()
|
||||
conn, _, err := websocket.Dial(ctx, "ws"+strings.TrimPrefix(server.URL, "http")+"/operator-connect", &websocket.DialOptions{HTTPHeader: map[string][]string{"X-Forwarded-For": {"127.0.0.1"}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.CloseNow()
|
||||
var welcome protocol.Message
|
||||
if err := wsjson.Read(ctx, conn, &welcome); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := wsjson.Write(ctx, conn, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource, ID: "forged"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var refused protocol.Message
|
||||
if err := wsjson.Read(ctx, conn, &refused); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if refused.Type != protocol.Error || !strings.Contains(refused.Error, "operator transport") {
|
||||
t.Fatalf("forged peer was accepted: %#v", refused)
|
||||
}
|
||||
after, err := store.InspectResource(ctx, visit.Resource)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(before, after) {
|
||||
t.Fatalf("forged request mutated recovery state: before=%#v after=%#v", before, after)
|
||||
}
|
||||
|
||||
trusted := httptest.NewServer(Handler(ctx, store))
|
||||
defer trusted.Close()
|
||||
normal, err := client.Open(ctx, "ws"+strings.TrimPrefix(trusted.URL, "http")+"/connect")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := normal.Call(ctx, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource}); err == nil || !strings.Contains(err.Error(), "operator transport") {
|
||||
t.Fatalf("ordinary connect accepted recovery: %v", err)
|
||||
}
|
||||
normal.Close()
|
||||
operator, err := client.Open(ctx, "ws"+strings.TrimPrefix(trusted.URL, "http")+"/operator-connect")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer operator.Close()
|
||||
if _, err := operator.Call(ctx, protocol.Message{Type: protocol.InspectResource, Resource: visit.Resource}); err != nil {
|
||||
t.Fatalf("trusted operator transport refused inspection: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePoolOperatorCLIConfigureReadbackAndPreserveLease(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
dbPath := filepath.Join(t.TempDir(), "pool-operator.db")
|
||||
store, err := pipeline.Open(dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
if err := store.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{{Member: "phone", Enabled: true, Metadata: map[string]string{"serial": "old"}}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := store.Submit(ctx, "pool-owner", pipeline.Template([]string{"work"}, true)+"```resources\n{\"pool\":\"targets\"}\n```\n"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, err := store.List(ctx)
|
||||
if err != nil || len(items) != 1 {
|
||||
t.Fatal("pool pipeline missing", err)
|
||||
}
|
||||
detail, err := store.Get(ctx, items[0].ID, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runID, _, err := store.CreateRun(ctx, detail, "pool-owner", t.TempDir(), false, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lease, ready, _, err := store.RequestResourcePool(ctx, "targets", pipeline.ResourceVisit{Resource: "targets", RunID: runID, Stage: "work", Sequence: 1})
|
||||
if err != nil || ready != nil {
|
||||
t.Fatalf("pool lease: ready=%v err=%v", ready, err)
|
||||
}
|
||||
|
||||
handler := Handler(ctx, store)
|
||||
srv := httptest.NewServer(handler)
|
||||
defer srv.Close()
|
||||
endpoint := "ws" + strings.TrimPrefix(srv.URL, "http") + "/connect"
|
||||
normal, err := client.Open(ctx, endpoint)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := normal.Call(ctx, protocol.Message{Type: protocol.ConfigureResourcePool, ResourcePool: &protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{{Member: "phone", Enabled: true}}}}); err == nil || !strings.Contains(err.Error(), "operator transport") {
|
||||
t.Fatalf("remote configuration was accepted: %v", err)
|
||||
}
|
||||
normal.Close()
|
||||
|
||||
declaration := filepath.Join(t.TempDir(), "pool.json")
|
||||
if err := os.WriteFile(declaration, []byte(`{"name":"targets","members":[{"member":"phone","enabled":true,"metadata":{"serial":"new"}},{"member":"emulator","enabled":false}]}`), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out bytes.Buffer
|
||||
for _, fromStdin := range []bool{false, true} {
|
||||
oversized := strings.Repeat(" ", 256*1024+1)
|
||||
path := filepath.Join(t.TempDir(), "oversized.json")
|
||||
if err := os.WriteFile(path, []byte(oversized), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fromStdin {
|
||||
path = "-"
|
||||
}
|
||||
bounded := cli.App{Out: &out, Err: &out, In: strings.NewReader(oversized)}
|
||||
err := bounded.Run(ctx, []string{"--url", endpoint, "resource-pool-configure", path, "--yes", "--json"})
|
||||
if err == nil || !strings.Contains(err.Error(), "exceeds 262144 bytes") {
|
||||
t.Fatalf("oversized pool input (stdin=%v): %v", fromStdin, err)
|
||||
}
|
||||
out.Reset()
|
||||
}
|
||||
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
|
||||
if err := app.Run(ctx, []string{"--url", "ws" + strings.TrimPrefix(srv.URL, "http"), "resource-pool-configure", declaration, "--yes", "--json"}); err != nil {
|
||||
t.Fatalf("configure CLI: %v\n%s", err, out.String())
|
||||
}
|
||||
out.Reset()
|
||||
app = cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
|
||||
if err := app.Run(ctx, []string{"--url", "ws" + strings.TrimPrefix(srv.URL, "http"), "resource-pool-inspect", "targets", "--json"}); err != nil {
|
||||
t.Fatalf("inspect CLI: %v\n%s", err, out.String())
|
||||
}
|
||||
var pool protocol.ResourcePool
|
||||
if err := json.Unmarshal(out.Bytes(), &pool); err != nil || len(pool.Members) != 2 || pool.Members[0].Metadata["serial"] != "new" {
|
||||
t.Fatalf("bad pool readback: %s (%v)", out.String(), err)
|
||||
}
|
||||
inspection, err := store.InspectResource(ctx, "phone")
|
||||
if err != nil || inspection.Owner == nil || inspection.Owner.RunID != runID {
|
||||
t.Fatalf("configuration released active lease: %#v (%v)", inspection, err)
|
||||
}
|
||||
if err := store.SetResourceCleanup(ctx, lease, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := store.ReleaseResource(ctx, lease); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.Close()
|
||||
store.Close()
|
||||
reopened, err := pipeline.Open(dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reopened.Close()
|
||||
persisted, err := reopened.InspectResourcePool(ctx, "targets")
|
||||
if err != nil || persisted.Members[0].Metadata["serial"] != "new" {
|
||||
t.Fatalf("pool did not survive reopen: %#v (%v)", persisted, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceAndroidDraftPublishedV3Provenance(t *testing.T) {
|
||||
baseline, err := os.ReadFile("testdata/android-v3-published.md")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if hash := fmt.Sprintf("%x", sha256.Sum256(baseline)); hash != "4a3a4b8b947eb20c115c5470e6e378f117d5fa291a7b6c8ae966a3745bc826de" {
|
||||
t.Fatal("baseline is not retrieved Android v3", hash)
|
||||
}
|
||||
draft, err := os.ReadFile("testdata/android-resource-draft.md")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old, err := pipeline.Parse(string(baseline))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
updated, err := pipeline.Parse(string(draft))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(old.Stages) != len(updated.Stages) {
|
||||
t.Fatal("draft changed graph shape")
|
||||
}
|
||||
for i := range old.Stages {
|
||||
a, b := old.Stages[i], updated.Stages[i]
|
||||
if b.Configs["provider"] != "openai" || b.Configs["model"] != a.Configs["model"] {
|
||||
t.Fatal("draft changed approved model")
|
||||
}
|
||||
if b.Key == "ui_reviewer" {
|
||||
if b.Resources == nil || b.Resources.Exclusive != "phone-47241FDAS00DLK" {
|
||||
t.Fatal("reviewer resource missing")
|
||||
}
|
||||
} else if b.Resources != nil {
|
||||
t.Fatal("non-UI stage acquired phone")
|
||||
}
|
||||
a.Prompt, b.Prompt = "", ""
|
||||
a.Resources, b.Resources = nil, nil
|
||||
if !reflect.DeepEqual(a, b) {
|
||||
t.Fatal("draft changed fields outside approved role split/resource", a.Key)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,72 +0,0 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/pipeline"
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/stage"
|
||||
)
|
||||
|
||||
func (r *runner) acquireResource(ctx context.Context, visit pipeline.ResourceVisit, workspace string, completed int) (pipeline.ResourceLease, error) {
|
||||
return r.acquireResourceWithPool(ctx, visit, "", workspace, completed)
|
||||
}
|
||||
|
||||
func (r *runner) acquireResourceWithPool(ctx context.Context, visit pipeline.ResourceVisit, pool string, workspace string, completed int) (pipeline.ResourceLease, error) {
|
||||
var lease pipeline.ResourceLease
|
||||
var ready <-chan pipeline.ResourceLease
|
||||
var wait *protocol.ResourceWait
|
||||
var err error
|
||||
if pool != "" {
|
||||
lease, ready, wait, err = r.store.RequestResourcePool(ctx, pool, visit)
|
||||
} else {
|
||||
lease, ready, wait, err = r.store.RequestResource(ctx, visit)
|
||||
}
|
||||
if err != nil {
|
||||
return lease, err
|
||||
}
|
||||
if ready != nil {
|
||||
r.emitRun(protocol.RunEventFrame{RunID: visit.RunID, Workspace: workspace, Status: protocol.RunStatusQueued, CurrentStage: visit.Stage, CompletedSteps: completed, ResourceWaitSet: true, ResourceWait: wait}, false)
|
||||
select {
|
||||
case lease = <-ready:
|
||||
r.store.AcceptResource(lease)
|
||||
case <-ctx.Done():
|
||||
cleanup, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if err := r.store.CancelResource(cleanup, visit); err != nil {
|
||||
return lease, errors.Join(ctx.Err(), err)
|
||||
}
|
||||
return lease, ctx.Err()
|
||||
}
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
err := r.releaseResource(lease, true) // Never launched, including a concurrent grant.
|
||||
return pipeline.ResourceLease{}, errors.Join(ctx.Err(), err)
|
||||
}
|
||||
r.emitRun(protocol.RunEventFrame{RunID: visit.RunID, Workspace: workspace, ResourceWaitSet: true}, false)
|
||||
return lease, nil
|
||||
}
|
||||
|
||||
func (r *runner) releaseResource(lease pipeline.ResourceLease, confirmed bool) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if err := r.store.SetResourceCleanup(ctx, lease, confirmed); err != nil {
|
||||
return err
|
||||
}
|
||||
if confirmed {
|
||||
return r.store.ReleaseResource(ctx, lease)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *runner) resourceQuiescent(id int64, executionErr error, panicked bool) bool {
|
||||
if panicked || errors.Is(executionErr, stage.ErrStageCleanup) {
|
||||
return false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
sessions, err := r.store.ActiveAgentSessionsForRun(ctx, id)
|
||||
return err == nil && len(sessions) == 0
|
||||
}
|
||||
@@ -1,418 +0,0 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/pipeline"
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/stage"
|
||||
)
|
||||
|
||||
func resourceRunner(t *testing.T) (*runner, <-chan protocol.RunEventFrame, func(string, string, string) *protocol.RunDetail) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s, err := pipeline.Open(filepath.Join(t.TempDir(), "resources.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, err := newRunner(ctx, s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.events = newRunEventHub()
|
||||
r.events.interval = time.Millisecond
|
||||
events := make(chan protocol.RunEventFrame, 128)
|
||||
sub := r.events.subscribe("resource-test", "", nil, func(m protocol.Message) error {
|
||||
if m.RunEvent != nil {
|
||||
events <- *m.RunEvent
|
||||
}
|
||||
return nil
|
||||
})
|
||||
sub.ready()
|
||||
t.Cleanup(func() { cancel(); r.wg.Wait(); sub.close(); s.Close() })
|
||||
count := 0
|
||||
return r, events, func(resource, timeout, prompt string) *protocol.RunDetail {
|
||||
count++
|
||||
source := strings.Replace(pipeline.Template([]string{"work"}, true), "timeout=1h", "timeout="+timeout, 1)
|
||||
if resource != "" {
|
||||
source += "```resources\n{\"exclusive\":\"" + resource + "\"}\n```\n"
|
||||
}
|
||||
name := fmt.Sprint("resource-", count)
|
||||
if _, err := s.Submit(ctx, name, source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := s.List(ctx)
|
||||
var pid int64
|
||||
for _, p := range items {
|
||||
if p.Name == name {
|
||||
pid = p.ID
|
||||
}
|
||||
}
|
||||
d, err := r.start(protocol.Message{PipelineID: pid, Version: 1, RequestKey: name, Workspace: t.TempDir(), Input: map[string]any{"prompt": prompt}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return d
|
||||
}
|
||||
}
|
||||
|
||||
func awaitResourceEvent(t *testing.T, events <-chan protocol.RunEventFrame, id int64, predicate func(protocol.RunEventFrame) bool) protocol.RunEventFrame {
|
||||
t.Helper()
|
||||
timer := time.NewTimer(3 * time.Second)
|
||||
defer timer.Stop()
|
||||
for {
|
||||
select {
|
||||
case event := <-events:
|
||||
if event.RunID == id && predicate(event) {
|
||||
return event
|
||||
}
|
||||
case <-timer.C:
|
||||
t.Fatalf("resource event missing for run %d", id)
|
||||
return protocol.RunEventFrame{}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePauseBeyondTimeoutAndAutomaticResume(t *testing.T) {
|
||||
r, events, start := resourceRunner(t)
|
||||
aStarted, releaseA := make(chan struct{}), make(chan struct{})
|
||||
bStarted := make(chan struct{}, 1)
|
||||
var bCalls atomic.Int32
|
||||
r.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
|
||||
switch input["prompt"] {
|
||||
case "A":
|
||||
close(aStarted)
|
||||
select {
|
||||
case <-releaseA:
|
||||
case <-ctx.Done():
|
||||
return stage.Result{}, ctx.Err()
|
||||
}
|
||||
case "B":
|
||||
bCalls.Add(1)
|
||||
deadline, ok := ctx.Deadline()
|
||||
if !ok || time.Until(deadline) < 15*time.Millisecond {
|
||||
return stage.Result{}, errors.New("execution timeout was consumed in queue")
|
||||
}
|
||||
bStarted <- struct{}{}
|
||||
}
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
a := start("X", "2s", "A")
|
||||
<-aStarted
|
||||
b := start("X", "30ms", "B")
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
|
||||
<-time.After(70 * time.Millisecond) // Deliberately longer than B's whole execution timeout.
|
||||
d, err := r.store.GetRun(context.Background(), b.ID)
|
||||
if err != nil || d.Status != protocol.RunStatusQueued || d.ResourceWait == nil || len(d.Steps) != 0 || bCalls.Load() != 0 {
|
||||
t.Fatal("queued execution consumed accounting", d, err, bCalls.Load())
|
||||
}
|
||||
if _, err := r.decide(protocol.Message{RunID: b.ID, Option: "success"}); err == nil {
|
||||
t.Fatal("resource wait exposed a human decision")
|
||||
}
|
||||
c := start("Y", "1s", "C")
|
||||
awaitResourceEvent(t, events, c.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
|
||||
select {
|
||||
case <-bStarted:
|
||||
t.Fatal("B ran while A held X")
|
||||
default:
|
||||
}
|
||||
owner, err := r.store.GetRun(context.Background(), a.ID)
|
||||
if err != nil || owner.Status != "running" {
|
||||
t.Fatal("A did not remain owner", err)
|
||||
}
|
||||
close(releaseA)
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
|
||||
r.wg.Wait()
|
||||
d, err = r.store.GetRun(context.Background(), b.ID)
|
||||
if err != nil || bCalls.Load() != 1 || d.ResourceWait != nil || len(d.Steps) != 1 || d.Steps[0].Control.Visit != 1 || d.Steps[0].Sequence != 1 {
|
||||
t.Fatal("resume did not execute exactly one fresh visit", d, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcePoolRunnerHandoffUsesConcreteTargetAfterQueue(t *testing.T) {
|
||||
r, events, _ := resourceRunner(t)
|
||||
ctx := context.Background()
|
||||
source := pipeline.Template([]string{"work"}, true) + "```resources\n{\"pool\":\"targets\"}\n```\n"
|
||||
if _, err := r.store.Submit(ctx, "pool-runner", source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, err := r.store.List(ctx)
|
||||
if err != nil || len(items) != 1 {
|
||||
t.Fatal("pool definition missing", err)
|
||||
}
|
||||
definition, err := r.store.Get(ctx, items[0].ID, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.store.ConfigureResourcePool(ctx, protocol.ResourcePool{Name: "targets", Members: []protocol.ResourcePoolMember{
|
||||
{Member: "phone", Enabled: true, Metadata: map[string]string{"serial": "phone-serial", "type": "phone"}},
|
||||
{Member: "emulator", Enabled: true, Metadata: map[string]string{"serial": "emu-serial", "type": "emulator"}},
|
||||
}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ownerRun, _, err := r.store.CreateRun(ctx, definition, "phone-owner", t.TempDir(), false, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
phone, ch, _, err := r.store.RequestResource(ctx, pipeline.ResourceVisit{Resource: "phone", RunID: ownerRun, Stage: "work", Sequence: 1})
|
||||
if err != nil || ch != nil {
|
||||
t.Fatalf("phone owner acquire: channel=%v err=%v", ch, err)
|
||||
}
|
||||
defer r.releaseResource(phone, true)
|
||||
|
||||
aStarted, releaseA := make(chan struct{}), make(chan struct{})
|
||||
bRan := make(chan *protocol.ResourceTarget, 1)
|
||||
r.execute = func(stageCtx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
|
||||
target, ok := stage.ResourceTarget(stageCtx)
|
||||
if !ok || target == nil {
|
||||
return stage.Result{}, errors.New("missing resource target context")
|
||||
}
|
||||
if input["prompt"] == "A" {
|
||||
if target.Member != "emulator" || target.Metadata["serial"] != "emu-serial" || target.Metadata["type"] != "emulator" {
|
||||
return stage.Result{}, fmt.Errorf("wrong first target: %#v", target)
|
||||
}
|
||||
close(aStarted)
|
||||
<-releaseA
|
||||
} else {
|
||||
if deadline, ok := stageCtx.Deadline(); !ok || time.Until(deadline) < 15*time.Millisecond {
|
||||
return stage.Result{}, errors.New("execution deadline was consumed in pool queue")
|
||||
}
|
||||
bRan <- target
|
||||
}
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
start := func(key, prompt string) *protocol.RunDetail {
|
||||
run, err := r.start(protocol.Message{PipelineID: definition.ID, Version: 1, RequestKey: key, Workspace: t.TempDir(), Input: map[string]any{"prompt": prompt}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return run
|
||||
}
|
||||
a := start("pool-a", "A")
|
||||
<-aStarted
|
||||
b := start("pool-b", "B")
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
|
||||
time.Sleep(80 * time.Millisecond)
|
||||
select {
|
||||
case <-bRan:
|
||||
t.Fatal("pool waiter ran before either member was released")
|
||||
default:
|
||||
}
|
||||
if err := r.releaseResource(phone, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var target *protocol.ResourceTarget
|
||||
select {
|
||||
case target = <-bRan:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("pool waiter did not resume")
|
||||
}
|
||||
if target.Member != "phone" || target.Metadata["serial"] != "phone-serial" {
|
||||
t.Fatalf("wrong queued target: %#v", target)
|
||||
}
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
|
||||
close(releaseA)
|
||||
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
|
||||
r.wg.Wait()
|
||||
detail, err := r.store.GetRun(ctx, b.ID)
|
||||
if err != nil || len(detail.Steps) != 1 || detail.Steps[0].Control.ResourceTarget == nil {
|
||||
t.Fatalf("pool target evidence missing: detail=%#v err=%v", detail, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceCancellationWaitsForOwnerCleanup(t *testing.T) {
|
||||
r, events, start := resourceRunner(t)
|
||||
started, cleaning, cleanupDone := make(chan struct{}), make(chan struct{}), make(chan struct{})
|
||||
var waitingCalls atomic.Int32
|
||||
r.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
|
||||
if input["prompt"] == "owner" {
|
||||
close(started)
|
||||
<-ctx.Done()
|
||||
close(cleaning)
|
||||
<-cleanupDone
|
||||
return stage.Result{}, ctx.Err()
|
||||
}
|
||||
waitingCalls.Add(1)
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
a := start("X", "2s", "owner")
|
||||
<-started
|
||||
b := start("X", "30ms", "waiter")
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
|
||||
if err := r.cancel(a.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
<-cleaning
|
||||
select {
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
if waitingCalls.Load() != 0 {
|
||||
t.Fatal("handoff before cleanup")
|
||||
}
|
||||
}
|
||||
close(cleanupDone)
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
|
||||
if waitingCalls.Load() != 1 {
|
||||
t.Fatal("waiter did not resume exactly once")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceQuarantineAndQueuedCancellation(t *testing.T) {
|
||||
for _, failure := range []string{"cleanup", "panic", "restoration"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
r, events, start := resourceRunner(t)
|
||||
started, release := make(chan struct{}), make(chan struct{})
|
||||
var calls atomic.Int32
|
||||
r.execute = func(ctx context.Context, _ stage.Definition, input stage.Values) (stage.Result, error) {
|
||||
if input["prompt"] == "owner" {
|
||||
close(started)
|
||||
select {
|
||||
case <-release:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
if failure == "panic" {
|
||||
panic("unknown cleanup")
|
||||
}
|
||||
if failure == "restoration" {
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{protocol.ResourceCleanupConfirmedKey: false}}, nil
|
||||
}
|
||||
return stage.Result{}, stage.ErrStageCleanup
|
||||
}
|
||||
calls.Add(1)
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
a := start("X", "1s", "owner")
|
||||
<-started
|
||||
b := start("X", "30ms", "waiter")
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.ResourceWait != nil })
|
||||
close(release)
|
||||
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "failed" })
|
||||
if err := r.cancel(b.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool {
|
||||
return e.Status == "canceled" && e.ResourceWaitSet && e.ResourceWait == nil
|
||||
})
|
||||
r.wg.Wait()
|
||||
if calls.Load() != 0 {
|
||||
t.Fatal("quarantined resource handed off")
|
||||
}
|
||||
exists, err := r.store.HasResourceOwner(context.Background(), a.ID)
|
||||
if err != nil || !exists {
|
||||
t.Fatal("quarantine evidence lost", err)
|
||||
}
|
||||
d, err := r.store.GetRun(context.Background(), b.ID)
|
||||
if err != nil || d.ResourceWait != nil || len(d.Steps) != 0 {
|
||||
t.Fatal("canceled queue consumed step", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceEventClearCoalescing(t *testing.T) {
|
||||
frame := protocol.RunEventFrame{ResourceWaitSet: true, ResourceWait: &protocol.ResourceWait{Resource: "X"}}
|
||||
mergeRunEventFrame(&frame, &protocol.RunEventFrame{ResourceWaitSet: true})
|
||||
if !frame.ResourceWaitSet || frame.ResourceWait != nil {
|
||||
t.Fatal("coalescer retained stale resource wait")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceReleaseOnBeginAndPersistenceFailure(t *testing.T) {
|
||||
for _, failure := range []string{"begin", "persist"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
r, events, start := resourceRunner(t)
|
||||
db, err := sql.Open("sqlite", filepath.Join(r.store.Directory(), "resources.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
trigger := `CREATE TRIGGER resource_fail BEFORE INSERT ON run_steps WHEN json_extract(NEW.input_json,'$.prompt')='fail' BEGIN SELECT RAISE(FAIL,'test BeginStep failure'); END`
|
||||
if failure == "persist" {
|
||||
trigger = `CREATE TRIGGER resource_fail BEFORE UPDATE OF output_json ON run_steps WHEN json_extract(OLD.input_json,'$.prompt')='fail' BEGIN SELECT RAISE(FAIL,'test persistence failure'); END`
|
||||
}
|
||||
if _, err := db.Exec(trigger); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var calls atomic.Int32
|
||||
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
|
||||
calls.Add(1)
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
a := start("X", "1s", "fail")
|
||||
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "failed" })
|
||||
r.wg.Wait()
|
||||
if exists, err := r.store.HasResourceOwner(context.Background(), a.ID); err != nil || exists {
|
||||
t.Fatal("safe failure stranded resource", err)
|
||||
}
|
||||
if failure == "begin" && calls.Load() != 0 {
|
||||
t.Fatal("BeginStep failure executed work")
|
||||
}
|
||||
b := start("X", "1s", "pass")
|
||||
awaitResourceEvent(t, events, b.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceCleanupQueryFailureQuarantines(t *testing.T) {
|
||||
r, events, start := resourceRunner(t)
|
||||
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
|
||||
if err := r.store.Close(); err != nil {
|
||||
return stage.Result{}, err
|
||||
}
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
a := start("X", "1s", "query-failure")
|
||||
awaitResourceEvent(t, events, a.ID, func(e protocol.RunEventFrame) bool { return e.Status == "failed" })
|
||||
r.wg.Wait()
|
||||
reopened, err := pipeline.Open(filepath.Join(r.store.Directory(), "resources.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reopened.Close()
|
||||
if err := reopened.RecoverRuns(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if exists, err := reopened.HasResourceOwner(context.Background(), a.ID); err != nil || !exists {
|
||||
t.Fatal("query failure released uncertain owner", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceSkippedVisitDoesNotAcquire(t *testing.T) {
|
||||
r, events, _ := resourceRunner(t)
|
||||
ctx := context.Background()
|
||||
source := strings.Replace(pipeline.Template([]string{"review"}, true), "success -> #success", "success -> #success\nskipped -> #success", 1) + "```resources\n{\"exclusive\":\"X\"}\n```\n"
|
||||
if _, err := r.store.Submit(ctx, "skip", source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := r.store.List(ctx)
|
||||
d, _ := r.store.Get(ctx, items[0].ID, 1)
|
||||
owner, _, err := r.store.CreateRun(ctx, d, "held", t.TempDir(), false, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lease, ch, _, err := r.store.RequestResource(ctx, pipeline.ResourceVisit{Resource: "X", RunID: owner, Stage: "review", Sequence: 1})
|
||||
if err != nil || ch != nil {
|
||||
t.Fatal("owner fixture failed", err)
|
||||
}
|
||||
defer r.releaseResource(lease, true)
|
||||
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
|
||||
t.Error("skipped stage executed")
|
||||
return stage.Result{}, nil
|
||||
}
|
||||
run, err := r.start(protocol.Message{PipelineID: d.ID, Version: 1, RequestKey: "skip", Workspace: t.TempDir(), Input: map[string]any{"prompt": "no UI"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
awaitResourceEvent(t, events, run.ID, func(e protocol.RunEventFrame) bool { return e.Status == "succeeded" })
|
||||
detail, err := r.store.GetRun(ctx, run.ID)
|
||||
if err != nil || len(detail.Steps) != 1 || detail.Steps[0].Outcome != "skipped" {
|
||||
t.Fatal("skip not preserved", detail, err)
|
||||
}
|
||||
}
|
||||
@@ -424,9 +424,6 @@ func (s *runSubscription) close() {
|
||||
}
|
||||
|
||||
func mergeRunEventFrame(into, in *protocol.RunEventFrame) {
|
||||
if in.ResourceWaitSet {
|
||||
into.ResourceWaitSet, into.ResourceWait = true, in.ResourceWait
|
||||
}
|
||||
if in.Workspace != "" {
|
||||
into.Workspace = in.Workspace
|
||||
}
|
||||
@@ -503,20 +500,18 @@ func handleRunEvents(request protocol.Message, events *runEventHub, runs *runner
|
||||
continue
|
||||
}
|
||||
frame := protocol.RunEventFrame{
|
||||
ResourceWaitSet: true,
|
||||
ResourceWait: summary.ResourceWait,
|
||||
RunID: summary.ID,
|
||||
Workspace: summary.Workspace,
|
||||
PipelineID: summary.PipelineID,
|
||||
Name: summary.Name,
|
||||
Title: summary.Title,
|
||||
Version: summary.Version,
|
||||
Status: summary.Status,
|
||||
Outcome: summary.Outcome,
|
||||
CurrentStage: summary.CurrentStage,
|
||||
CompletedSteps: summary.CompletedSteps,
|
||||
CreatedAt: summary.CreatedAt,
|
||||
FinishedAt: summary.FinishedAt,
|
||||
RunID: summary.ID,
|
||||
Workspace: summary.Workspace,
|
||||
PipelineID: summary.PipelineID,
|
||||
Name: summary.Name,
|
||||
Title: summary.Title,
|
||||
Version: summary.Version,
|
||||
Status: summary.Status,
|
||||
Outcome: summary.Outcome,
|
||||
CurrentStage: summary.CurrentStage,
|
||||
CompletedSteps: summary.CompletedSteps,
|
||||
CreatedAt: summary.CreatedAt,
|
||||
FinishedAt: summary.FinishedAt,
|
||||
}
|
||||
if !sub.writeSnapshot(protocol.Message{Type: protocol.RunEvent, RunEvent: &frame}) {
|
||||
// The connection write failed; the subscription is closed and the
|
||||
|
||||
+15
-83
@@ -86,10 +86,6 @@ func newRunner(ctx context.Context, s *pipeline.Store) (*runner, error) {
|
||||
}
|
||||
cancel()
|
||||
if err != nil {
|
||||
if resource, checkErr := s.HasResourceOwner(ctx, owner.RunID); checkErr == nil && resource {
|
||||
log.Printf("resource owner run=%d remains quarantined: cleanup not confirmed", owner.RunID)
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("cannot interrupt previously active agent %s: %w", owner.ID, err)
|
||||
}
|
||||
if err := s.ConfirmRecoveredSession(ctx, owner); err != nil {
|
||||
@@ -224,9 +220,6 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
|
||||
// workspace) must not block new work.
|
||||
for _, owner := range owners {
|
||||
if _, ok := r.active[owner.RunID]; !ok {
|
||||
if resource, checkErr := r.store.HasResourceOwner(r.ctx, owner.RunID); checkErr == nil && resource && owner.Directory != workspace {
|
||||
continue
|
||||
}
|
||||
r.mu.Unlock()
|
||||
return nil, fmt.Errorf("an agent may still be active; restart recovery must interrupt it before new work")
|
||||
}
|
||||
@@ -452,7 +445,7 @@ func retryInputOverride(base map[string]any, override map[string]any) (map[strin
|
||||
// retryStageContractEqual compares the complete persisted stage contract. A
|
||||
// retry may carry completed output across a published version only when the
|
||||
// stages that produced that output are unchanged; comparing only effective
|
||||
// types would allow prompt, config, edge, resource, or limit changes through.
|
||||
// types would allow prompt, config, edge, or limit changes through.
|
||||
func retryStageContractEqual(left, right protocol.Stage) bool {
|
||||
a, err := json.Marshal(left)
|
||||
if err != nil {
|
||||
@@ -1182,14 +1175,13 @@ func (r *runner) finishRunState(id int64, name, workspace, status, outcome, diag
|
||||
// Push the terminal delta before the attention notification, so a slow ntfy
|
||||
// never delays the live update the app folds.
|
||||
r.emitRun(protocol.RunEventFrame{
|
||||
RunID: id,
|
||||
Workspace: workspace,
|
||||
Status: status,
|
||||
ResourceWaitSet: true,
|
||||
Outcome: outcome,
|
||||
CurrentStage: currentStage,
|
||||
CompletedSteps: completed,
|
||||
FinishedAt: time.Now().UTC().Format(time.RFC3339),
|
||||
RunID: id,
|
||||
Workspace: workspace,
|
||||
Status: status,
|
||||
Outcome: outcome,
|
||||
CurrentStage: currentStage,
|
||||
CompletedSteps: completed,
|
||||
FinishedAt: time.Now().UTC().Format(time.RFC3339),
|
||||
}, true)
|
||||
r.notifyRun(id, name, status, sequence, currentStage, diagnostic)
|
||||
// The run is terminal: nothing more can notify for it.
|
||||
@@ -1257,21 +1249,6 @@ func (r *runner) finishStage(id int64, name, workspace string, selected protocol
|
||||
}
|
||||
|
||||
func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, name, workspace string, request protocol.Message, state *runState) {
|
||||
var held *pipeline.ResourceLease
|
||||
resourceQuiet := true
|
||||
release := func() error {
|
||||
if held != nil {
|
||||
err := r.releaseResource(*held, resourceQuiet)
|
||||
held = nil
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
defer func() {
|
||||
if err := release(); err != nil {
|
||||
log.Printf("resource cleanup run=%d: %v", id, err)
|
||||
}
|
||||
}()
|
||||
finish := func(status, outcome, diagnostic string) {
|
||||
r.finishRunState(id, name, workspace, status, outcome, diagnostic, state.completed, state.nextSequence-1, state.key, state.input)
|
||||
}
|
||||
@@ -1336,7 +1313,7 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
maxVisits = selected.Limits.MaxVisits
|
||||
}
|
||||
// Validate the next visit without consuming it. An already-exhausted
|
||||
// stage must not queue for a resource it can never execute.
|
||||
// stage must not begin work it can never execute.
|
||||
if !isHuman {
|
||||
if state.visits[key]+1 > 5 {
|
||||
finish("failed", "error", "stage visit budget exceeded")
|
||||
@@ -1349,27 +1326,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
}
|
||||
_, declaresSkip := def.Edges["skipped"]
|
||||
skip := !isHuman && declaresSkip && !state.uiSurface
|
||||
if selected.Resources != nil && !skip {
|
||||
if isHuman || (selected.Resources.Exclusive == "" && selected.Resources.Pool == "") || (selected.Resources.Exclusive != "" && selected.Resources.Pool != "") || (selected.Resources.Exclusive != "" && !pipeline.ValidResourceKey(selected.Resources.Exclusive)) || (selected.Resources.Pool != "" && !pipeline.ValidResourceKey(selected.Resources.Pool)) {
|
||||
finish("failed", "error", "invalid persisted resource declaration")
|
||||
return
|
||||
}
|
||||
resource := selected.Resources.Exclusive
|
||||
pool := selected.Resources.Pool
|
||||
if pool != "" {
|
||||
resource = pool
|
||||
}
|
||||
lease, err := r.acquireResourceWithPool(ctx, pipeline.ResourceVisit{Resource: resource, RunID: id, Stage: key, Sequence: sequence}, pool, workspace, state.completed)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
finish("canceled", "", err.Error())
|
||||
} else {
|
||||
finish("failed", "error", err.Error())
|
||||
}
|
||||
return
|
||||
}
|
||||
held, resourceQuiet = &lease, true
|
||||
}
|
||||
state.nextSequence = sequence + 1
|
||||
state.visits[key]++
|
||||
// Publish the run's server-captured changed paths into the input memory
|
||||
@@ -1403,13 +1359,12 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
}
|
||||
// queued -> running: report the new current stage and the step start.
|
||||
r.emitRun(protocol.RunEventFrame{
|
||||
RunID: id,
|
||||
Workspace: workspace,
|
||||
Status: protocol.RunStatusRunning,
|
||||
CurrentStage: key,
|
||||
ResourceWaitSet: true,
|
||||
CompletedSteps: state.completed,
|
||||
Step: &protocol.RunStepDelta{Sequence: sequence, Stage: key, Status: protocol.StepStatusRunning},
|
||||
RunID: id,
|
||||
Workspace: workspace,
|
||||
Status: protocol.RunStatusRunning,
|
||||
CurrentStage: key,
|
||||
CompletedSteps: state.completed,
|
||||
Step: &protocol.RunStepDelta{Sequence: sequence, Stage: key, Status: protocol.StepStatusRunning},
|
||||
}, false)
|
||||
if isHuman {
|
||||
// A human step is O(1): one BeginStep plus one MarkWaiting, no
|
||||
@@ -1516,9 +1471,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
var piClient *stage.PiClient
|
||||
var err error
|
||||
stageCtx := ctx
|
||||
if held != nil && held.Target != nil {
|
||||
stageCtx = stage.WithResourceTarget(stageCtx, held.Target)
|
||||
}
|
||||
var modelConfig stage.ModelConfig
|
||||
var budgetBefore stage.BudgetSnapshot
|
||||
if stage.EffectiveType(def.Type) == stage.ModelType {
|
||||
@@ -1542,9 +1494,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
snapshotCancel()
|
||||
var stageSession string
|
||||
panicked := false
|
||||
if held != nil {
|
||||
resourceQuiet = false
|
||||
}
|
||||
func() {
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
@@ -1600,12 +1549,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
result, err = stage.DefaultRegistry().Run(stageCtx, def, input)
|
||||
}
|
||||
}()
|
||||
if held != nil {
|
||||
resourceQuiet = r.resourceQuiescent(id, err, panicked)
|
||||
if confirmed, reported := result.Output[protocol.ResourceCleanupConfirmedKey]; reported && confirmed != true {
|
||||
resourceQuiet = false
|
||||
}
|
||||
}
|
||||
// Capture after cleanup, even when the stage's context was canceled.
|
||||
snapshotCtx, snapshotCancel = context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
|
||||
after := filediff.Capture(snapshotCtx, workspace, before)
|
||||
@@ -1698,9 +1641,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
}
|
||||
step.Target = def.Edges[step.Outcome]
|
||||
step.Control = &protocol.ControlEvidence{Visit: state.visits[key], MaxVisits: maxVisits, Memory: reads}
|
||||
if held != nil && held.Target != nil {
|
||||
step.Control.ResourceTarget = held.Target
|
||||
}
|
||||
if err == nil && selected.Limits != nil && state.visits[key] >= maxVisits {
|
||||
for _, outcome := range selected.Limits.On {
|
||||
if outcome == step.Outcome {
|
||||
@@ -1765,14 +1705,6 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
return
|
||||
}
|
||||
}
|
||||
if held != nil && !resourceQuiet {
|
||||
finish("failed", "error", "resource cleanup/restoration not confirmed; ownership quarantined pending inspection")
|
||||
return
|
||||
}
|
||||
if err := release(); err != nil {
|
||||
finish("failed", "error", "resource release failed; retained ownership needs reconciliation: "+err.Error())
|
||||
return
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
finish("canceled", "", ctx.Err().Error())
|
||||
return
|
||||
|
||||
-312
@@ -1,312 +0,0 @@
|
||||
# stage implementer
|
||||
|
||||
```limits
|
||||
{"max_visits":5,"on":["blocked_retryable"]}
|
||||
```
|
||||
|
||||
|
||||
```memory
|
||||
{"task": "input.prompt", "code_findings": "reviewer.findings?", "ui_findings": "ui_reviewer.findings?"}
|
||||
```
|
||||
|
||||
```output
|
||||
{ task, acceptance_checks, changed_files, implementation_summary, new_helpers }
|
||||
```
|
||||
|
||||
```prompt
|
||||
Implement input.prompt in the run directory using the repository's actual languages and
|
||||
file types; create files as needed.
|
||||
|
||||
Read android/AGENTS.md first. It is the map of this project - how to build, test, run and capture here, where
|
||||
the pieces live, and the conventions that apply. Do not re-derive what it tells you and do not go
|
||||
looking for what it already names; if it is wrong or out of date, say so in your output.
|
||||
|
||||
Read the surrounding code before editing so you extend the existing helpers and patterns rather than
|
||||
duplicating them. Keep each change as small as the task allows and follow the repository's own
|
||||
conventions and error handling.
|
||||
|
||||
This is the Android app. Build and test it with the project's own Gradle invocation as the map
|
||||
gives it, and treat the map's geometry tokens as the only source of dimensions - a UI change that
|
||||
adds a raw value fails the repository's own check. Perform code/build/unit tests only;
|
||||
no adb, install, capture, device-setting changes or restoring main's build. The UI reviewer
|
||||
alone verifies rendered surfaces under its declared runner resource lease.
|
||||
|
||||
Record the acceptance checks in acceptance_checks, the actual changes in changed_files, and any new
|
||||
shared helper in new_helpers. Verify with the checks this repository uses - as the map gives them -
|
||||
and report the exact commands and observed output. Address any concrete findings the runner supplies
|
||||
and note the actual fixes. Return blocked if the required scope or tools are unavailable. Do not
|
||||
commit or push; the committer stage lands the work.
|
||||
```
|
||||
|
||||
```tools
|
||||
read
|
||||
glob
|
||||
grep
|
||||
edit
|
||||
shell
|
||||
```
|
||||
|
||||
```configs
|
||||
provider=openai
|
||||
model=gpt-5.6-terra
|
||||
retries=1
|
||||
timeout=30m
|
||||
```
|
||||
|
||||
|
||||
```edges
|
||||
success -> reviewer
|
||||
blocked_retryable -> implementer
|
||||
blocked -> #failure
|
||||
error -> #failure
|
||||
```
|
||||
|
||||
|
||||
# stage reviewer
|
||||
|
||||
```limits
|
||||
{"max_visits": 3, "on": ["rejected"]}
|
||||
```
|
||||
|
||||
```memory
|
||||
{"task": "input.prompt", "acceptance_checks": "implementer.acceptance_checks", "changed_files": "implementer.changed_files", "implementation": "implementer.implementation_summary?"}
|
||||
```
|
||||
|
||||
|
||||
```output
|
||||
{ summary, changed_files, verification_evidence, duplication_review, performance_review, findings }
|
||||
```
|
||||
|
||||
```prompt
|
||||
Verify with builds, unit tests and source review as android/AGENTS.md gives them.
|
||||
Do not use adb, install, capture, change device settings or restore main's build.
|
||||
Device verification belongs exclusively to the UI reviewer under its runner resource lease.
|
||||
Enforce the map's geometry-token rule: a raw dimension outside the token file is a must-fix finding.
|
||||
|
||||
Review the actual changes against the task in input.prompt and the acceptance
|
||||
checks the implementer recorded. Run the repository's relevant verification checks
|
||||
with the stage tools (for example build, lint, tests or type checks) and report the
|
||||
exact commands and observed results in verification_evidence.
|
||||
|
||||
Independently assess code quality:
|
||||
- Duplication: search the run directory for repeated or near-identical functions,
|
||||
blocks and copy-pasted logic introduced by the change; identify code that should be
|
||||
shared and record concrete locations in duplication_review.
|
||||
- Performance and resources: flag redundant computation, unnecessary dependencies,
|
||||
and repeated I/O or database/filesystem work that wastes time, CPU or memory;
|
||||
record it in performance_review.
|
||||
- Workarounds: flag hacks, suppressed errors and shortcuts that trade reliability for
|
||||
speed in findings.
|
||||
|
||||
Keep verification proportionate to the change: exercise the files and screens the diff
|
||||
actually touches, capture at most one screenshot per changed surface plus one regression
|
||||
screen, and do not repeat sweeps or measurements that an earlier visit already recorded.
|
||||
|
||||
Return rejected ONLY when at least one must-fix finding exists: a defect that blocks
|
||||
acceptance and that you can locate concretely (file and line), with evidence and a
|
||||
remedy. Do not return rejected for advisory-only, cosmetic or nit items. When there
|
||||
is no must-fix finding but you have non-blocking notes (duplication, performance or
|
||||
workaround observations), record them in findings and return advisory, which
|
||||
continues the run without another implementer visit. Return success only when the
|
||||
observed checks pass and you have nothing further to record. Do not edit files or
|
||||
claim unexecuted checks.
|
||||
```
|
||||
|
||||
```tools
|
||||
read
|
||||
glob
|
||||
grep
|
||||
shell
|
||||
```
|
||||
|
||||
```configs
|
||||
provider=openai
|
||||
model=gpt-5.6-luna
|
||||
retries=1
|
||||
timeout=45m
|
||||
```
|
||||
|
||||
|
||||
```edges
|
||||
success -> ui_reviewer
|
||||
advisory -> ui_reviewer
|
||||
rejected -> implementer
|
||||
error -> #failure
|
||||
```
|
||||
|
||||
|
||||
# stage ui_reviewer
|
||||
|
||||
```resources
|
||||
{"exclusive":"phone-47241FDAS00DLK"}
|
||||
```
|
||||
|
||||
```limits
|
||||
{"max_visits": 3, "on": ["rejected"]}
|
||||
```
|
||||
|
||||
```memory
|
||||
{"task": "input.prompt", "acceptance_checks": "implementer.acceptance_checks", "changed_files": "implementer.changed_files", "implementation": "implementer.implementation_summary?"}
|
||||
```
|
||||
|
||||
|
||||
```output
|
||||
{ summary, changed_files, ui_applicable, screenshots, ui_review, platform_conventions, findings }
|
||||
```
|
||||
|
||||
```prompt
|
||||
Decide whether this change touches a user-facing UI. If it does not (a backend,
|
||||
library, CLI, config or data-only change with no rendered surface), set
|
||||
ui_applicable=false, explain briefly in summary, and return skipped. Do not invent
|
||||
screenshots or visual work.
|
||||
|
||||
When the change does affect a user-facing UI, verify it visually:
|
||||
Read android/AGENTS.md first. You alone perform phone work while the runner holds
|
||||
this stage's declared resource lease. Use controlled fixtures for changed states,
|
||||
including human-decision sheets, on a scratch server/database and throwaway session.
|
||||
Never mutate an operator session or restart deployed services. Record original
|
||||
settings and connection/workspace; use the configured authoritative endpoint, not a
|
||||
hardcoded scratch default. Keep every shell/ADB operation foreground and awaited.
|
||||
Restore original settings and main's build before successful return, under the same
|
||||
lease. Report capture/restoration failures explicitly; do not approve incomplete
|
||||
verification or advertise readiness when cleanup/restoration is uncertain.
|
||||
Include resource_cleanup_confirmed in output: true only after confirmed restoration;
|
||||
false on any uncertainty (a release veto that quarantines ownership even if tools are idle).
|
||||
- Build and launch the application the way the repository actually does it, then
|
||||
capture screenshots of every relevant screen, state and viewport (default, empty,
|
||||
loading, error, long content, small and large sizes) with the target platform's
|
||||
screenshot tooling through shell. Prefer a headless or scripted capture when the
|
||||
project supports one. Save the images under the run directory and list the exact
|
||||
capture commands and image paths in screenshots.
|
||||
- Read the captured images and review them as a UI/UX designer: layout, spacing,
|
||||
alignment, typographic hierarchy, color and contrast, iconography, interaction
|
||||
and focus states, loading/empty/error states, accessibility, and consistency.
|
||||
- Judge the UI against established industry patterns and, above all, the native
|
||||
conventions of the platform the app targets (operating system, toolkit or web
|
||||
standards). Record the platform you reviewed against in platform_conventions,
|
||||
and any deviations and concrete fixes in findings. Infer the target platform from
|
||||
the task and the repository's actual UI toolkit when it is unstated.
|
||||
- Approve only when the screenshots show the task and the acceptance checks are met
|
||||
and the UI is correct, consistent and native to the target platform; otherwise
|
||||
return rejected with specific, actionable findings the implementer can fix.
|
||||
|
||||
Keep the capture pass proportionate: capture each changed state once (for example one
|
||||
collapsed and one expanded section, one run state, one stuck header) instead of every
|
||||
screen at multiple viewports or pan directions, and reuse what this run already captured
|
||||
rather than re-doing it.
|
||||
|
||||
If the change is UI-facing but screenshots genuinely cannot be captured
|
||||
(for example no runnable UI or no display), return blocked rather than approving.
|
||||
Do not edit files, and do not approve a UI you did not actually see.
|
||||
|
||||
Return rejected only for a must-fix visual defect with a concrete location, evidence
|
||||
and remedy; record non-blocking polish, duplication or convention notes in findings
|
||||
and return advisory, which continues the run without another implementer visit.
|
||||
```
|
||||
|
||||
```tools
|
||||
read
|
||||
glob
|
||||
grep
|
||||
shell
|
||||
```
|
||||
|
||||
```configs
|
||||
provider=openai
|
||||
model=gpt-5.6-luna
|
||||
retries=1
|
||||
timeout=45m
|
||||
```
|
||||
|
||||
|
||||
```edges
|
||||
skipped -> committer
|
||||
approved -> committer
|
||||
advisory -> committer
|
||||
rejected -> implementer
|
||||
blocked -> #failure
|
||||
error -> #failure
|
||||
```
|
||||
|
||||
# stage committer
|
||||
|
||||
```memory
|
||||
{"task": "input.prompt", "changed_files": "implementer.changed_files", "review": "reviewer.summary?", "ui_review": "ui_reviewer.summary?"}
|
||||
```
|
||||
|
||||
```output
|
||||
{ commit, files, conflicts_resolved, summary }
|
||||
```
|
||||
|
||||
```prompt
|
||||
Commit the work this run produced, and resolve any conflict that is present. This stage
|
||||
runs after the reviews have approved and is the only stage allowed to commit.
|
||||
|
||||
Before committing, check that the change is reflected where it belongs: the packaged
|
||||
templates, the packaged
|
||||
authoring skill references. Update anything the change makes stale, and state in the
|
||||
commit body what you checked, saying explicitly when none of them needs updating.
|
||||
|
||||
Stage exactly the files this run changed - the paths in changed_files that still exist -
|
||||
using the repository's own version control commands, and commit them as one commit whose
|
||||
subject says what the change does and whose body summarises it. Do not stage unrelated
|
||||
files, generated or ignored files, or anything another change owns. Do not invent work
|
||||
that was not done; describe what is actually in the diff.
|
||||
|
||||
If the workspace is in a conflicted state - unmerged paths, or a merge, rebase or
|
||||
cherry-pick left unfinished - resolve each conflict: keep both intents where they are
|
||||
compatible, and choose the result that satisfies the task where they are not, then
|
||||
complete the operation so the tree is clean and the commit is finished. Never commit
|
||||
conflict markers, and never discard the other side's work silently; record what you
|
||||
resolved.
|
||||
|
||||
Do not push, tag, revert, or rewrite unrelated history. If there is nothing to commit,
|
||||
report that instead of creating an empty commit. Report the commit identifier, the files
|
||||
committed, whatever you resolved, and anything you deliberately left uncommitted.
|
||||
|
||||
|
||||
Finally, bring this branch up to date with main, in this same stage. After committing the work
|
||||
above, merge main into this branch here, resolve any conflict here - keeping both intents where they
|
||||
are compatible, and choosing the result that satisfies this task where they are not - run the
|
||||
repository's checks again after resolving, and commit the completed merge with a message naming what
|
||||
was combined. The point is that this branch can later be merged back without conflict: leave the
|
||||
tree clean and the merge finished. Never leave conflict markers, never discard the other side's work
|
||||
silently. If the merge genuinely cannot be completed, say so with the path and the reason instead of
|
||||
leaving the merge half-finished.
|
||||
|
||||
Then land it: this branch is the work and main is where it belongs, so merge this branch into main
|
||||
as the last act of this stage. Find the repository's main working tree with `git worktree list` (the
|
||||
entry checked out on main, not this one) and merge this branch into it there with `--no-ff`, with a
|
||||
message saying what the change does. Two lanes can land at the same moment, so take a lock first -
|
||||
`mkdir` a lock directory beside main's `git` directory in a bounded retry loop, and remove it when
|
||||
done - and retry briefly if another lane holds it. Then verify with
|
||||
`git merge-base --is-ancestor <this branch> main` and report the commit main now has. If main cannot
|
||||
be updated, say exactly why and leave this branch committed and clean rather than leaving main
|
||||
half-merged. There is no separate merge pipeline and nothing to hand this to: if it cannot be landed
|
||||
here, it is reported here.
|
||||
|
||||
|
||||
Do not edit README or other shared documentation files: an item records what it changed and which documentation is now stale in its own commit body, and the shared documents are folded by a single writer.
|
||||
|
||||
```
|
||||
|
||||
```tools
|
||||
read
|
||||
glob
|
||||
grep
|
||||
edit
|
||||
shell
|
||||
```
|
||||
|
||||
```configs
|
||||
provider=openai
|
||||
model=gpt-5.6-terra
|
||||
retries=1
|
||||
timeout=30m
|
||||
```
|
||||
|
||||
```edges
|
||||
success -> #success
|
||||
blocked -> #failure
|
||||
error -> #failure
|
||||
```
|
||||
@@ -2,9 +2,7 @@ package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -270,44 +268,6 @@ func HandlerWithOptions(ctx context.Context, options Options, stores ...*pipelin
|
||||
response.Error = err.Error()
|
||||
}
|
||||
}
|
||||
} else if (request.Type == protocol.InspectResource || request.Type == protocol.ReconcileResource || request.Type == protocol.InspectResourcePool || request.Type == protocol.ConfigureResourcePool) && len(stores) > 0 {
|
||||
var err error
|
||||
response.Type = protocol.ResourceInspectionResult
|
||||
// Recovery has a separate transport path. A stage/model client may
|
||||
// also be local, so loopback alone is not an authority boundary.
|
||||
// The CLI uses /operator-connect; the ordinary /connect path is
|
||||
// deliberately refused even from loopback.
|
||||
if r.URL.Path != "/operator-connect" || !isLoopbackPeer(r.RemoteAddr) {
|
||||
response.Type = protocol.Error
|
||||
response.Error = "resource recovery requires the local operator transport"
|
||||
} else if request.Type == protocol.InspectResourcePool {
|
||||
response.Type = protocol.ResourcePoolInspection
|
||||
var pool protocol.ResourcePool
|
||||
pool, err = stores[0].InspectResourcePool(ctx, request.Resource)
|
||||
response.ResourcePool = &pool
|
||||
} else if request.Type == protocol.ConfigureResourcePool {
|
||||
response.Type = protocol.ResourcePoolConfigured
|
||||
if request.ResourcePool == nil {
|
||||
err = fmt.Errorf("resource pool declaration is required")
|
||||
} else {
|
||||
err = stores[0].ConfigureResourcePool(ctx, *request.ResourcePool)
|
||||
}
|
||||
} else if request.Type == protocol.InspectResource {
|
||||
var inspection protocol.ResourceInspection
|
||||
inspection, err = stores[0].InspectResource(ctx, request.Resource)
|
||||
response.ResourceInspection = &inspection
|
||||
} else {
|
||||
response.Type = protocol.ResourceReconciled
|
||||
if request.ResourceRecovery == nil {
|
||||
err = fmt.Errorf("resource recovery evidence is required")
|
||||
} else {
|
||||
err = stores[0].ReconcileResource(ctx, *request.ResourceRecovery)
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
response.Type = protocol.Error
|
||||
response.Error = err.Error()
|
||||
}
|
||||
} else if runs != nil && (request.Type == protocol.RunPipeline || request.Type == protocol.RetryRun || request.Type == protocol.ListRuns || request.Type == protocol.GetRun || request.Type == protocol.CancelRun || request.Type == protocol.DecideRun || request.Type == protocol.SetRunTitle || request.Type == protocol.GetRunEvidence) {
|
||||
var err error
|
||||
switch request.Type {
|
||||
@@ -365,19 +325,9 @@ func HandlerWithOptions(ctx context.Context, options Options, stores ...*pipelin
|
||||
}
|
||||
}
|
||||
mux.HandleFunc("GET /connect", connect)
|
||||
mux.HandleFunc("GET /operator-connect", connect)
|
||||
return &runHandler{Handler: mux, runner: runs}
|
||||
}
|
||||
|
||||
func isLoopbackPeer(remoteAddr string) bool {
|
||||
host, _, err := net.SplitHostPort(remoteAddr)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
ip := net.ParseIP(host)
|
||||
return ip != nil && ip.IsLoopback()
|
||||
}
|
||||
|
||||
type runHandler struct {
|
||||
http.Handler
|
||||
runner *runner
|
||||
|
||||
@@ -8,8 +8,6 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
var ErrInvalidModelResult = errors.New("invalid model stage result")
|
||||
@@ -38,13 +36,6 @@ func harnessUnreachable(err error) error {
|
||||
}
|
||||
|
||||
func modelPrompt(def Definition, spec ModelSpec, input Values) (string, error) {
|
||||
return modelPromptForTarget(def, spec, input, nil)
|
||||
}
|
||||
|
||||
// modelPromptForTarget carries a server-selected resource target separately from
|
||||
// task input. The target comes only from the runner context; task input remains
|
||||
// data and cannot select or replace it.
|
||||
func modelPromptForTarget(def Definition, spec ModelSpec, input Values, target *protocol.ResourceTarget) (string, error) {
|
||||
outcomes := make([]string, 0, len(def.Edges))
|
||||
for name := range def.Edges {
|
||||
outcomes = append(outcomes, name)
|
||||
@@ -68,16 +59,15 @@ func modelPromptForTarget(def Definition, spec ModelSpec, input Values, target *
|
||||
toolsLine = "This call has these stage tools: " + strings.Join(tools, ", ") + "."
|
||||
}
|
||||
data, err := json.Marshal(struct {
|
||||
Key string `json:"stage"`
|
||||
Instructions string `json:"instructions"`
|
||||
OutputContract string `json:"output_contract"`
|
||||
Outcomes []string `json:"allowed_outcomes"`
|
||||
Input Values `json:"input"`
|
||||
ResourceTarget *protocol.ResourceTarget `json:"resource_target,omitempty"`
|
||||
ReportFileArtifacts bool `json:"report_file_artifacts"`
|
||||
ExecutionTime string `json:"execution_time"`
|
||||
Timezone string `json:"timezone"`
|
||||
}{def.Key, spec.Prompt, spec.Output, outcomes, input, target, def.ReportFileArtifacts, time.Now().Format(time.RFC3339), time.Local.String()})
|
||||
Key string `json:"stage"`
|
||||
Instructions string `json:"instructions"`
|
||||
OutputContract string `json:"output_contract"`
|
||||
Outcomes []string `json:"allowed_outcomes"`
|
||||
Input Values `json:"input"`
|
||||
ReportFileArtifacts bool `json:"report_file_artifacts"`
|
||||
ExecutionTime string `json:"execution_time"`
|
||||
Timezone string `json:"timezone"`
|
||||
}{def.Key, spec.Prompt, spec.Output, outcomes, input, def.ReportFileArtifacts, time.Now().Format(time.RFC3339), time.Local.String()})
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("model stage input is not JSON-serializable: %w", err)
|
||||
}
|
||||
|
||||
@@ -16,8 +16,7 @@ func (m ModelStage) Run(ctx context.Context, def Definition, input Values) (Resu
|
||||
return Result{}, err
|
||||
}
|
||||
spec.Config.ResultSchema = resultSchema(def.Edges)
|
||||
target, _ := ResourceTarget(ctx)
|
||||
prompt, err := modelPromptForTarget(def, spec, input, target)
|
||||
prompt, err := modelPrompt(def, spec, input)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
|
||||
@@ -9,8 +9,6 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
func modelDefinition() Definition {
|
||||
@@ -196,107 +194,6 @@ func TestModelRunUsesInstructionsAndStructuredOutput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestModelRunForwardsOnlyServerGrantedResourceTarget(t *testing.T) {
|
||||
target := &protocol.ResourceTarget{Member: "emulator", Metadata: map[string]string{"serial": "emu-5554", "type": "emulator"}}
|
||||
budget := NewBudget(0, 0)
|
||||
input := Values{"resource_target": map[string]any{"member": "user-selected"}, "prompt": "task"}
|
||||
ctx := WithBudget(WithResourceTarget(context.Background(), target), budget)
|
||||
|
||||
var received ModelConfig
|
||||
client := generatorFunc(func(_ context.Context, config ModelConfig, prompt string) (string, error) {
|
||||
received = config
|
||||
assertPromptResourceTarget(t, prompt, target, input)
|
||||
return `{"outcome":"success","output":{}}`, nil
|
||||
})
|
||||
if _, err := (ModelStage{Client: client}).Run(ctx, modelDefinition(), input); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if received.Budget != budget {
|
||||
t.Fatal("budget context was not preserved with resource target")
|
||||
}
|
||||
}
|
||||
|
||||
func TestModelRunForwardsResourceTargetToNativeOpenCode(t *testing.T) {
|
||||
target := &protocol.ResourceTarget{Member: "phone", Metadata: map[string]string{"serial": "phone-serial"}}
|
||||
var prompt string
|
||||
client := &OpenCodeClient{Directory: "/work/repo", command: func(_ context.Context, _ string, args ...string) ([]byte, error) {
|
||||
switch args[2] {
|
||||
case "/api/session":
|
||||
return []byte(`{"data":{"id":"ses_target"}}`), nil
|
||||
case "/api/session/ses_target/generate":
|
||||
var body struct {
|
||||
Prompt string `json:"prompt"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(args[4]), &body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prompt = body.Prompt
|
||||
return []byte(`{"data":{"text":"{\"outcome\":\"success\",\"output\":{}}"}}`), nil
|
||||
default:
|
||||
t.Fatalf("unexpected fake OpenCode endpoint %q", args[2])
|
||||
return nil, nil
|
||||
}
|
||||
}}
|
||||
if _, err := (ModelStage{Client: client}).Run(WithResourceTarget(context.Background(), target), modelDefinition(), Values{"prompt": "task"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assertPromptResourceTarget(t, prompt, target, Values{"prompt": "task"})
|
||||
}
|
||||
|
||||
func TestModelPromptWithoutResourceTargetPreservesLegacyInput(t *testing.T) {
|
||||
input := Values{"prompt": "task", "count": "1"}
|
||||
spec, err := (ModelStage{}).Spec(modelDefinition())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prompt, err := modelPrompt(modelDefinition(), spec, input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var payload struct {
|
||||
Input Values `json:"input"`
|
||||
ResourceTarget *protocol.ResourceTarget `json:"resource_target"`
|
||||
}
|
||||
if _, data, ok := strings.Cut(prompt, "Stage definition and runtime input:\n"); !ok || json.Unmarshal([]byte(data), &payload) != nil {
|
||||
t.Fatal("could not decode legacy runtime payload")
|
||||
}
|
||||
if payload.ResourceTarget != nil || !reflect.DeepEqual(payload.Input, input) {
|
||||
t.Fatalf("legacy runtime input changed: %#v", payload)
|
||||
}
|
||||
}
|
||||
|
||||
func assertPromptResourceTarget(t *testing.T, prompt string, want *protocol.ResourceTarget, wantInput Values) {
|
||||
t.Helper()
|
||||
_, data, ok := strings.Cut(prompt, "Stage definition and runtime input:\n")
|
||||
if !ok {
|
||||
t.Fatal("missing runtime payload")
|
||||
}
|
||||
var payload struct {
|
||||
Input Values `json:"input"`
|
||||
ResourceTarget *protocol.ResourceTarget `json:"resource_target"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(data), &payload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(payload.ResourceTarget, want) {
|
||||
t.Fatalf("resource target = %#v, want %#v", payload.ResourceTarget, want)
|
||||
}
|
||||
var fields map[string]json.RawMessage
|
||||
if err := json.Unmarshal([]byte(data), &fields); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var targetFields map[string]json.RawMessage
|
||||
if err := json.Unmarshal(fields["resource_target"], &targetFields); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(targetFields) != 2 || targetFields["member"] == nil || targetFields["metadata"] == nil {
|
||||
t.Fatalf("resource target leaked non-target fields: %s", fields["resource_target"])
|
||||
}
|
||||
if !reflect.DeepEqual(payload.Input, wantInput) {
|
||||
t.Fatalf("input = %#v, want %#v", payload.Input, wantInput)
|
||||
}
|
||||
}
|
||||
|
||||
func TestModelResultValidationAndRetryBudget(t *testing.T) {
|
||||
for _, text := range []string{
|
||||
`not json`, `{"outcome":"missing","output":{}}`, `{"outcome":"success"}`,
|
||||
|
||||
@@ -240,7 +240,7 @@ func (c *PiClient) Generate(ctx context.Context, config ModelConfig, prompt stri
|
||||
close(heartbeatDone)
|
||||
// agent_end/abort acknowledgement and the parent's exit alone do not
|
||||
// prove its shell descendants exited. A surviving process group retains
|
||||
// ownership rather than handing the resource to another stage.
|
||||
// ownership rather than handing it to another stage.
|
||||
quiet = quiet && errors.Is(syscall.Kill(-c.cmd.Process.Pid, 0), syscall.ESRCH)
|
||||
if quiet {
|
||||
_ = os.WriteFile(filepath.Join(c.Reference, "quiescent"), []byte("quiescent"), 0600)
|
||||
|
||||
@@ -12,7 +12,7 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestResourcePiReceiptRequiresProcessExit(t *testing.T) {
|
||||
func TestPiReceiptRequiresProcessExit(t *testing.T) {
|
||||
// A test-owned process only: no Pi provider, shell/ADB command or phone.
|
||||
cmd := exec.Command("sleep", "5")
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
@@ -5,34 +5,8 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
type resourceTargetContextKey struct{}
|
||||
|
||||
// WithResourceTarget attaches the server-granted concrete target to a stage's
|
||||
// runtime context. It is deliberately separate from model input and contains
|
||||
// no capability or credential material.
|
||||
func WithResourceTarget(ctx context.Context, target *protocol.ResourceTarget) context.Context {
|
||||
if target == nil {
|
||||
return ctx
|
||||
}
|
||||
copy := &protocol.ResourceTarget{Member: target.Member}
|
||||
if target.Metadata != nil {
|
||||
copy.Metadata = map[string]string{}
|
||||
for key, value := range target.Metadata {
|
||||
copy.Metadata[key] = value
|
||||
}
|
||||
}
|
||||
return context.WithValue(ctx, resourceTargetContextKey{}, copy)
|
||||
}
|
||||
|
||||
func ResourceTarget(ctx context.Context) (*protocol.ResourceTarget, bool) {
|
||||
target, ok := ctx.Value(resourceTargetContextKey{}).(*protocol.ResourceTarget)
|
||||
return target, ok
|
||||
}
|
||||
|
||||
// Values are runtime data, not the prose output contract in a definition.
|
||||
type Values map[string]any
|
||||
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
package tui
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
"github.com/charmbracelet/x/ansi"
|
||||
"github.com/muesli/termenv"
|
||||
)
|
||||
|
||||
func TestResourceWaitFrames(t *testing.T) {
|
||||
previous := lipgloss.ColorProfile()
|
||||
defer lipgloss.SetColorProfile(previous)
|
||||
for _, width := range []int{60, 120} {
|
||||
for _, plain := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("width-%d-plain-%t", width, plain), func(t *testing.T) {
|
||||
if plain {
|
||||
lipgloss.SetColorProfile(termenv.Ascii)
|
||||
} else {
|
||||
lipgloss.SetColorProfile(termenv.TrueColor)
|
||||
}
|
||||
m := decisionModel()
|
||||
m.width = width
|
||||
m.runDetail.Status = protocol.RunStatusQueued
|
||||
m.runDetail.Steps = nil
|
||||
m.runDetail.ResourceWait = &protocol.ResourceWait{Resource: "phone-X", Stage: "review", Sequence: 1, QueuedSince: "2026-10-18T10:00:00Z"}
|
||||
frame := ansi.Strip(m.runsDetailsView(24))
|
||||
if !strings.Contains(frame, "Waiting for resource phone-X") || strings.Contains(frame, "c decide") || strings.Contains(frame, "Waiting for a person") {
|
||||
t.Fatal("wrong resource wait frame:\n", frame)
|
||||
}
|
||||
for _, line := range strings.Split(frame, "\n") {
|
||||
if ansi.StringWidth(line) > width {
|
||||
t.Fatalf("overflow: %q", line)
|
||||
}
|
||||
}
|
||||
t.Log("Rendered resource wait (no phone):\n" + frame)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -11,9 +11,6 @@ import (
|
||||
)
|
||||
|
||||
func resultLabel(r protocol.RunSummary) string {
|
||||
if r.ResourceWait != nil {
|
||||
return r.StatusLabel() + " · stage " + r.ResourceWait.Stage
|
||||
}
|
||||
preview := ansi.Strip(r.ResultPreview)
|
||||
if r.Status == "succeeded" {
|
||||
if preview != "" {
|
||||
|
||||
@@ -276,9 +276,6 @@ func runStatus(status string) string {
|
||||
}
|
||||
|
||||
func (m Model) runSummaryStatus(run protocol.RunSummary) string {
|
||||
if run.ResourceWait != nil {
|
||||
return run.StatusLabel()
|
||||
}
|
||||
return m.progress(run.Status)
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user