Merge main into pill-run-decision for Android recovery

This commit is contained in:
2026-10-18 15:06:52 +01:00
85 changed files with 6293 additions and 178 deletions
+37
View File
@@ -0,0 +1,37 @@
# Background shell visibility
The chat server now opens the broker stream for read-only subscribers and reconciles verified running shell operations from OpenCode's authoritative `/api/shell` readback. The packaged OpenCode skill references and chat documentation should describe that shell visibility survives reconnects only when OpenCode reports explicit `metadata.sessionID` ownership; shared working directories are not ownership evidence.
## Current source behavior
Observers share one initial readback per room/stream attachment. The adapter and hub
fold bounded lifecycle journals over each snapshot, so a stale read cannot resurrect
an exited shell or erase a concurrent create. Shell replay and live delivery enqueue
in fold order. Cancellation rejects old-room callbacks and releases local identity
state before reconnect. Journals retain at most 256 identities and 1 MiB of id/command
text; capacity/read failures surface stable limited-visibility text, never broker
response bodies. No shell polling was added. OpenCode's terminal `timeout` and
`killed` statuses are excluded from retained running operations as well as `exited`.
## Verification evidence — 2026-10-18
`opencode api get /openapi.json` authenticated through supported service discovery.
The live schema advertises shell list/detail/output and generic `metadata`; it does
not type `metadata.sessionID`. A throwaway session
`ses_eb12ab7b3ffe0ZAle7PHpftHMG` and its own `sleep 1` shell
`sh_14ed57134001e6fuYqvn62pKit` verified actual create/detail readback preserving
explicit `metadata.sessionID`, status `running`, subsequent `exited`/exit 0, and
absence from the running list. This verifies explicit API metadata, not automatic
agent-tool ownership injection. The CLI event attempt yielded no captured events;
live-provider event ordering is **not tested**. Forced ordering evidence comes from
channel/barrier regressions, including a real WebSocket handler on a loopback scratch
server with fresh SQLite, not from the deployed service.
## Documentation follow-up / deployment
`docs/chat-verification.md` and the chat subscribe/replay section of `docs/cli.md`
are stale: docs-fold should describe authoritative ownership/readback, coalesced
observation, bounded reconciliation and limited-visibility errors, distinguishing
fixture ordering evidence from the live readback above. Shared docs and packaged
skill references were not edited here. No deployment, installation, service restart
or physical-phone/UI verification was performed; this changes no rendered layout.
+47
View File
@@ -0,0 +1,47 @@
# Delivery stabilization
Date: 2026-10-18.
The server now has draft-only `delivery=true` support for OpenCode model stages.
At run time it derives the canonical `main` worktree and linked-worktree Git
metadata from Git rather than accepting paths from a stage prompt. It verifies
the committed item tree in a disposable candidate worktree before fast-forwarding
canonical `main`; declared checks are bounded configuration, not proof that any
deployed definition has used them. It is denied for Pi (until equivalent
capability enforcement exists), foreign repositories, missing main worktrees,
and malformed capabilities. This item does not publish or deploy a definition.
The source also supports an optional controlled-fixture verification declaration
(`backend`, `revision`, `capability`, and `fixture`). Ordinary server setup wires
an explicit, bounded `reconnect-foreground` HTTP fixture registry when
`PIPELINES_VERIFICATION_RECONNECT_FOREGROUND_URL` and its expected `BACKEND`,
`REVISION`, `CAPABILITY`, and `STATE` values are configured. There is no default
remote URL. The server, not the model, probes that registered endpoint and
collects its returned backend identity, build revision, capability, fixture ID,
and state before comparing provenance. Unknown fixtures, incomplete setup,
wrong backend/revision/capability/state, non-200 responses, cancellation, and
oversized or malformed responses fail closed. A model result's labels are not
evidence. This is controlled local/backend fixture infrastructure, not evidence
of a rendered layout or a live Android device.
Definitions must be scratch-validated against the deployed server only after
the server support is deployed; no future delivery or verification definition
is activated by this source change.
`internal/author/skill/references/format.md` and
`internal/author/skill/references/queue.md` are stale: a later docs fold should
describe `delivery=true`, OpenCode-only availability, candidate verification,
and the required activation order (deploy server support before scratch-validating
and then publishing a definition).
OpenCode V2 resolves external resources at a containing directory, so delivery
uses the verified repository metadata directory rather than an impossible
single-lock-file scope. It does not grant a parent, home, foreign repository,
or global policy. Unrestricted shell permissions are not an OS sandbox.
Activation remains draft-only: deploy compatible server support, configure the
controlled fixture endpoint and matching provenance, scratch-validate the
fixture, then publish a definition. No definition is activated by this source
change.
Recovery checklist: retain `connection-reconnect` and unfinished feature
branches; merge only complete committed trees that pass mapped checks; do not
infer delivery from a dirty worktree or an old deployed-backend capture.
+95
View File
@@ -0,0 +1,95 @@
# Android: coming back to the app restores the connection
## What this item changed
Android app only.
- `data/transport/ConnectionRecovery.kt` (new): `ForegroundAction` and the one
pure `foregroundAction(state, confirmedAlive)` policy — `Idle` → leave alone (a
deliberate Disconnect is the only way the client reaches Idle), `Connected` with
a confirmed-alive socket → leave alone, `Connected` unconfirmed → reconnect,
`Connecting`/`Disconnected` → reconnect.
- `data/ForegroundReconnector.kt` (new): the one lifecycle-aware owner. It is
created once with the Application and observes `ProcessLifecycleOwner`, never
the Activity or the ViewModel (both recreated). On `ON_START` it applies the
policy: a non-Connected state reconnects at once, and the reconnect cancels the
client's pending backoff delay, so the operator never waits it out; a state
that still claims `Connected` is first confirmed with the existing `ping`,
bounded to 3 s, and the state is re-read afterwards (the socket's own failure
callback can land while the check is in flight).
- `PipelinesApplication.kt`: wires the owner to the process lifecycle;
`confirmAlive` is `repository.ping()`, `reconnect` is
`ServerUrl.validate(...) + repository.recover(...)`. Recovery shares the
client's state lock with Disconnect, so a Disconnect after the probe has made
its decision but before its reconnect effect runs stays authoritative. Manual
Connect/Reconnect continue to use `repository.connect(...)`; the client's
bounded backoff stays the only reconnect mechanism and the policy lives in the
one pure place.
- `data/transport/PipelinesClient.kt`: `open()` now closes the socket it
supersedes, and every `WebSocketListener` callback ignores a socket that is no
longer the current one (`isCurrent`). A failure the replaced socket reports late
— the common case when the process was frozen while it died — can then neither
clear the new connection nor schedule a reconnect that would undo the recovery.
- `app/build.gradle.kts` and `gradle/libs.versions.toml`:
`androidx.lifecycle:lifecycle-process` 2.8.7 (already in the offline cache,
with its `androidx.startup`/`androidx.tracing` transitives).
- Tests: `data/transport/ConnectionRecoveryTest.kt` (5 cases) and
`data/ForegroundReconnectorTest.kt` (8 cases).
## Shared documents this change makes stale
`android/README.md`.
- The Configuration section (`android/README.md:99-101`) — "Tap **Connect** to
open the single shared WebSocket (probed with `ping`, showing the server
build). Reconnect uses bounded exponential backoff; **Disconnect** stops it." —
should record that coming back to the app restores the connection on its own:
the process coming to the front reconnects immediately, cancelling the pending
backoff delay, and when the last state still claims Connected it first confirms
the socket with `ping` before trusting it. An explicit Disconnect stays
authoritative and is never reconnected over; the manual Reconnect control
stays, but normal recovery never needs it.
- The Architecture block (`android/README.md:310-323`) lists `data/transport` and
`data`. It should name the two new files — `data/transport/ConnectionRecovery.kt`
(the pure return-to-foreground policy) and `data/ForegroundReconnector.kt` (the
process-lifecycle owner) — and the new `androidx.lifecycle:lifecycle-process`
dependency behind the owner.
## Not stale
- `android/AGENTS.md` names neither the presence nor the absence of a lifecycle
observer; its "Where things are" summary stays correct (it could optionally add
the new owner beside `data/PipelinesRepository.kt`).
- `docs/app-integration-verification.md` — its connection mentions are about
restoring the stub URL, which this item does not change.
- No packaged OpenCode skill reference (`internal/author/skill/references/`)
documents the app's connection lifecycle, so nothing there is stale.
## Verified on the device, and what was not
Built and captured on the operator's Pixel 9 Pro XL (`47241FDAS00DLK`) from
`51a2ba4` + this change; after merging main the same tree (now `b93eee2`) was
rebuilt and the full unit suite re-run (691 tests, 0 failures). The operator's
`ws://10.0.0.75:8080/connect` and `/Users/alan/repos/pipelines` were used
throughout and were not changed. The phone was left on main's build.
Captured, all with `android/scripts/capture_png.sh` (every image validated):
- a **live** socket on return: the server log shows a `ping`/`pong` on the same
connection at the second of the return and no reconnect — the positive check
runs and a healthy connection is left alone;
- a **dead** socket on return: the reconnect and re-subscribe land in the same
second as the return, with no interaction (the process was frozen, so the
client's own backoff had not run);
- **no network** on return: the banner reads "Disconnected: Failed to connect to
…:8080 · retrying with backoff" — never a false Connected — and recovery
follows on its own once the network returns;
- an explicit **Disconnect**: still "Not connected. Enter the server URL in
Settings, then connect." after returning, with no reconnect in the server log.
**Not verified live:** the stale-Connected probe branch. In every capture the
frozen socket's own failure was delivered to the client before the lifecycle
owner ran, so the client already read `Disconnected` and no probe was attempted
(confirmed by the client's request-id sequence never skipping a value in the
server log). That branch — `Connected` still claimed, probe fails, reconnect — is
covered by unit tests only.
+158
View File
@@ -0,0 +1,158 @@
# Stage-exclusive resources — 2026-10-18, not deployed
This is source work for run 355, **not a deployment, installation or publication**.
No physical phone, real operator session or deployed service was used or changed.
The Android role split does not retrospectively gate pinned versions or existing runs.
## Contract and scope
An executable non-human stage can declare one runner-owned control block:
```resources
{"exclusive":"phone-47241FDAS00DLK"}
```
The key is 1–128 ASCII identifier bytes (letter first, then letters, digits, `_`, `-`).
Duplicate/unknown fields, multiple resources and human-stage resources are rejected.
One capacity-one resource per stage visit, one server/store authority; no expressions,
nested acquisition, distributed scheduling, capacity settings or multi-resource requests.
Resource-free definitions retain their execution path. Old servers reject this new block;
do not publish to an incompatible server.
Acquisition follows selection and automatic UI-skip evaluation, before BeginStep,
visits/sequences, artifacts, filediff, stage timeout, budget setup or executor calls.
Queued runs retain workspace reservation, use legacy `queued` (never human `waiting`),
and carry bounded `resource_wait` identity/stage/sequence/queued-since metadata.
Events use `resource_wait_set=true` with absent `resource_wait` to clear caches.
CLI lists and TUI/Android details show “Waiting for resource …”; human decisions and
attention notifications remain separate. Older clients still see an unfinished run.
SQLite owners and FIFO waiters are indexed by resource/order and run identity.
Private random capabilities attribute ownership to resource/run/stage/sequence and
never enter wire data, model input, transcripts or artifacts. Only the full capability
and positive cleanup evidence authorize release; mismatches and repeated release cannot
release a subsequent owner. One buffered channel per live queued run, no resource
polling, waiter tickers, model-based acquisition, queue driver or new daemon.
The gate holds through tool cleanup. OpenCode interruption now confirms session idle/
V1 settlement before clearing session ownership. Pi recovery requires both its receipt
and process/process-group exit, not merely abort acknowledgement. Executor panic,
ErrStageCleanup, retained active sessions and failed cleanup queries quarantine the owner.
An explicit `output.resource_cleanup_confirmed=false` (or malformed value) also vetoes
release for failed restoration; true never substitutes for harness cleanup evidence.
The generic runner does not inspect Android settings or magically verify restoration.
The Android UI reviewer must keep all device work foreground/awaited, restore original
settings/connection and main's build, and report uncertainty with the release veto.
## Restart and retention
Automatic FIFO progress is promised only within a live server process. Startup interrupts
persisted queued resource visits, removes their waiter rows, and never replays an executor.
Durably cleanup-confirmed owners can be reclaimed through the token-checked delete helper.
Unconfirmed owners remain quarantined, including crashes before BeginStep, before session
registration, or after cleanup but before its confirmation is persisted. Session interruption
alone cannot prove resource-specific restoration. Age never authorizes reclamation.
Quarantined ownership/session evidence survives recovery and protects even terminal runs
from retention. Reconciliation requires inspected cleanup and restoration, not lock deletion.
There is intentionally no force-unlock CLI or automatic uncertain-owner reclamation.
## Android draft, provenance and activation
Read-only retrieval: `pipelines show Android --version 2 --json` returned pipeline 20,
v2, hash `8c93aeb9b76f1e6e7696415d6308d786ec08d805a1a346fbabe4d252d5d65032`.
`python3` recomputed SHA-256 of the returned source and asserted equality.
The exact source is retained as `internal/server/testdata/android-v2-published.md`;
the proposed source is `internal/server/testdata/android-resource-draft.md`.
The provenance test compares graph fields/outcomes unchanged, apart from the approved
three stage prompts and the UI review resource; every stage keeps its published v3 model: Terra for implementer/committer and Luna for reviewer/UI reviewer. No unrelated published definition was edited.
Artifact handoff recovery: the validated v3 draft was copied to this run's retained
artifact directory as `android-resource-draft.md`. The similarly named artifact from
run 355 remains the old v2 baseline and must not be published. The proposed source in
this workspace and this run's artifact have matching SHA-256 digests
`a10d8b9fa3dc6a1875b10469061ed25fb4a7a404d463d1a2d304a309d55aa5d9`.
Activation, requiring separate explicit operator authorization:
1. Review/land the source change and independently rerun the required checks.
2. Arrange a compatible deployment separately; nothing here deploys or restarts it.
3. Reconcile every existing phone user, pinned/live run and any quarantined owner.
4. Transfer the **workspace proposed draft**, provenance and this note into the retained
stage artifacts after resolving the artifact edit permission.
5. Validate that retained draft against the configured authoritative compatible endpoint;
only then explicitly publish it. Do not use a disposable endpoint as a project default.
6. Opt in other device users explicitly before relying on mutual exclusion system-wide.
Read-only published inventory (`pipelines show <name> --json`, v2 for each):
- UI sweep: `survey` and `ui_reviewer` request visual/screenshots work; phone work there
needs explicit resource opt-in and restoration/fixture policy.
- Generic: `reviewer` and `ui_reviewer` have visual/device verification wording; when
targeting a phone, separate code review from leased device review first.
- Plan and land: `implementer`, `reviewer`, `ui_reviewer` contain rendered-surface work;
they need an explicit technology-specific role split/lease before phone use.
- Probe: one harness smoke stage; no default phone declaration. Any device-specific
probe needs explicit opt-in, never an inferred resource or Android stage-name special case.
Server/TUI/other definitions were not changed. Inventory is not activation evidence.
## Verification evidence and remaining limits
Executed focused checks:
- Run 358 recovery revalidation: `go build ./... && go vet ./... && go test ./...`
passed; `go test -race ./internal/pipeline ./internal/server ./internal/stage -run
'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1` passed; and the
Android map invocation with `JAVA_HOME`/`ANDROID_HOME`, installed Gradle,
`--offline --no-daemon --rerun-tasks :app:assembleDebug :app:testDebugUnitTest`
executed all 41 tasks and reported `BUILD SUCCESSFUL`. These remain build/unit and
fake-executor scheduler evidence,
not physical-phone verification or screenshots.
- Recovery after integrating main `88595eb`: `go build ./...`, `go vet ./...`,
`go test ./internal/server -run Resource -count=1`,
`go test ./internal/pipeline ./internal/protocol ./internal/tui -run Resource -count=1`,
`go test ./internal/stage -count=1`,
`go test -race ./internal/pipeline ./internal/server ./internal/stage -run 'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1`, and
`go test ./...` all passed. `gofmt -l .` and `git diff --check` produced no output.
- Recovery Android build/unit check with the map's offline, no-daemon Gradle invocation:
`:app:assembleDebug :app:testDebugUnitTest` passed. This remains build/unit evidence,
not phone verification or screenshots.
- `go test ./internal/server ./internal/pipeline ./internal/protocol ./internal/tui -run Resource -count=1`:
passed. Includes fake-executor queue >30ms timeout, independent-resource progress,
FIFO/mismatch/duplicate/grant-cancel cycles, blocked cleanup, panic/quarantine,
persistence/BeginStep/query failure, no-acquire skip, restart/retention, wire defaults,
narrow/wide/plain TUI frames and real CLI/WebSocket scratch exercise.
- `go test ./internal/stage -count=1`: passed; the later full suite also included
`TestResourcePiReceiptRequiresProcessExit` (a test-owned sleep process, not a device).
- Android map invocation with `JAVA_HOME`/`ANDROID_HOME` and installed Gradle,
`--offline --no-daemon :app:assembleDebug :app:testDebugUnitTest`: BUILD SUCCESSFUL.
These are build/unit checks, not device verification or screenshots.
- After the Go tree settled: `go build ./...`, `go vet ./...`, `go test ./...`
all exited zero (including author-package tests).
- `go test -race ./internal/pipeline ./internal/server ./internal/stage -run 'Resource|Scheduler|Concurrent|Human|Decision|RetryWaits' -count=1`:
all three packages passed, including concurrent grant/cancel cycles.
- `gofmt -l .` and `git diff --check`: exited zero with no output.
The scratch CLI test uses the real CLI and WebSocket handler with temporary SQLite and
an explicitly **FAKE executor**. It validates the proposed Android draft on compatible
scratch code, observes B as queued with zero steps while A holds fake-phone, inspects
subscription metadata, releases A, runs `wait`, and observes B succeeded automatically.
It does not call a model or use a physical phone. Test log output is evidence, not a
live-deployment claim; no retained CLI transcript was written to the denied artifact path.
Live OpenCode/Pi shell/ADB termination and Android visual/device restoration are **not
tested** here. The checks prove scheduler behavior with controlled fakes, plus test-owned
process-exit checks, not device readiness on the current deployment.
Commit/merge/landing belong to the later committer stage; no commit or push here.
## Stale shared documentation for docs-fold (do not edit in this item)
- `docs/cli.md`: resource queue presentation, inspection metadata, unchanged wait/decide semantics.
- `internal/author/skill/references/format.md`: optional resources block and validation/key bounds.
- `internal/author/skill/references/control.md`: runner-owned declaration, timeout/accounting boundary,
release veto and automatic skip placement.
- `internal/author/skill/references/queue.md`: FIFO resource waits vs workspace exclusion and human waits.
- `internal/author/skill/references/operations.md`: authority, restart interruption/quarantine,
inspected recovery, retention and explicit compatible-deployment/publication prerequisites.
- `internal/author/skill/references/verification.md`: fake-vs-device evidence, fixtures and leased restoration.
- `internal/author/skill/references/orchestration.md`: no queue drivers/polling/extra agents; explicit
opt-in of every device user, no retroactive gating or distributed guarantees.
No packaged skill/reference, shared README/topic documentation, pipeline registry,
installation, deployment, provider setting or `internal/stage/model_failure.go` was edited.
+20
View File
@@ -0,0 +1,20 @@
# Scoped stage artifact access
## What this item changed
The server now derives one exact artifact capability for an OpenCode or Pi
model-stage visit: its own directory is writable when the edit tool is present,
and its own plus completed earlier visit directories in the same run are
readable when a file-reading tool is present. No artifact root, sibling, future
visit, other run, traversal path, or symlink escape is granted. The scope is
empty if the runner cannot validate its artifact root.
Pi retains its lease and explicit extension policy; unrestricted shell remains
outside this file-tool boundary and is not an OS sandbox.
## Shared documents this change makes stale
- `docs/stage-artifacts.md` and `internal/author/skill/references/operations.md`
should explain that model-stage artifact directories are usable by the
supported file tools only within the exact same-run visit scope, not as a
general external-directory permission.
+42
View File
@@ -0,0 +1,42 @@
# Stage-timeout diagnostics
## Source behavior
The V2 active-session wait now reports `STAGE DEADLINE` when the configured
attempt timer expires between successful polls of a running admitted session.
Its bounded diagnostic names the idle-wait phase, configured timeout, retained
session, and that the stage had not finished (active at the last poll). It does
not claim that a completed response was lost. Cancellation is no longer recorded
as a stage deadline. A bare deadline alone is not `HARNESS TRANSPORT`.
Actual request/HTTP/operation failures keep `HARNESS TRANSPORT`, their sanitized
operation/status evidence and underlying error identities, including when joined
with a deadline. A parent deadline no longer discards an accompanying transport
failure. Final-result validation, budget accounting, cleanup, interrupts,
side effects and replay policy are unchanged. No timeout was increased.
## Documentation to fold manually
`internal/author/skill/references/debugging.md` needs a separate stage-deadline
playbook alongside its HARNESS TRANSPORT section: inspect the retained session,
last activity, configured limits and unfinished final-result evidence before
deciding what to do. Do not blindly retry an admitted mutating stage; edits/tool
side effects remain. Explain the distinction between a configured idle-wait
overrun, cancellation, an actual failed request with a deadline, budget failure,
and final-result validation. Do not infer that all deadline errors are transport
failures or that a successful last tool means the stage returned a result.
## Evidence and deployment
2026-10-18: installed server build metadata (`go version -m
/Users/alan/.local/share/pipelines/bin/pipelines-server`) identifies revision
`0d1ddc73a444fa24110ba5223646856c8f053d29`. Comparing that revision with this
worktree's initial HEAD showed identical model-failure classification, model
attempt timing, V2 polling and agent wait/cleanup paths. The runner difference
was unrelated session-export HTML-only metadata.
Regression tests use isolated loopback servers and scratch SQLite databases,
with a controlled V2 CLI fixture for the admitted-active-session deadline and
real loopback HTTP/refused-connection checks for native transport evidence.
This is not a live-provider timeout test. No deployment or deployed-service
restart is part of this item; retained historical runs are not rewritten.