Retain Codex interaction callbacks across server restarts

This commit is contained in:
2026-11-05 14:32:59 +00:00
parent c7e57973a0
commit c2a8f5ff16
20 changed files with 1356 additions and 18 deletions
+35 -4
View File
@@ -36,10 +36,12 @@ chat_close; see `internal/server/chat_manager.go` and `chat_detach.go`):
`<cache>/pipelines/private/opencode-chat-service.json`). The next server
reuses it at the same address and password, and it is not stopped on
shutdown. A V1 serve child still ends with the server.
- **Codex**: threads and turns live in the App Server. Detaching closes the
control connection, and the next server reconciles through `thread/read`.
An approval or question the App Server sent to the old connection is bound to
it, so one still pending at shutdown cannot be answered after the restart.
- **Codex**: threads and turns live in the App Server. The installed server
connects through `pipelines-codex-host`, which retains the original App Server
connection and unanswered approvals/questions while Pipelines detaches. A new
server receives those same callbacks and reconciles ordinary turn state through
readback. Direct mode (hosting disabled or helper absent) still loses pending
connection-bound interactions on restart.
- **Claude Code**: its detachable hosts keep the CLI running (see the Claude
Code section).
@@ -72,6 +74,35 @@ not a fake empty success or silent auto-approval. App Server socket loss does
not provide durable callback recovery; a disconnected transport is reported as
an error rather than claiming old requests are still actionable.
### Pending interactions across restarts
The installed `pipelines-codex-host` helper owns one WebSocket connection to the
existing Codex App Server Unix socket. It never starts a second Codex daemon or
changes Codex settings. Helpers are scoped by database path and upstream socket,
with mode-0600 Unix sockets and singleton locks under the mode-0700 directory
`<user cache dir>/pipelines/codex-hosts/<database hash>/`.
On reconnect the helper returns its cached initialization handshake and replays
only unanswered server requests, preserving exact callback IDs and parameters.
Responses and `serverRequest/resolved` remove pending entries. Old/new client RPC
IDs are remapped so a delayed response for an old Pipelines process cannot answer
a new process's call. A new attach closes the obsolete downstream connection.
Ordinary notifications are not spooled; the existing thread readback repairs gaps.
State is bounded to 128 callbacks / 8 MiB and 256 in-flight client calls, with a
16 MiB frame limit. Exceeding pending bounds closes the bridge without granting,
declining or otherwise auto-answering an interaction. A detached bridge expires
after 20 minutes with no pending interactions, or after 24 hours even if pending.
An upstream App Server restart, a bridge crash, or expiry loses its connection-
bound callbacks: no disk JSON can recreate them on a new connection. The initial
deployment cannot recover requests already owned by the old direct connection.
`PIPELINES_CODEX_HOST` overrides the helper executable; `off` disables hosting.
The server otherwise finds it beside `pipelines-server`, then on PATH. Without
hosting, direct-mode fixtures and intentional direct connections retain their
previous behaviour and limitation. Installing a new helper binary does not kill
an existing helper or its pending requests.
## Claude Code
The adapter in `internal/claude` owns one Claude Code CLI process per active
+4 -3
View File
@@ -42,8 +42,9 @@ Linux needs `systemctl --user`. Other platforms are refused.
2. Builds the Flutter web app in Docker (pinned to `app/.flutter-version`).
Builds are cached in `~/.cache/pipelines/app-builds` by a digest of the app
sources, so an unchanged app is reused.
3. Builds `pipelines-server`, `pipelines` and `pipelines-claude-host` (keeps
Claude Code chats alive across server restarts) with `go build -trimpath`.
3. Builds `pipelines-server`, `pipelines`, `pipelines-claude-host` and
`pipelines-codex-host` (retain Claude processes and Codex interaction callbacks
across server restarts) with `go build -trimpath`.
4. Installs into the prefix: binaries in `bin/`, web app in `lib/app-web`, web
proxy in `lib/web`, `lib/manage.py`, and an empty `lib/app-updates` for APKs.
5. Runs `pipelines skill install` (the authoring skill for OpenCode and Claude
@@ -239,7 +240,7 @@ autostart, not boot-without-login.
| Path | Contents |
| --- | --- |
| `<prefix>/pipelines-install.json` | manifest (ports, paths, install args, environment) |
| `<prefix>/bin/` | `pipelines`, `pipelines-server`, `pipelines-claude-host`, `ntfy` |
| `<prefix>/bin/` | `pipelines`, `pipelines-server`, `pipelines-claude-host`, `pipelines-codex-host`, `ntfy` |
| `<prefix>/data/pipelines.db` | SQLite database (definitions, runs, chats, settings) |
| `<prefix>/data/artifacts/` | run artifacts and session exports |
| `<prefix>/data/ntfy/` | ntfy config, cache and accounts |
@@ -0,0 +1,49 @@
# Codex pending interactions across Pipelines restarts
The App Server's pending approval/question callback IDs belong to the connection
that received them. Its current official schema exposes no pending-request
listing/replay method. Saving JSON in the Pipelines database cannot make an old
callback answerable on a new upstream connection.
Added `pipelines-codex-host`: a detached, private local WebSocket bridge that owns
the existing App Server connection. It does not launch Codex. The installed
server scopes hosts by database and upstream socket, finds/reuses the helper,
and initializes only its downstream connection after a restart. The host caches
the one upstream initialization and replays exactly the still-unanswered server
requests. Responses or matching serverRequest/resolved notifications remove
pending entries. Client call IDs are translated to unique host IDs and routed
only to the originating client; replacing a client discards old routes. Client
and callback RPC IDs are capped at 256 bytes.
Unix sockets are mode 0600 under an owner-verified mode-0700 directory. Per-scope
file locks prevent simultaneous helpers. The existing daemon connection, not a
PID or a model statement, owns pending callbacks. Pending memory is capped at
128 requests / 8 MiB; calls at 256; incoming frames at 16 MiB. Overflow closes
rather than answering an interaction. With no Pipelines client, helpers expire
after 20 minutes with no pending callbacks, or after 24 hours even when pending.
Ordinary notifications are not spooled; current thread readback repairs gaps.
Native installation, tester packages and Docker include the helper; Go-bin
uninstallation verifies and removes its executable. Missing helper or explicit
PIPELINES_CODEX_HOST=off retains direct mode and logs its restart limitation.
Direct-mode fixtures still pass. Hosts run independently of a launching request's
context and are not killed by installing a replacement binary.
Verification:
- `go test -race ./internal/codexhost ./internal/codex ./internal/installation
./cmd/pipelines-server` passes. Real Unix-socket fixtures cover pending approval
and question responses across client replacement, duplicate response rejection,
resolved callbacks, matching thread ownership, old/new RPC-ID collision
isolation, detached arrival/replay, bounded pending state, singleton ownership,
private permissions, shutdown cleanup, and actual helper process spawning/reuse.
- The actual Codex adapter's Open -> Detach -> Open integration retains both
permission and question maps, answers their original IDs, initializes upstream
once, and does not replay resolved requests on a third client.
- Existing direct Codex tests pass; `go build ./...`, `go vet ./...`, and all 53
native-installer Python tests pass. Full Go integration is the manager's check.
No live service, conversation, daemon, credential or deployment was changed.
The initial deployment cannot rescue callbacks already owned by the old direct
connection; pending interactions must settle before that first replacement.
A helper crash, upstream App Server restart, or expiry breaks callback identity
and remains an explicit limitation. No disk reconstruction is claimed.