2307d45417
Ensure /new and chat close stop native foreground and background work before changing registry state. Preserve transcripts while surfacing release and recovery cleanup failures instead of reporting false success, with focused adapter and server regressions.
307 lines
18 KiB
Markdown
307 lines
18 KiB
Markdown
# Chat backends
|
|
|
|
Chats can be created on more than one harness. OpenCode remains the default;
|
|
Codex is available when its App Server control socket exists, and Claude Code
|
|
when the `claude` CLI is installed. The selected backend is stored on the
|
|
operator-session row and returned as `backend` on every `chat_sessions` row, so a
|
|
restart does not have to guess from the session id.
|
|
|
|
When a `chat_prompt` is accepted during a running turn, the reply sets
|
|
`chat_steered: true` when the backend hands it to that turn; otherwise it sets
|
|
`chat_queued: true` while the prompt waits for the turn to finish.
|
|
|
|
## Restarts keep sessions running
|
|
|
|
A server restart or stop does not stop chat sessions. Only an operator action
|
|
stops one: closing the chat, or Stop on its turn. The server durably journals
|
|
chat work as it changes (a turn in flight, a pending question or approval, or
|
|
running background tasks), and refreshes `chat_live_sessions` on shutdown.
|
|
It then detaches each backend instead of closing it, and never sends an
|
|
interrupt.
|
|
|
|
On the next start each recorded chat is reattached. The server asks the harness
|
|
whether the turn is still running, re-subscribes to its event stream with no
|
|
client attached, re-reads pending approvals/questions and background shells,
|
|
and holds queued prompts behind the turn. If a turn ended while the server was
|
|
down, or its end event was lost with the old stream, the end is folded in from
|
|
the harness's history. Clients get a `resync`, the reply notifies once (with
|
|
the text the harness recorded), and queued prompts are delivered in order. The
|
|
startup discovery checks all open registered chats for work not yet recorded,
|
|
including after a crash. Unavailable sessions remain journaled and retry;
|
|
browser requests wait until journaled turns and queued prompts are restored.
|
|
|
|
Per backend (the contract is `chatShutdownDetacher.Detach`, which
|
|
`chatManager.Close` calls instead of `Close`, plus `chatSessionReleaser` for
|
|
chat_close; see `internal/server/chat_manager.go` and `chat_detach.go`):
|
|
|
|
- **OpenCode**: sessions live in the OpenCode service. An attached endpoint
|
|
(the operator's CLI-managed service) already outlives the server. Without
|
|
one, a V2 server now uses a detached Pipelines-owned chat service
|
|
(`opencode serve` in its own session, registered in
|
|
`<cache>/pipelines/private/opencode-chat-service.json`). The next server
|
|
reuses it at the same address and password, and it is not stopped on
|
|
shutdown. A V1 serve child still ends with the server.
|
|
- **Codex**: threads and turns live in the App Server. The installed server
|
|
connects through `pipelines-codex-host`, which retains the original App Server
|
|
connection and unanswered approvals/questions while Pipelines detaches. A new
|
|
server receives those same callbacks and reconciles ordinary turn state through
|
|
readback. Direct mode (hosting disabled or helper absent) still loses pending
|
|
connection-bound interactions on restart.
|
|
- **Claude Code**: its detachable hosts keep the CLI running (see the Claude
|
|
Code section).
|
|
|
|
`PIPELINES_SHUTDOWN_SESSIONS=stop` restores the old shutdown, which closes the
|
|
chat clients and records nothing. See [Installation](install.md#restarts-keep-sessions-running).
|
|
|
|
## Codex
|
|
|
|
The adapter in `internal/codex` connects to the Codex App Server's Unix
|
|
WebSocket control socket (`$CODEX_HOME/app-server-control/app-server-control.sock`,
|
|
falling back to `~/.codex`). It uses the operator's existing Codex login. A new
|
|
chat issues `thread/start`; a prompt issues `turn/start`; history comes from
|
|
`thread/read`; questions map from `item/tool/requestUserInput` to the normal
|
|
question card and `chat_answer` returns the JSON-RPC response. Threads created
|
|
this way are ordinary Codex threads and remain visible to the Codex app.
|
|
|
|
New threads retain the `workspace-write` sandbox and inherit Codex's configured
|
|
approval policy (Pipelines no longer forces `never`). Command, file-change,
|
|
network and explicit permission requests appear in the web approval cards. Each
|
|
JSON-RPC callback has its own identity, including Code Mode subcommands sharing
|
|
one item. Allow once maps to `accept`, session grants to `acceptForSession`, and
|
|
Reject to `decline`; explicit permission profiles grant only requested fields.
|
|
The web labels session scope explicitly, never as a permanent policy change.
|
|
|
|
Pending questions and approvals survive browser reconnects while the App Server
|
|
connection lives. Standard MCP form/URL elicitation supports typed answers,
|
|
Accept/Decline/Cancel and safe external links. Extended OpenAI elicitation is not
|
|
advertised; unknown request formats return an explicit unsupported RPC error,
|
|
not a fake empty success or silent auto-approval. App Server socket loss does
|
|
not provide durable callback recovery; a disconnected transport is reported as
|
|
an error rather than claiming old requests are still actionable.
|
|
|
|
### Pending interactions across restarts
|
|
|
|
The installed `pipelines-codex-host` helper owns one WebSocket connection to the
|
|
existing Codex App Server Unix socket. It never starts a second Codex daemon or
|
|
changes Codex settings. Helpers are scoped by database path and upstream socket,
|
|
with mode-0600 Unix sockets and singleton locks under the mode-0700 directory
|
|
`<user cache dir>/pipelines/codex-hosts/<database hash>/`.
|
|
|
|
The helper completes upstream initialize -> initialized exactly once before
|
|
accepting clients. On reconnect it returns the cached initialization result and replays
|
|
only unanswered server requests, preserving exact callback IDs and parameters.
|
|
Responses and `serverRequest/resolved` remove pending entries. Old/new client RPC
|
|
IDs are remapped so a delayed response for an old Pipelines process cannot answer
|
|
a new process's call. A new attach closes the obsolete downstream connection.
|
|
Ordinary notifications are not spooled; the existing thread readback repairs gaps.
|
|
|
|
State is bounded to 128 callbacks / 8 MiB and 256 in-flight client calls, with a
|
|
16 MiB frame limit. Exceeding pending bounds closes the bridge without granting,
|
|
declining or otherwise auto-answering an interaction. A detached bridge keeps its original upstream connection indefinitely by default.
|
|
Only explicitly supplied positive `-idle`/`-max-detached` limits enable expiry.
|
|
An upstream App Server restart, a bridge crash, or expiry loses its connection-
|
|
bound callbacks: no disk JSON can recreate them on a new connection. The initial
|
|
deployment cannot recover requests already owned by the old direct connection.
|
|
|
|
`PIPELINES_CODEX_HOST` overrides the helper executable; `off` disables hosting.
|
|
The server otherwise finds it beside `pipelines-server`, then on PATH. Without
|
|
hosting, direct-mode fixtures and intentional direct connections retain their
|
|
previous behaviour and limitation. Installing a new helper binary does not kill
|
|
an existing helper or its pending requests.
|
|
|
|
## Claude Code
|
|
|
|
The adapter in `internal/claude` owns one Claude Code CLI process per active
|
|
session. It starts `claude -p` with stream-json input and output, partial
|
|
messages, and `--permission-prompt-tool=stdio`, then performs the SDK-style
|
|
`initialize` handshake before any turn. The operator's existing Claude Code login
|
|
and configuration are used unchanged; Pipelines does not store a second
|
|
credential or alter global settings. `PIPELINES_CLAUDE` may name the executable;
|
|
otherwise `claude` is resolved from `PATH` and common per-user locations.
|
|
|
|
A new session pins a generated UUID with `--session-id`. When the process is
|
|
released (idle, an explicit close, or a host that crashed), the next prompt
|
|
resumes the same conversation with `--resume <id>`, so context and the selected
|
|
model survive. Chat creation is local: the CLI starts on the first prompt or
|
|
model-catalog read.
|
|
|
|
### Surviving a server restart
|
|
|
|
A server restart does not stop a Claude Code chat. The installed server runs each
|
|
CLI inside `pipelines-claude-host` (`cmd/pipelines-claude-host`,
|
|
`internal/claudehost`), a small helper installed next to `pipelines-server`. The
|
|
host starts in its own session and process group, so neither launchd nor the
|
|
supervisor stops it with the server (the systemd unit uses `KillMode=process`
|
|
for the same reason). The host owns the CLI's stdin/stdout. It appends every
|
|
output line to a per-session spool file with byte offsets and serves a Unix
|
|
socket. The socket is mode 0600 in a 0700 directory:
|
|
`<user cache dir>/pipelines/claude-hosts/<hash of the database path>/`. Two
|
|
servers of one user never adopt each other's chats. Each host also writes a
|
|
registry file (`<key>.json`) there.
|
|
|
|
The socket speaks one JSON object per line: `attach` (deliver output from an
|
|
offset; a newer attach replaces the older connection), `in` (one stdin frame),
|
|
`ack` (output processed up to an offset), `status` and `close`. Interrupts,
|
|
model and mode switches travel as the CLI's own control requests inside `in`. The
|
|
host snoops these frames to keep the CLI's initialize response (models, account),
|
|
the applied model and permission mode, a turn's running state, and the control
|
|
requests nobody has answered yet (`can_use_tool`, `AskUserQuestion`).
|
|
|
|
On shutdown the server only disconnects (`Detach`). On start it reads the
|
|
registry and reattaches to every live host before serving. It replays the spool
|
|
from its start without broadcasting, which rebuilds the session's stream state,
|
|
background commands and agents, and pending approvals and questions. Requests
|
|
already answered are skipped. Output after the last acknowledged offset, written
|
|
while no server was attached, is then processed live. The turn is shown running
|
|
if it was running at that offset, watchers get a resync, and the start-up turn
|
|
probe keeps observing it for notifications. An approval or question asked before
|
|
the restart can be answered after it. Stop works as usual, and the CLI is never
|
|
restarted or initialized twice.
|
|
|
|
The host stops the CLI (EOF, then SIGINT, then SIGKILL to its process group) and
|
|
removes its files only when one of these happens:
|
|
|
|
- The operator closes the chat (`chat_close`).
|
|
|
|
An explicit close releases the backend runtime before recording `closed_at`; a
|
|
release failure is returned to the client and does not report a successful
|
|
close. The registry row and native transcript are retained in either case.
|
|
- A genuinely idle unhosted CLI is released. For hosted sessions, server cache
|
|
eviction only detaches; the host owns authoritative background-task liveness.
|
|
- The CLI exits; the server then gets its stderr.
|
|
- With no server attached, the CLI has been genuinely idle for 20 minutes,
|
|
with no foreground turn, pending interaction, or native background task.
|
|
There is no default hard detached lifetime limit.
|
|
|
|
If the control connection is lost, the server first tries to reconnect. A verified
|
|
live host is preserved when its socket is temporarily unavailable, and another
|
|
CLI is not started alongside it. Dead host recovery verifies process identity
|
|
before stopping an orphan. On attachment, the host's authoritative task snapshot
|
|
restores running task identities even after older spool records were trimmed.
|
|
Legacy hosts without that snapshot remain attached during cache eviction, so
|
|
their older detached idle timers cannot mistake background work for an idle CLI.
|
|
|
|
Limits:
|
|
|
|
- A container restart (Docker) still ends every host. So does a reboot or
|
|
logout. The transcript survives and `--resume` applies.
|
|
- The spool keeps at most 64 MiB of acknowledged output. Older output is
|
|
dropped. Current hosts retain running task identities separately from output.
|
|
Legacy hosts with an incomplete replay report task visibility as unknown,
|
|
rather than claiming there are no tasks. Unacknowledged output is dropped only
|
|
beyond 256 MiB; the server
|
|
then logs the gap and clients reload the transcript.
|
|
- Turn bookkeeping that lived only in server memory restarts fresh: the turn id,
|
|
and which mid-turn messages are still outstanding. Output that begins a
|
|
follow-up turn after a reconnect is adopted as a running turn.
|
|
- Without the host binary (`PIPELINES_CLAUDE_HOST=off`, or not found next to the
|
|
server or on `PATH`), the CLI is the server's child as before and a restart
|
|
resumes it.
|
|
|
|
Text and reasoning stream from the CLI's partial-message events; completed
|
|
messages are de-duplicated against the streamed text. Tool calls appear as normal
|
|
tool cards, Bash commands as the existing background-command rows, and results
|
|
attach to the call. `can_use_tool` becomes the normal approval card: Allow once
|
|
returns the original input, Always allow forwards the CLI's own permission
|
|
suggestions, and Reject returns a deny. `AskUserQuestion` becomes the normal
|
|
question card and `chat_answer` returns the chosen answers. Interrupt sends the
|
|
CLI's own `interrupt` control request.
|
|
|
|
The model catalog comes from the CLI's initialize response and is switched live
|
|
with `set_model`; Pipelines persists the choice for the next turn so a resumed
|
|
process starts with it. Usage and context totals come from the CLI's `get_usage`
|
|
and `get_context_usage` queries: session cost, processed tokens, context tokens
|
|
and the model window are reported values, not estimates. Compaction boundaries
|
|
surface as native compaction activity.
|
|
|
|
History is read from Claude Code's own session transcript
|
|
(`$CLAUDE_CONFIG_DIR/projects/*/<id>.jsonl` or `~/.claude`). Entries are merged by
|
|
message id, tool results attach to their calls, subagent sidechains and local
|
|
command noise are excluded, and paging is bounded. A session file larger than
|
|
64 MiB is reported as exceeding the history bound rather than silently truncated.
|
|
The live user turn keeps the client's optimistic id when it can be matched to the
|
|
recorded prompt.
|
|
|
|
## Fresh context (`/new`)
|
|
|
|
The composer's `/new` sends `chat_new_session`. It keeps the chat and clears the
|
|
agent's context, like `/new` or `/clear` in the harness's own TUI. The server
|
|
creates a fresh native session on the chat's own harness, in the same directory
|
|
and with the same title. OpenCode makes a new `ses_` session, Codex a new thread
|
|
and Claude Code a new session UUID. The server then re-keys the
|
|
`operator_sessions` row in one transaction. The new row keeps the title,
|
|
directory, tag, backend, colour, star, creation time, model override and mode.
|
|
The old row is not deleted: it records `replaced_by`, is no longer listed, and
|
|
its native session can still be resumed in a terminal. Before the row is moved,
|
|
the server releases the old native runtime; this stops active/background work
|
|
without deleting the native transcript. If release fails, `/new` reports an
|
|
error and leaves the registry row on the old session.
|
|
|
|
The row is re-keyed rather than mapped from a stable id because every other
|
|
layer is keyed by the native id: rooms, queues, transcripts, shells, exports and
|
|
resume commands. The reply, and the `chat_session_changed` push to other
|
|
connections, carry the new row with `previous_id`. Each client swaps that row in
|
|
place, and an open conversation follows it to the fresh, empty transcript. A
|
|
running turn is refused, so stop it first. A second `/new` from an id that has
|
|
already been replaced is refused too.
|
|
|
|
Claude Code also accepts `/clear` in `-p --input-format stream-json` mode, but
|
|
the CLI then picks the new session id itself. Re-keying keeps one path for all
|
|
three harnesses and lets the server choose the id.
|
|
|
|
## Chat list activity
|
|
|
|
Every `chat_sessions` row carries `activity` (turn running, running background
|
|
tasks), and `chat_activity` pushes each change to every connection (see
|
|
`docs/cli.md`). Background tasks are the same shells all three harnesses already
|
|
report: OpenCode shells, Codex `commandExecution` items, and Claude Code
|
|
background Bash and Agent/Task subagents. A chat whose last viewer left stays
|
|
observed while its tasks run, so their exits are seen. That is bounded by the
|
|
unattended limits (16 chats, 6 hours) and by the harness: when a Claude Code
|
|
process exits, its background tasks are reported exited because they ran inside
|
|
it. After an OpenCode or Codex stream reconnects, the shell set is re-read from
|
|
the harness.
|
|
|
|
## Web interaction contract
|
|
|
|
All advertised chat adapters (OpenCode V2, Codex and Claude Code) implement the
|
|
same server contract for uploads, models, pending approvals, questions, form
|
|
replies, interrupts, command readback and bounded output. Compile-time assertions
|
|
and browser regressions protect this contract; support is not satisfied by text
|
|
streaming alone. Pi's pipeline executor is not advertised as a Pi web-chat
|
|
adapter, and a harness whose CLI is absent is not presented as a working choice.
|
|
|
|
Codex uploads use native `localImage` for verified supported images. Other files
|
|
are retained intact with an explicit server-local reference; the model's own
|
|
tools inspect them rather than pretending their contents were already loaded.
|
|
The saved next-turn model is also respected with uploads. Model sandbox and
|
|
approval policy remain in force. Claude Code receives images as native image
|
|
blocks, small text files inline, and larger files by absolute server path; the
|
|
saved next-turn model is applied before the turn.
|
|
|
|
Codex's paged history retains tool and reasoning items, including structured MCP
|
|
errors and Code Mode outputs, instead of discarding everything except text.
|
|
Unknown item kinds remain inspectable. Tool detail is explicitly truncated at
|
|
64 KiB, not silently represented as complete. Running commands and their output
|
|
use real command items plus read-only thread history; no unsupported terminal
|
|
endpoint, process control, resumed turn or experimental flag is invented.
|
|
Output reads are bounded and verify thread ownership. The installed Codex
|
|
version exposes no API for controlling an agent-owned terminal through the
|
|
separate, client-owned `command/exec` process IDs; those IDs are never conflated.
|
|
|
|
A fresh Codex connection can accept a turn without subscribing to its native
|
|
thread notifications. The adapter therefore reconciles the latest turn through
|
|
the read-only paged API and emits an authoritative-history resync when it changes.
|
|
It does not call `thread/resume` on open/model selection, so merely watching a
|
|
chat cannot awaken a queued goal. Native turn IDs protect the queue/UI from stale
|
|
completion events. A real, isolated tool-free turn passed after replacing the
|
|
owning connection; no command, permission grant or global setting was involved.
|
|
|
|
An explicit Send can load a persisted thread if `turn/start` is refused with
|
|
the exact application error `thread not found: <owned-thread-id>`. Only then
|
|
does the adapter call `thread/resume` (metadata only, no policy/configuration
|
|
override) and submit the original message/model/files. Transport failures and
|
|
unrelated errors are never retried. This loading is not performed by browsing,
|
|
history reads or model selection. As in Codex's native client, explicitly
|
|
continuing a thread also resumes that thread's configured execution context.
|