Keep OpenCode/Codex chat and stage sessions running across server restarts

A stopping server no longer interrupts harness sessions. Runs are canceled
with a shutdown cause, so stage clients leave admitted sessions running and
owned; start-up recovery re-adopts live OpenCode V2 stage sessions (wait,
read back the result, continue the graph), restarts unadmitted visits,
continues runs between stages, and falls back to interrupted with a reason
when the harness is gone. Chats with live work are recorded on shutdown and
reattached on start; turns that ended during the downtime are folded in
from history. The server-spawned V2 chat serve is now a detached owned
service. PIPELINES_SHUTDOWN_SESSIONS=stop restores the old behaviour.
This commit is contained in:
2026-11-04 16:17:08 +00:00
parent 9351cd7d74
commit aa8eddd686
26 changed files with 2318 additions and 99 deletions
+39
View File
@@ -6,6 +6,45 @@ when the `claude` CLI is installed. The selected backend is stored on the
operator-session row and returned as `backend` on every `chat_sessions` row, so a
restart does not have to guess from the session id.
## Restarts keep sessions running
A server restart or stop does not stop chat sessions. Only an operator action
stops one: closing the chat, or Stop on its turn. On shutdown the server records
each chat that still has work on its harness (a turn in flight, a pending
question or approval, or running background tasks) in `chat_live_sessions`.
It then detaches each backend instead of closing it, and never sends an
interrupt.
On the next start each recorded chat is reattached. The server asks the harness
whether the turn is still running, re-subscribes to its event stream with no
client attached, re-reads pending approvals/questions and background shells,
and holds queued prompts behind the turn. If a turn ended while the server was
down, or its end event was lost with the old stream, the end is folded in from
the harness's history. Clients get a `resync`, the reply notifies once (with
the text the harness recorded), and queued prompts are delivered in order. The
older start-up probe of recently used chats still runs for anything not
recorded, for example after a crash.
Per backend (the contract is `chatSessionDetacher` in
`internal/server/chat_detach.go`):
- **OpenCode**: sessions live in the OpenCode service. An attached endpoint
(the operator's CLI-managed service) already outlives the server. Without
one, a V2 server now uses a detached Pipelines-owned chat service
(`opencode serve` in its own session, registered in
`<cache>/pipelines/private/opencode-chat-service.json`). The next server
reuses it at the same address and password, and it is not stopped on
shutdown. A V1 serve child still ends with the server.
- **Codex**: threads and turns live in the App Server. Detaching closes the
control connection, and the next server reconciles through `thread/read`.
An approval or question the App Server sent to the old connection is bound to
it, so one still pending at shutdown cannot be answered after the restart.
- **Claude Code**: closed as before until its host implements
`DetachSessions`.
`PIPELINES_SHUTDOWN_SESSIONS=stop` restores the old shutdown, which closes the
chat clients and records nothing. See [Installation](install.md#restarts-keep-sessions-running).
## Codex
The adapter in `internal/codex` connects to the Codex App Server's Unix
+4 -1
View File
@@ -92,7 +92,10 @@ How the stage service behaves:
session is active on it. A busy service keeps its old limits until it is idle.
- **Lifetime.** It runs detached in its own process session, so a stopping
Pipelines service does not take active stage work with it. On server shutdown
it is stopped only when no session is active on it. Otherwise restart
it is stopped only when no session is active on it. A busy service keeps
running, and the next start re-adopts its sessions and continues their runs
(see [Installation](install.md#restarts-keep-sessions-running)). With
`PIPELINES_SHUTDOWN_SESSIONS=stop`, or when the service is gone, restart
recovery interrupts the retained sessions through it, as before.
- `pipelines debug` is unchanged: it opens a fresh interactive harness
conversation, not the stage session.
+46 -2
View File
@@ -173,8 +173,8 @@ another installation.
| Command | Effect |
| --- | --- |
| `pipelines start` | start server, web proxy and ntfy; print the startup report |
| `pipelines stop` | stop all three (active runs are interrupted; edits are not rolled back) |
| `pipelines restart` | stop then start, keeping data |
| `pipelines stop` | stop all three; harness sessions keep running and the next start reattaches them (see below) |
| `pipelines restart` | stop then start, keeping data; chat turns and stage sessions continue |
| `pipelines doctor` | startup report, health of server and ntfy, OpenCode API check (no inference) |
| `pipelines status` | server, host and ntfy publishing health, from the running server |
| `pipelines logs` | follow `logs/server.log` |
@@ -184,6 +184,50 @@ another installation.
| `pipelines service disable` | disable autostart and stop the supervisor |
| `pipelines service status` | autostart and supervisor state |
### Restarts keep sessions running
A server restart, `pipelines stop` and `pipelines update` stop the Pipelines
processes, not the work running on harnesses. On SIGTERM the server leaves
every OpenCode and Codex session running, records what it needs to reattach,
and exits. It does not interrupt or abort anything. On the next start:
- **Runs**: a stage whose OpenCode V2 session is still alive is re-adopted. The
runner waits for that same session (no new prompt), or reads the result it
reached while the server was down, and then continues the graph. A stage
that had not yet handed work to a harness starts again. A run between stages
continues with its next stage. The stage timeout restarts when the stage is
reattached. Its file-change list is marked partial, because the snapshot
taken before the restart is gone.
- **Chats**: turns, pending questions/approvals, background tasks and queued
prompts are picked up again, and a reply that arrived during the downtime
still notifies. See [Chat backends](chat-backends.md#restarts-keep-sessions-running).
If the harness is gone, the old behaviour applies and the run is interrupted
with the reason in its diagnostic, for example "Server restarted; the OpenCode
service running stage … was gone". This happens when the OpenCode service
process died, or the session belongs to a Pi or OpenCode V1 process that ended
with the server.
Only explicit operator actions stop sessions:
- `pipelines cancel <run>` (or Cancel in the app) interrupts a run's stage.
- Closing a chat, or Stop on its turn, interrupts that chat.
To stop everything: cancel the active runs and close or stop the busy chats,
then run `pipelines stop`. The owned OpenCode stage service stops itself at
shutdown when idle. The owned chat service, used only without an attached
endpoint, keeps running and is reused. Stop it by killing the `pid` in
`~/Library/Caches/pipelines/private/opencode-chat-service.json`
(`~/.cache/...` on Linux). To make every shutdown interrupt runs and close chat
clients as before, add `"PIPELINES_SHUTDOWN_SESSIONS": "stop"` to
`opencode_env` in `<prefix>/pipelines-install.json` and restart.
Limit on Linux: systemd stops every process in the service's cgroup. The
OpenCode services the server starts (stage service, owned chat service) end
with `pipelines stop`/`restart`, and their runs fall back as described above.
An attached OpenCode endpoint and the Codex App Server are separate and
survive. macOS launchd does not kill them.
The login service is `~/Library/LaunchAgents/net.a-wan.pipelines.plist` on
macOS and `~/.config/systemd/user/net.a-wan.pipelines.service` on Linux. It
runs `manage.py supervise`, which owns all three processes. This is login