Keep OpenCode/Codex chat and stage sessions running across server restarts
A stopping server no longer interrupts harness sessions. Runs are canceled with a shutdown cause, so stage clients leave admitted sessions running and owned; start-up recovery re-adopts live OpenCode V2 stage sessions (wait, read back the result, continue the graph), restarts unadmitted visits, continues runs between stages, and falls back to interrupted with a reason when the harness is gone. Chats with live work are recorded on shutdown and reattached on start; turns that ended during the downtime are folded in from history. The server-spawned V2 chat serve is now a detached owned service. PIPELINES_SHUTDOWN_SESSIONS=stop restores the old behaviour.
This commit is contained in:
@@ -6,6 +6,45 @@ when the `claude` CLI is installed. The selected backend is stored on the
|
||||
operator-session row and returned as `backend` on every `chat_sessions` row, so a
|
||||
restart does not have to guess from the session id.
|
||||
|
||||
## Restarts keep sessions running
|
||||
|
||||
A server restart or stop does not stop chat sessions. Only an operator action
|
||||
stops one: closing the chat, or Stop on its turn. On shutdown the server records
|
||||
each chat that still has work on its harness (a turn in flight, a pending
|
||||
question or approval, or running background tasks) in `chat_live_sessions`.
|
||||
It then detaches each backend instead of closing it, and never sends an
|
||||
interrupt.
|
||||
|
||||
On the next start each recorded chat is reattached. The server asks the harness
|
||||
whether the turn is still running, re-subscribes to its event stream with no
|
||||
client attached, re-reads pending approvals/questions and background shells,
|
||||
and holds queued prompts behind the turn. If a turn ended while the server was
|
||||
down, or its end event was lost with the old stream, the end is folded in from
|
||||
the harness's history. Clients get a `resync`, the reply notifies once (with
|
||||
the text the harness recorded), and queued prompts are delivered in order. The
|
||||
older start-up probe of recently used chats still runs for anything not
|
||||
recorded, for example after a crash.
|
||||
|
||||
Per backend (the contract is `chatSessionDetacher` in
|
||||
`internal/server/chat_detach.go`):
|
||||
|
||||
- **OpenCode**: sessions live in the OpenCode service. An attached endpoint
|
||||
(the operator's CLI-managed service) already outlives the server. Without
|
||||
one, a V2 server now uses a detached Pipelines-owned chat service
|
||||
(`opencode serve` in its own session, registered in
|
||||
`<cache>/pipelines/private/opencode-chat-service.json`). The next server
|
||||
reuses it at the same address and password, and it is not stopped on
|
||||
shutdown. A V1 serve child still ends with the server.
|
||||
- **Codex**: threads and turns live in the App Server. Detaching closes the
|
||||
control connection, and the next server reconciles through `thread/read`.
|
||||
An approval or question the App Server sent to the old connection is bound to
|
||||
it, so one still pending at shutdown cannot be answered after the restart.
|
||||
- **Claude Code**: closed as before until its host implements
|
||||
`DetachSessions`.
|
||||
|
||||
`PIPELINES_SHUTDOWN_SESSIONS=stop` restores the old shutdown, which closes the
|
||||
chat clients and records nothing. See [Installation](install.md#restarts-keep-sessions-running).
|
||||
|
||||
## Codex
|
||||
|
||||
The adapter in `internal/codex` connects to the Codex App Server's Unix
|
||||
|
||||
@@ -92,7 +92,10 @@ How the stage service behaves:
|
||||
session is active on it. A busy service keeps its old limits until it is idle.
|
||||
- **Lifetime.** It runs detached in its own process session, so a stopping
|
||||
Pipelines service does not take active stage work with it. On server shutdown
|
||||
it is stopped only when no session is active on it. Otherwise restart
|
||||
it is stopped only when no session is active on it. A busy service keeps
|
||||
running, and the next start re-adopts its sessions and continues their runs
|
||||
(see [Installation](install.md#restarts-keep-sessions-running)). With
|
||||
`PIPELINES_SHUTDOWN_SESSIONS=stop`, or when the service is gone, restart
|
||||
recovery interrupts the retained sessions through it, as before.
|
||||
- `pipelines debug` is unchanged: it opens a fresh interactive harness
|
||||
conversation, not the stage session.
|
||||
|
||||
+46
-2
@@ -173,8 +173,8 @@ another installation.
|
||||
| Command | Effect |
|
||||
| --- | --- |
|
||||
| `pipelines start` | start server, web proxy and ntfy; print the startup report |
|
||||
| `pipelines stop` | stop all three (active runs are interrupted; edits are not rolled back) |
|
||||
| `pipelines restart` | stop then start, keeping data |
|
||||
| `pipelines stop` | stop all three; harness sessions keep running and the next start reattaches them (see below) |
|
||||
| `pipelines restart` | stop then start, keeping data; chat turns and stage sessions continue |
|
||||
| `pipelines doctor` | startup report, health of server and ntfy, OpenCode API check (no inference) |
|
||||
| `pipelines status` | server, host and ntfy publishing health, from the running server |
|
||||
| `pipelines logs` | follow `logs/server.log` |
|
||||
@@ -184,6 +184,50 @@ another installation.
|
||||
| `pipelines service disable` | disable autostart and stop the supervisor |
|
||||
| `pipelines service status` | autostart and supervisor state |
|
||||
|
||||
### Restarts keep sessions running
|
||||
|
||||
A server restart, `pipelines stop` and `pipelines update` stop the Pipelines
|
||||
processes, not the work running on harnesses. On SIGTERM the server leaves
|
||||
every OpenCode and Codex session running, records what it needs to reattach,
|
||||
and exits. It does not interrupt or abort anything. On the next start:
|
||||
|
||||
- **Runs**: a stage whose OpenCode V2 session is still alive is re-adopted. The
|
||||
runner waits for that same session (no new prompt), or reads the result it
|
||||
reached while the server was down, and then continues the graph. A stage
|
||||
that had not yet handed work to a harness starts again. A run between stages
|
||||
continues with its next stage. The stage timeout restarts when the stage is
|
||||
reattached. Its file-change list is marked partial, because the snapshot
|
||||
taken before the restart is gone.
|
||||
- **Chats**: turns, pending questions/approvals, background tasks and queued
|
||||
prompts are picked up again, and a reply that arrived during the downtime
|
||||
still notifies. See [Chat backends](chat-backends.md#restarts-keep-sessions-running).
|
||||
|
||||
If the harness is gone, the old behaviour applies and the run is interrupted
|
||||
with the reason in its diagnostic, for example "Server restarted; the OpenCode
|
||||
service running stage … was gone". This happens when the OpenCode service
|
||||
process died, or the session belongs to a Pi or OpenCode V1 process that ended
|
||||
with the server.
|
||||
|
||||
Only explicit operator actions stop sessions:
|
||||
|
||||
- `pipelines cancel <run>` (or Cancel in the app) interrupts a run's stage.
|
||||
- Closing a chat, or Stop on its turn, interrupts that chat.
|
||||
|
||||
To stop everything: cancel the active runs and close or stop the busy chats,
|
||||
then run `pipelines stop`. The owned OpenCode stage service stops itself at
|
||||
shutdown when idle. The owned chat service, used only without an attached
|
||||
endpoint, keeps running and is reused. Stop it by killing the `pid` in
|
||||
`~/Library/Caches/pipelines/private/opencode-chat-service.json`
|
||||
(`~/.cache/...` on Linux). To make every shutdown interrupt runs and close chat
|
||||
clients as before, add `"PIPELINES_SHUTDOWN_SESSIONS": "stop"` to
|
||||
`opencode_env` in `<prefix>/pipelines-install.json` and restart.
|
||||
|
||||
Limit on Linux: systemd stops every process in the service's cgroup. The
|
||||
OpenCode services the server starts (stage service, owned chat service) end
|
||||
with `pipelines stop`/`restart`, and their runs fall back as described above.
|
||||
An attached OpenCode endpoint and the Codex App Server are separate and
|
||||
survive. macOS launchd does not kill them.
|
||||
|
||||
The login service is `~/Library/LaunchAgents/net.a-wan.pipelines.plist` on
|
||||
macOS and `~/.config/systemd/user/net.a-wan.pipelines.service` on Linux. It
|
||||
runs `manage.py supervise`, which owns all three processes. This is login
|
||||
|
||||
Reference in New Issue
Block a user