200 lines
11 KiB
Markdown
200 lines
11 KiB
Markdown
# Chat: Pipelines tools (show file / image / HTML, notify phone)
|
||
|
||
A chat's model can call four tools that act on Pipelines, not on the harness
|
||
or the workspace. The tools come from an MCP server that runs inside
|
||
`pipelines-server`. Each call adds an entry to the chat transcript, and the
|
||
entry is persisted.
|
||
|
||
| Tool | Arguments | Effect |
|
||
| --- | --- | --- |
|
||
| `show_file` | `path`, `title?` | Shows an existing file inside the operator's home. The path can be absolute, `~/…`, or relative to the chat's directory. Images render inline; other files render as a preview card. |
|
||
| `show_image` | `url_or_path`, `title?`, `alt?` | Shows a home image file (it must be `image/*`) or an `https://` URL. |
|
||
| `show_html` | `html`, `title?`, `height_hint?` | Renders model-authored HTML or SVG inline in a sandbox, with an **Open** button for full screen. |
|
||
| `notify` | `title`, `message`, `priority?` | Sends a phone notification through the existing ntfy "system" destination. Tapping it opens `pipelines://chat/<id>`. |
|
||
|
||
Each tool returns one short text line to the model. A refusal comes back as
|
||
`isError` together with the reason.
|
||
|
||
## Security model
|
||
|
||
- **Endpoint.** The endpoint is `POST /mcp/chat/<token>` on the server's own
|
||
listener. It speaks JSON-RPC 2.0 over MCP streamable HTTP: `initialize`,
|
||
`ping`, `tools/list`, `tools/call`, and notifications (answered `202`).
|
||
- `GET` and other methods get `405`.
|
||
- The protocol version is negotiated: 2025-11-25, 2025-06-18, 2025-03-26 or
|
||
2024-11-05.
|
||
- Batches are refused, and request bodies are capped at 4 MiB.
|
||
- Requests that carry an `Origin` header (from a browser) get `403`, as do
|
||
requests from non-loopback peers. Unknown tokens get `404`.
|
||
- **Not exposed through the web.** `web/serve.mjs` and the peer relay forward
|
||
only an allowlist (`/connect`, `/files`, `/artifacts/`, `/chat-uploads`,
|
||
`/updates/`). Both new tests (`web/test/mcpRefused.test.mjs` and the
|
||
peerRelay test) confirm `/mcp…` never reaches the server, including for a
|
||
signed-in browser and `%6dcp`-encoded paths.
|
||
- **Tokens.** Each token is 256 random bits, stored in `chat_tool_tokens` with
|
||
the scope it serves:
|
||
- `session:<id>`: one chat. Claude Code and Codex use this, and the token
|
||
stays stable across restarts so detached hosts keep working.
|
||
- `pending:`: a Codex thread before its id is known. Calls are refused until
|
||
the token is bound.
|
||
- `instance:opencode`: the shared OpenCode service. See "OpenCode attribution"
|
||
below.
|
||
- **Paths.** Paths use the same containment as `/files`:
|
||
`expandHomePath`, then `resolveInsideHome` (resolves every symlink), then
|
||
"regular file only", then the `specialFilesystem` check. The item records
|
||
the real path, and the client loads it from the chat's own machine's
|
||
`/files` (or `/peers/<id>/files` for a peer).
|
||
- **Image URLs.** Only `https` URLs without credentials, at most 2048 chars.
|
||
The client fetches them with a plain network image, without cookies.
|
||
- **HTML views.** These use the same inert view as artifact HTML previews:
|
||
- On web, an iframe with `sandbox="allow-popups allow-popups-to-escape-sandbox"`
|
||
and `srcdoc`. There are no scripts, no same-origin access, no forms and no
|
||
top navigation.
|
||
- Links are rewritten first: `#anchors` scroll inside the frame,
|
||
http(s)/mailto links open a new tab, and every other link is made inert.
|
||
- On Android/iOS, a WebView with JavaScript disabled that refuses navigation.
|
||
- Interactivity is therefore limited to CSS (`:hover`, `<details>`) and SVG.
|
||
The tool description tells the model this.
|
||
|
||
## Limits
|
||
|
||
- `show_html` documents are refused, not truncated, above **512 KiB**. The
|
||
model is told the size and the limit.
|
||
- At most **100 items are retained per chat**. HTML bodies are kept only for the
|
||
newest documents within **2 MiB per chat**; older documents keep their entry
|
||
but are marked `expired`, and the card says "This view is no longer retained".
|
||
- `notify` allows **5 per minute per chat** (a sliding window, held in memory).
|
||
Rate-limited calls record nothing.
|
||
- An undelivered notification is recorded with its reason, for example ntfy
|
||
not configured or switched off.
|
||
- `priority` is recorded, but ntfy always publishes at the default priority,
|
||
because `stage.PublishNtfy` has no priority parameter.
|
||
|
||
## Protocol and persistence
|
||
|
||
Items are stored in `chat_shown_items` (pipeline store). There are two ways
|
||
they reach clients:
|
||
|
||
- The `chat_subscribe` reply carries `chat_shown: {session_id, items}` with
|
||
every retained item, oldest first.
|
||
- A pushed `chat_shown` frame carries one new item.
|
||
|
||
The client interleaves items into the transcript by `created_at`. Items older
|
||
than the loaded history wait until that page loads. In a multi-machine chat,
|
||
the session id is rewritten by `ChatServerGateway`, and file URLs resolve
|
||
against the chat's own machine endpoint.
|
||
|
||
The generic tool rows for `mcp__pipelines__*` (Claude), `pipelines_*`
|
||
(OpenCode tool parts) and `pipelines/*` (Codex) are hidden; failed calls stay
|
||
visible. OpenCode V2 runs MCP tools from its code-mode `execute` script, and
|
||
that row stays visible because it can contain other code.
|
||
|
||
## Per-harness registration
|
||
|
||
`server.ConfigureChatTools(store, addr)` runs at startup, before any harness
|
||
starts, and installs `chattools.Registry`. The registry's base URL is the
|
||
loopback address of `-addr`; a wildcard address maps to 127.0.0.1, and a
|
||
non-loopback-only bind disables the tools with a log line.
|
||
|
||
- **Claude Code.** Started with `--mcp-config '{"mcpServers":{"pipelines":{"type":"http","url":…}}}'`
|
||
and `--allowedTools mcp__pipelines`. Both direct and hosted chats get the
|
||
session token.
|
||
- The config is added to the operator's own MCP servers; it does not replace
|
||
them.
|
||
- If a start fails with these flags, the chat logs the reason and retries
|
||
once without them (`session.toolsOff`).
|
||
- **Codex.** `thread/start` gets `config.mcp_servers.pipelines.url` with a
|
||
pending token, which is then bound to the returned thread id.
|
||
- A `thread/resume` (after "thread not found") registers the thread's
|
||
session token.
|
||
- If the App Server refuses the override, the thread starts or resumes
|
||
without it, and this is logged.
|
||
- Threads already loaded by the Codex desktop app before Pipelines touched
|
||
them have no tools.
|
||
- **OpenCode.** The spawned chat service gets
|
||
`mcp.pipelines = {type: "remote", url, enabled: true}` through
|
||
`OPENCODE_CONFIG_CONTENT`. This covers both `OpenChat` and the owned detached
|
||
`EnsureOwnedChatService`.
|
||
- The operator's chat overrides win.
|
||
- If the service does not start with the tools, it starts without them, and
|
||
this is logged.
|
||
- An operator-supplied endpoint (`-opencode-endpoint`) is not reconfigured,
|
||
so it has no tools.
|
||
- **Pi.** There is no Pi chat harness in this repository (chat backends are
|
||
opencode, claude and codex), so it is not supported.
|
||
|
||
**OpenCode attribution.** One OpenCode service serves every chat, and its MCP
|
||
requests carry no session (verified: OpenCode 2.0.32 sends only
|
||
`User-Agent: opencode/latest/2.0.32/cli` and `Mcp-Protocol-Version`).
|
||
|
||
A call on the instance token waits up to 3 s for exactly one chat whose
|
||
retained stream shows that call running. That means one of:
|
||
|
||
- a `pipelines_<tool>` part with identical arguments, or
|
||
- a running code-mode `execute` script containing `pipelines.<tool>(`. If
|
||
several such scripts are running, they are told apart by the argument
|
||
values.
|
||
|
||
When no chat matches, or several still do, the call is refused. It is never
|
||
guessed into a chat. Only chats Pipelines is observing can use the tools.
|
||
|
||
## Verification
|
||
|
||
- Go:
|
||
- `go test ./internal/server/ ./internal/chattools/ ./internal/pipeline/ ./internal/protocol/ ./internal/codex/ ./internal/codexhost/ ./internal/claudehost/`: all `ok` (server 72 s).
|
||
- `go test ./internal/claude/ ./internal/opencode/`: `ok` under `umask 022`.
|
||
Under this sandbox's `umask 077`, the existing tests
|
||
`TestServiceTokenIsPassedOnlyFromAPrivateFile` and
|
||
`TestStageDiscoveryRequiresPrivateRegistrationBeforeSessionWork` fail,
|
||
because both depend on 0644 file creation. They are unrelated to this
|
||
change.
|
||
- New Go tests:
|
||
- `chattools/mcp_test.go`: handshake, list, call, refusals, harness config.
|
||
- `server/chat_tools_test.go`: token isolation, pending binding, path
|
||
containment, image URLs, HTML limit and retention, notify rate limit,
|
||
instance and code-mode attribution.
|
||
- `server/chat_tools_http_test.go`: endpoint on the real listener,
|
||
`loopbackBase`.
|
||
- `claude|codex|opencode/pipelines_tools_test.go`: registration config and
|
||
the refuse-then-start-without fallback.
|
||
- `cd web && node --test test/`: `ℹ tests 18 … ℹ pass 18 ℹ fail 0`.
|
||
- Flutter:
|
||
- `tool/memwatch/full.sh test/features/chat`:
|
||
`memwatch: PASS: 1080 passed, 0 failed, 0 skipped; peak 2204 MB; 89s; exit 0 (test/features/chat)`.
|
||
- `tool/analyze.sh`: `0 errors, 8 warnings`. All 8 are the existing
|
||
`invalid_null_aware_operator` warnings in the unchanged
|
||
`integration_test/discovery_probe_test.dart`.
|
||
- Real end-to-end check, run on 2026-11-09:
|
||
- Setup: a scratch `pipelines-server` on `127.0.0.1:53834`, with
|
||
`HOME=/tmp/pl-e2e-tools/home`, a scratch database,
|
||
`PIPELINES_SHUTDOWN_SESSIONS=stop` and ntfy unset.
|
||
- OpenCode 2.0.32 (its default free model) was asked to `show_file
|
||
~/scratch/checker.png` and to `show_html` an SVG circle. Its `execute`
|
||
script returned "Shown in the operator's Pipelines chat: checker.png (217
|
||
bytes)" and "Shown … (137 bytes, sandboxed…)".
|
||
- `chat_subscribe` returned both items. The file item had
|
||
`path=/private/tmp/pl-e2e-tools/home/scratch/checker.png`,
|
||
`mime=image/png` and `size=217`. `/files` served that path with
|
||
`200 image/png 217`.
|
||
- `notify` was recorded as not delivered: "phone notifications (ntfy) are
|
||
not configured on this Pipelines server".
|
||
- A POST carrying an `Origin` header got `403`.
|
||
- The first OpenCode attempt was refused, and this is how code-mode
|
||
attribution was found to be necessary.
|
||
- Not run end to end:
|
||
- Claude Code under the isolated HOME answered "Not logged in", and
|
||
`~/.config/pipelines/claude-token` is empty.
|
||
- Codex has no app-server under the isolated HOME.
|
||
- Both registrations are covered only by unit tests against fake CLI
|
||
arguments and a fake App Server.
|
||
- Screenshots, real Chrome via `tool/capture` on the full-app fixture with
|
||
`?chats=1×tamps=1&shown=1&files=1`, all in
|
||
`notes/images/chat-pipelines-tools/`:
|
||
- `cards-light-880.png`: image, markdown card with first lines, HTML
|
||
diagram with Open, and notify row.
|
||
- `cards-dark-880.png`
|
||
- `cards-cupertino-560.png`
|
||
- `html-open-light-880.png`: full-screen HTML with zoom.
|
||
- `image-open-dark-880.png`: the existing viewer with download.
|
||
- Mobile (Android/iOS WebView, native thumbnails) is unverified: no device run.
|