Files
pipelines/app/notes/chat-pipelines-tools.md
T

200 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Chat: Pipelines tools (show file / image / HTML, notify phone)
A chat's model can call four tools that act on Pipelines, not on the harness
or the workspace. The tools come from an MCP server that runs inside
`pipelines-server`. Each call adds an entry to the chat transcript, and the
entry is persisted.
| Tool | Arguments | Effect |
| --- | --- | --- |
| `show_file` | `path`, `title?` | Shows an existing file inside the operator's home. The path can be absolute, `~/…`, or relative to the chat's directory. Images render inline; other files render as a preview card. |
| `show_image` | `url_or_path`, `title?`, `alt?` | Shows a home image file (it must be `image/*`) or an `https://` URL. |
| `show_html` | `html`, `title?`, `height_hint?` | Renders model-authored HTML or SVG inline in a sandbox, with an **Open** button for full screen. |
| `notify` | `title`, `message`, `priority?` | Sends a phone notification through the existing ntfy "system" destination. Tapping it opens `pipelines://chat/<id>`. |
Each tool returns one short text line to the model. A refusal comes back as
`isError` together with the reason.
## Security model
- **Endpoint.** The endpoint is `POST /mcp/chat/<token>` on the server's own
listener. It speaks JSON-RPC 2.0 over MCP streamable HTTP: `initialize`,
`ping`, `tools/list`, `tools/call`, and notifications (answered `202`).
- `GET` and other methods get `405`.
- The protocol version is negotiated: 2025-11-25, 2025-06-18, 2025-03-26 or
2024-11-05.
- Batches are refused, and request bodies are capped at 4 MiB.
- Requests that carry an `Origin` header (from a browser) get `403`, as do
requests from non-loopback peers. Unknown tokens get `404`.
- **Not exposed through the web.** `web/serve.mjs` and the peer relay forward
only an allowlist (`/connect`, `/files`, `/artifacts/`, `/chat-uploads`,
`/updates/`). Both new tests (`web/test/mcpRefused.test.mjs` and the
peerRelay test) confirm `/mcp…` never reaches the server, including for a
signed-in browser and `%6dcp`-encoded paths.
- **Tokens.** Each token is 256 random bits, stored in `chat_tool_tokens` with
the scope it serves:
- `session:<id>`: one chat. Claude Code and Codex use this, and the token
stays stable across restarts so detached hosts keep working.
- `pending:`: a Codex thread before its id is known. Calls are refused until
the token is bound.
- `instance:opencode`: the shared OpenCode service. See "OpenCode attribution"
below.
- **Paths.** Paths use the same containment as `/files`:
`expandHomePath`, then `resolveInsideHome` (resolves every symlink), then
"regular file only", then the `specialFilesystem` check. The item records
the real path, and the client loads it from the chat's own machine's
`/files` (or `/peers/<id>/files` for a peer).
- **Image URLs.** Only `https` URLs without credentials, at most 2048 chars.
The client fetches them with a plain network image, without cookies.
- **HTML views.** These use the same inert view as artifact HTML previews:
- On web, an iframe with `sandbox="allow-popups allow-popups-to-escape-sandbox"`
and `srcdoc`. There are no scripts, no same-origin access, no forms and no
top navigation.
- Links are rewritten first: `#anchors` scroll inside the frame,
http(s)/mailto links open a new tab, and every other link is made inert.
- On Android/iOS, a WebView with JavaScript disabled that refuses navigation.
- Interactivity is therefore limited to CSS (`:hover`, `<details>`) and SVG.
The tool description tells the model this.
## Limits
- `show_html` documents are refused, not truncated, above **512 KiB**. The
model is told the size and the limit.
- At most **100 items are retained per chat**. HTML bodies are kept only for the
newest documents within **2 MiB per chat**; older documents keep their entry
but are marked `expired`, and the card says "This view is no longer retained".
- `notify` allows **5 per minute per chat** (a sliding window, held in memory).
Rate-limited calls record nothing.
- An undelivered notification is recorded with its reason, for example ntfy
not configured or switched off.
- `priority` is recorded, but ntfy always publishes at the default priority,
because `stage.PublishNtfy` has no priority parameter.
## Protocol and persistence
Items are stored in `chat_shown_items` (pipeline store). There are two ways
they reach clients:
- The `chat_subscribe` reply carries `chat_shown: {session_id, items}` with
every retained item, oldest first.
- A pushed `chat_shown` frame carries one new item.
The client interleaves items into the transcript by `created_at`. Items older
than the loaded history wait until that page loads. In a multi-machine chat,
the session id is rewritten by `ChatServerGateway`, and file URLs resolve
against the chat's own machine endpoint.
The generic tool rows for `mcp__pipelines__*` (Claude), `pipelines_*`
(OpenCode tool parts) and `pipelines/*` (Codex) are hidden; failed calls stay
visible. OpenCode V2 runs MCP tools from its code-mode `execute` script, and
that row stays visible because it can contain other code.
## Per-harness registration
`server.ConfigureChatTools(store, addr)` runs at startup, before any harness
starts, and installs `chattools.Registry`. The registry's base URL is the
loopback address of `-addr`; a wildcard address maps to 127.0.0.1, and a
non-loopback-only bind disables the tools with a log line.
- **Claude Code.** Started with `--mcp-config '{"mcpServers":{"pipelines":{"type":"http","url":…}}}'`
and `--allowedTools mcp__pipelines`. Both direct and hosted chats get the
session token.
- The config is added to the operator's own MCP servers; it does not replace
them.
- If a start fails with these flags, the chat logs the reason and retries
once without them (`session.toolsOff`).
- **Codex.** `thread/start` gets `config.mcp_servers.pipelines.url` with a
pending token, which is then bound to the returned thread id.
- A `thread/resume` (after "thread not found") registers the thread's
session token.
- If the App Server refuses the override, the thread starts or resumes
without it, and this is logged.
- Threads already loaded by the Codex desktop app before Pipelines touched
them have no tools.
- **OpenCode.** The spawned chat service gets
`mcp.pipelines = {type: "remote", url, enabled: true}` through
`OPENCODE_CONFIG_CONTENT`. This covers both `OpenChat` and the owned detached
`EnsureOwnedChatService`.
- The operator's chat overrides win.
- If the service does not start with the tools, it starts without them, and
this is logged.
- An operator-supplied endpoint (`-opencode-endpoint`) is not reconfigured,
so it has no tools.
- **Pi.** There is no Pi chat harness in this repository (chat backends are
opencode, claude and codex), so it is not supported.
**OpenCode attribution.** One OpenCode service serves every chat, and its MCP
requests carry no session (verified: OpenCode 2.0.32 sends only
`User-Agent: opencode/latest/2.0.32/cli` and `Mcp-Protocol-Version`).
A call on the instance token waits up to 3 s for exactly one chat whose
retained stream shows that call running. That means one of:
- a `pipelines_<tool>` part with identical arguments, or
- a running code-mode `execute` script containing `pipelines.<tool>(`. If
several such scripts are running, they are told apart by the argument
values.
When no chat matches, or several still do, the call is refused. It is never
guessed into a chat. Only chats Pipelines is observing can use the tools.
## Verification
- Go:
- `go test ./internal/server/ ./internal/chattools/ ./internal/pipeline/ ./internal/protocol/ ./internal/codex/ ./internal/codexhost/ ./internal/claudehost/`: all `ok` (server 72 s).
- `go test ./internal/claude/ ./internal/opencode/`: `ok` under `umask 022`.
Under this sandbox's `umask 077`, the existing tests
`TestServiceTokenIsPassedOnlyFromAPrivateFile` and
`TestStageDiscoveryRequiresPrivateRegistrationBeforeSessionWork` fail,
because both depend on 0644 file creation. They are unrelated to this
change.
- New Go tests:
- `chattools/mcp_test.go`: handshake, list, call, refusals, harness config.
- `server/chat_tools_test.go`: token isolation, pending binding, path
containment, image URLs, HTML limit and retention, notify rate limit,
instance and code-mode attribution.
- `server/chat_tools_http_test.go`: endpoint on the real listener,
`loopbackBase`.
- `claude|codex|opencode/pipelines_tools_test.go`: registration config and
the refuse-then-start-without fallback.
- `cd web && node --test test/`: `ℹ tests 18 … ℹ pass 18 ℹ fail 0`.
- Flutter:
- `tool/memwatch/full.sh test/features/chat`:
`memwatch: PASS: 1080 passed, 0 failed, 0 skipped; peak 2204 MB; 89s; exit 0 (test/features/chat)`.
- `tool/analyze.sh`: `0 errors, 8 warnings`. All 8 are the existing
`invalid_null_aware_operator` warnings in the unchanged
`integration_test/discovery_probe_test.dart`.
- Real end-to-end check, run on 2026-11-09:
- Setup: a scratch `pipelines-server` on `127.0.0.1:53834`, with
`HOME=/tmp/pl-e2e-tools/home`, a scratch database,
`PIPELINES_SHUTDOWN_SESSIONS=stop` and ntfy unset.
- OpenCode 2.0.32 (its default free model) was asked to `show_file
~/scratch/checker.png` and to `show_html` an SVG circle. Its `execute`
script returned "Shown in the operator's Pipelines chat: checker.png (217
bytes)" and "Shown … (137 bytes, sandboxed…)".
- `chat_subscribe` returned both items. The file item had
`path=/private/tmp/pl-e2e-tools/home/scratch/checker.png`,
`mime=image/png` and `size=217`. `/files` served that path with
`200 image/png 217`.
- `notify` was recorded as not delivered: "phone notifications (ntfy) are
not configured on this Pipelines server".
- A POST carrying an `Origin` header got `403`.
- The first OpenCode attempt was refused, and this is how code-mode
attribution was found to be necessary.
- Not run end to end:
- Claude Code under the isolated HOME answered "Not logged in", and
`~/.config/pipelines/claude-token` is empty.
- Codex has no app-server under the isolated HOME.
- Both registrations are covered only by unit tests against fake CLI
arguments and a fake App Server.
- Screenshots, real Chrome via `tool/capture` on the full-app fixture with
`?chats=1&timestamps=1&shown=1&files=1`, all in
`notes/images/chat-pipelines-tools/`:
- `cards-light-880.png`: image, markdown card with first lines, HTML
diagram with Open, and notify row.
- `cards-dark-880.png`
- `cards-cupertino-560.png`
- `html-open-light-880.png`: full-screen HTML with zoom.
- `image-open-dark-880.png`: the existing viewer with download.
- Mobile (Android/iOS WebView, native thumbnails) is unverified: no device run.