Files
pipelines/app/notes/chat-pipelines-tools.md

11 KiB
Raw Permalink Blame History

Chat: Pipelines tools (show file / image / HTML, notify phone)

A chat's model can call four tools that act on Pipelines, not on the harness or the workspace. The tools come from an MCP server that runs inside pipelines-server. Each call adds an entry to the chat transcript, and the entry is persisted.

Tool Arguments Effect
show_file path, title? Shows an existing file inside the operator's home. The path can be absolute, ~/…, or relative to the chat's directory. Images render inline; other files render as a preview card.
show_image url_or_path, title?, alt? Shows a home image file (it must be image/*) or an https:// URL.
show_html html, title?, height_hint? Renders model-authored HTML or SVG inline in a sandbox, with an Open button for full screen.
notify title, message, priority? Sends a phone notification through the existing ntfy "system" destination. Tapping it opens pipelines://chat/<id>.

Each tool returns one short text line to the model. A refusal comes back as isError together with the reason.

Security model

  • Endpoint. The endpoint is POST /mcp/chat/<token> on the server's own listener. It speaks JSON-RPC 2.0 over MCP streamable HTTP: initialize, ping, tools/list, tools/call, and notifications (answered 202).
    • GET and other methods get 405.
    • The protocol version is negotiated: 2025-11-25, 2025-06-18, 2025-03-26 or 2024-11-05.
    • Batches are refused, and request bodies are capped at 4 MiB.
    • Requests that carry an Origin header (from a browser) get 403, as do requests from non-loopback peers. Unknown tokens get 404.
  • Not exposed through the web. web/serve.mjs and the peer relay forward only an allowlist (/connect, /files, /artifacts/, /chat-uploads, /updates/). Both new tests (web/test/mcpRefused.test.mjs and the peerRelay test) confirm /mcp… never reaches the server, including for a signed-in browser and %6dcp-encoded paths.
  • Tokens. Each token is 256 random bits, stored in chat_tool_tokens with the scope it serves:
    • session:<id>: one chat. Claude Code and Codex use this, and the token stays stable across restarts so detached hosts keep working.
    • pending:: a Codex thread before its id is known. Calls are refused until the token is bound.
    • instance:opencode: the shared OpenCode service. See "OpenCode attribution" below.
  • Paths. Paths use the same containment as /files: expandHomePath, then resolveInsideHome (resolves every symlink), then "regular file only", then the specialFilesystem check. The item records the real path, and the client loads it from the chat's own machine's /files (or /peers/<id>/files for a peer).
  • Image URLs. Only https URLs without credentials, at most 2048 chars. The client fetches them with a plain network image, without cookies.
  • HTML views. These use the same inert view as artifact HTML previews:
    • On web, an iframe with sandbox="allow-popups allow-popups-to-escape-sandbox" and srcdoc. There are no scripts, no same-origin access, no forms and no top navigation.
    • Links are rewritten first: #anchors scroll inside the frame, http(s)/mailto links open a new tab, and every other link is made inert.
    • On Android/iOS, a WebView with JavaScript disabled that refuses navigation.
    • Interactivity is therefore limited to CSS (:hover, <details>) and SVG. The tool description tells the model this.

Limits

  • show_html documents are refused, not truncated, above 512 KiB. The model is told the size and the limit.
  • At most 100 items are retained per chat. HTML bodies are kept only for the newest documents within 2 MiB per chat; older documents keep their entry but are marked expired, and the card says "This view is no longer retained".
  • notify allows 5 per minute per chat (a sliding window, held in memory). Rate-limited calls record nothing.
    • An undelivered notification is recorded with its reason, for example ntfy not configured or switched off.
    • priority is recorded, but ntfy always publishes at the default priority, because stage.PublishNtfy has no priority parameter.

Protocol and persistence

Items are stored in chat_shown_items (pipeline store). There are two ways they reach clients:

  • The chat_subscribe reply carries chat_shown: {session_id, items} with every retained item, oldest first.
  • A pushed chat_shown frame carries one new item.

The client interleaves items into the transcript by created_at. Items older than the loaded history wait until that page loads. In a multi-machine chat, the session id is rewritten by ChatServerGateway, and file URLs resolve against the chat's own machine endpoint.

The generic tool rows for mcp__pipelines__* (Claude), pipelines_* (OpenCode tool parts) and pipelines/* (Codex) are hidden; failed calls stay visible. OpenCode V2 runs MCP tools from its code-mode execute script, and that row stays visible because it can contain other code.

Per-harness registration

server.ConfigureChatTools(store, addr) runs at startup, before any harness starts, and installs chattools.Registry. The registry's base URL is the loopback address of -addr; a wildcard address maps to 127.0.0.1, and a non-loopback-only bind disables the tools with a log line.

  • Claude Code. Started with --mcp-config '{"mcpServers":{"pipelines":{"type":"http","url":…}}}' and --allowedTools mcp__pipelines. Both direct and hosted chats get the session token.
    • The config is added to the operator's own MCP servers; it does not replace them.
    • If a start fails with these flags, the chat logs the reason and retries once without them (session.toolsOff).
  • Codex. thread/start gets config.mcp_servers.pipelines.url with a pending token, which is then bound to the returned thread id.
    • A thread/resume (after "thread not found") registers the thread's session token.
    • If the App Server refuses the override, the thread starts or resumes without it, and this is logged.
    • Threads already loaded by the Codex desktop app before Pipelines touched them have no tools.
  • OpenCode. The spawned chat service gets mcp.pipelines = {type: "remote", url, enabled: true} through OPENCODE_CONFIG_CONTENT. This covers both OpenChat and the owned detached EnsureOwnedChatService.
    • The operator's chat overrides win.
    • If the service does not start with the tools, it starts without them, and this is logged.
    • An operator-supplied endpoint (-opencode-endpoint) is not reconfigured, so it has no tools.
  • Pi. There is no Pi chat harness in this repository (chat backends are opencode, claude and codex), so it is not supported.

OpenCode attribution. One OpenCode service serves every chat, and its MCP requests carry no session (verified: OpenCode 2.0.32 sends only User-Agent: opencode/latest/2.0.32/cli and Mcp-Protocol-Version).

A call on the instance token waits up to 3 s for exactly one chat whose retained stream shows that call running. That means one of:

  • a pipelines_<tool> part with identical arguments, or
  • a running code-mode execute script containing pipelines.<tool>(. If several such scripts are running, they are told apart by the argument values.

When no chat matches, or several still do, the call is refused. It is never guessed into a chat. Only chats Pipelines is observing can use the tools.

Verification

  • Go:
    • go test ./internal/server/ ./internal/chattools/ ./internal/pipeline/ ./internal/protocol/ ./internal/codex/ ./internal/codexhost/ ./internal/claudehost/: all ok (server 72 s).
    • go test ./internal/claude/ ./internal/opencode/: ok under umask 022. Under this sandbox's umask 077, the existing tests TestServiceTokenIsPassedOnlyFromAPrivateFile and TestStageDiscoveryRequiresPrivateRegistrationBeforeSessionWork fail, because both depend on 0644 file creation. They are unrelated to this change.
  • New Go tests:
    • chattools/mcp_test.go: handshake, list, call, refusals, harness config.
    • server/chat_tools_test.go: token isolation, pending binding, path containment, image URLs, HTML limit and retention, notify rate limit, instance and code-mode attribution.
    • server/chat_tools_http_test.go: endpoint on the real listener, loopbackBase.
    • claude|codex|opencode/pipelines_tools_test.go: registration config and the refuse-then-start-without fallback.
  • cd web && node --test test/: ℹ tests 18 … ℹ pass 18 ℹ fail 0.
  • Flutter:
    • tool/memwatch/full.sh test/features/chat: memwatch: PASS: 1080 passed, 0 failed, 0 skipped; peak 2204 MB; 89s; exit 0 (test/features/chat).
    • tool/analyze.sh: 0 errors, 8 warnings. All 8 are the existing invalid_null_aware_operator warnings in the unchanged integration_test/discovery_probe_test.dart.
  • Real end-to-end check, run on 2026-11-09:
    • Setup: a scratch pipelines-server on 127.0.0.1:53834, with HOME=/tmp/pl-e2e-tools/home, a scratch database, PIPELINES_SHUTDOWN_SESSIONS=stop and ntfy unset.
    • OpenCode 2.0.32 (its default free model) was asked to show_file ~/scratch/checker.png and to show_html an SVG circle. Its execute script returned "Shown in the operator's Pipelines chat: checker.png (217 bytes)" and "Shown … (137 bytes, sandboxed…)".
    • chat_subscribe returned both items. The file item had path=/private/tmp/pl-e2e-tools/home/scratch/checker.png, mime=image/png and size=217. /files served that path with 200 image/png 217.
    • notify was recorded as not delivered: "phone notifications (ntfy) are not configured on this Pipelines server".
    • A POST carrying an Origin header got 403.
    • The first OpenCode attempt was refused, and this is how code-mode attribution was found to be necessary.
  • Not run end to end:
    • Claude Code under the isolated HOME answered "Not logged in", and ~/.config/pipelines/claude-token is empty.
    • Codex has no app-server under the isolated HOME.
    • Both registrations are covered only by unit tests against fake CLI arguments and a fake App Server.
  • Screenshots, real Chrome via tool/capture on the full-app fixture with ?chats=1&timestamps=1&shown=1&files=1, all in notes/images/chat-pipelines-tools/:
    • cards-light-880.png: image, markdown card with first lines, HTML diagram with Open, and notify row.
    • cards-dark-880.png
    • cards-cupertino-560.png
    • html-open-light-880.png: full-screen HTML with zoom.
    • image-open-dark-880.png: the existing viewer with download.
  • Mobile (Android/iOS WebView, native thumbnails) is unverified: no device run.