Files
pipelines/docs/cli.md
T
alan c700b28e6a Add cross-system read-only search
One query now finds anything in the system over the wire and from the CLI.

- protocol: add Search/SearchResults message types and a SearchRequest
  (query, optional kind filter, optional limit) / SearchResponse (typed,
  ranked matches plus truncation and scan-bound counters) pair, with typed
  payloads for pipeline, version, run, step/visit, field text and artifact.
- store: add Store.Search, a read-only query over pipelines,
  pipeline_versions, pipeline_runs, run_steps (+step_control) and
  step_artifacts. Ranks exact name (3) > name substring (2) > body text (1),
  breaks ties by recency then a stable kind/target order, and reports the
  matched field. Bounds work with a 50-default/200-max result cap, a 2000-row
  per-kind scan limit and a 2000-step / 1 MiB field-text scan budget.
- terms: a bare status word or lone date expression (today, yesterday, this
  week, YYYY-MM-DD) selects runs; anything else stays free text.
- server: dispatch protocol.Search to stores[0].Search; read-only and
  unconfirmed.
- cli: add `pipelines search QUERY [--type KIND] [--limit N] [--json]`, help
  text and a BROWSE line.
- store: add OpenReadOnly so the live-database check never runs DDL.
- docs: document the grammar, kinds, ranking and bounds in docs/cli.md,
  scope the TUI omnisearch to pipelines in docs/ui-ux-review.md, and add a
  search bullet to the packaged operations reference.
- tests: protocol round trip, store/ranking/term-parsing/limit/scan-bound
  unit tests, a server end-to-end test, and an env-guarded live check.

No new dependencies; go.mod/go.sum unchanged.
2026-10-16 15:43:25 +01:00

29 KiB

CLI and agent authoring

In the TUI, publishing is labelled Create pipeline for a new definition and Update pipeline for a new immutable version. The CLI submit command and :submit shortcut remain compatible. Templates offer description/graph panes: Tab expands the focused pane, n/p selects a template, and hjkl pans the focused graph. Create opens one confirmation with an editable, prefilled template name; Enter creates the pipeline without a separate naming or editor step. Duplicate names are rejected rather than updating an existing pipeline.

pipelines help lists commands; pipelines help edit (or edit --help) shows command-specific flags and examples. Installed commands work from any directory. The installed command is a compiled Go binary. Optional autostart uses the OS user service manager directly. Legacy Python-managed installations require an explicit PIPELINES_PREFIX; no configuration is inferred from the executable's location. For source development, use go run ./cmd/pipelines help; start the native server separately. Set PIPELINES_URL or pass --url to select another server.

Dashboard and status

pipelines --url ws://HOST:PORT/connect ping --json returns pong with the running server's compiled build.version, full Git build.commit, and build.dirty. Uncommitted builds have a -dirty suffix. Missing metadata is explicit (commit: "unknown", dirty: null); older servers omit build. This identifies the server binary, not the CLI or files mounted into the container. Plain ping also prints the server version/commit. pipelines version --json (or pipelines --version) identifies the local client without connecting to a server. Include both identities when reporting a problem; they can differ.

Home's r opens all runs, not only failures or the newest 50. Inside a pipeline, r and :runs show only that pipeline's runs across its versions. In any run list, press / for fuzzy search, type to filter, and Enter to return to the results. Search includes pipeline name, version, status, outcome, stage and diagnostic; #123 searches run-ID prefixes. Esc while searching clears the filter without closing the list. The filter survives inspection/back and live list refreshes. Wide terminals show runs and pipelines side by side, with stacked panels on narrow terminals. Home run cards prioritize reported results, status, completed visits and duration; active cards also show the current stage. Start/finish times remain in run details. The usage summary is aggregated over all stored history, including retry attempts: total/active runs, outcomes, success rate, average completed duration, pipeline usage and a zero-filled 14-day daily activity chart (UTC). Success rate is succeeded / (succeeded + failed); active, canceled and interrupted runs are excluded. The average includes all runs with a finish timestamp. No model cost/token counts are inferred from missing data. pipelines stats --json exposes these aggregates.

Use :status on any TUI page to open server and ntfy information. Scroll with arrows or j/k, PgUp/PgDn; r refreshes and Esc closes without changing the page. pipelines status [--json] returns the same server data, including hostname, start/uptime, the connected server endpoint/port, active interface IPs, ntfy publishing health, public phone URL/topic/port and candidate URLs. It does not start services or send a notification. The ntfy health probe has a two-second deadline; it checks the publishing server, not phone reachability.

IPs belong to the server computer or container, not necessarily the client. Candidate IP/port combinations are not reachability checks: loopback bindings, firewalls, container port mapping and TLS can prevent access. ntfy candidates are only derived when a configured endpoint identifies this computer; remote ntfy hosts are not misrepresented as local. Publishing URLs and credentials are never returned. Missing public phone configuration is labeled explicitly. pipelines service status remains the separate OS autostart status command.

pipelines search QUERY [--type KIND] [--limit N] [--json] runs one read-only query across the whole system: pipeline names and their versions, run identity, status and dates, stage and visit identifiers, artifact names and types, and the text of stage inputs, outputs, evidence and transcripts. It never reads artifact contents, never makes a per-result follow-up request, and asks for no confirmation.

  • Free text matches case-insensitively. A bare status word (queued, running, succeeded, failed, canceled, interrupted) selects those runs, and a lone date expression (today, yesterday, this week — Monday 00:00 local — or a literal YYYY-MM-DD) selects the runs created in that window; any other text, including a status word inside a sentence, stays free text.
  • --type is the only narrowing mechanism and takes one kind: pipeline, version, run, step, field or artifact. An unknown kind is an error, never a silently ignored filter.
  • Ranking: an exact name match (score 3) beats a name substring (2) beats a body-text match (1); equal tiers break by recency, then a stable kind/target order. Every result carries score, matched_field and recency so a client can re-rank.
  • Bounds: the result cap defaults to 50 and clamps to 200 (0 = default); every kind scans at most 2000 rows, and field-text scanning stops at 2000 steps or 1 MiB decoded. truncated plus scanned_steps, scanned_bytes, cap and scan_bound_bytes report when a bound applied.
  • Targets are stable and navigable: pipeline:<id>, pipeline:<id>:v<version>, run:<id>, run:<id>:<sequence>, run:<id>:<sequence>:<field>, or an artifact URL served by the artifact endpoint.
pipelines search "release notes" --json
pipelines search failed --json
pipelines search 2026-10-16 --type run --json
pipelines search "acceptance check" --type field --limit 10

Draft-first harness workflow

pipelines new                               # the configured harness asks what to build and names it
pipelines edit                              # fuzzy pipeline selection
pipelines edit "bug fix"                    # ask the harness what to change
pipelines edit "bug fix" "Add a review step"
pipelines new "release notes" "Summarize changes and notify my phone"
pipelines new "Text summary" --template summary

new opens the configured harness (Pi or OpenCode; host by default, or Docker-linked when a checkout is linked) immediately when its name/request is omitted. The skill asks only for missing decisions and records the chosen name in draft metadata. edit accepts an exact name or ID, with names taking precedence over numeric IDs. Both retain private drafts and launch the harness once with a short @pipelines reference and context-file path. There is no preliminary harness API call, temporary service, session creation, or instruction preload. The prompt's CLI prefix is Docker-aware: a Docker-linked profile embeds the in-container pipelines executable and, for this Compose service's own loopback endpoint, scoped transport aliases. Loopback is recognized only for parsed IP literals and the exact localhost name (case-insensitive), so a remote DNS name such as 127.remote.example stays remote with no Docker probe. The single bounded docker compose port probe must return a complete host:port binding that covers the requested loopback address and port (wildcard or matching family); malformed or non-matching bindings fail clearly, and a loopback wss endpoint is rejected as having no supported local TLS mapping rather than downgraded to plaintext. nvim remains a client-side editor. ./install installs/updates the global skill and format reference. Plain go install users run pipelines skill install separately. Shared login/config and user-modified skills are preserved; harness permissions remain in effect.

The skill encourages editing only the draft, running server validation, summarizing the changes, and leaving submission to the user. This is an instruction, not a security sandbox: an agent with CLI/shell access can submit when explicitly asked. Every submission, including agent-issued submit/rpc, still uses server validation.

Drafts in the TUI

The server runs one active run per resolved workspace. A new request key in a different workspace is admitted immediately and executes concurrently; a second run in a workspace an active run is already changing is refused, naming the conflicting run and advising you to wait or use a different workspace (for example another checkout of the repository). Reusing the original request key reattaches idempotently.

Use :templates for five simple, manual-only starters: task, summary, review, research, and plan/implement/review. Submit template (s or Enter) asks for a name without opening an editor. Enter saves a retained draft, then a separate confirmation publishes it as a new pipeline. Esc before naming creates nothing; Esc at publication keeps the draft. Customize (e) opens the editor picker instead. Both paths inherit the configured default model and preserve existing pipelines/history. The research starter requires configured web search support. Starter stages use goal-appropriate tools: the task stage and the three-stage implementer edit and run checks, the task, planner and review stages may search/fetch official sources only when external knowledge is genuinely needed, the three-stage implementer has no web tools and follows the planner's research, the standalone review stage never edits or runs commands and says so explicitly, and the three-stage reviewer independently inspects the real changes and receives the planner-owned acceptance checks rather than the implementer's summary, approving only when the applicable acceptance checks were independently observed passing and no unresolved substantive issue remains. Editing a bundled template source changes no installed or published definition; a starter becomes an immutable version only after explicit authoring and publication.

Home's n / :new and Customize (e) on a template first ask how to author the new pipeline: nvim opens a retained Markdown draft file, or a configured harness (Pi/OpenCode) opens the collaborative session with @pipelines. Tab/Shift+Tab or arrows select; typing fuzzy-searches the choices; Enter launches and Esc cancels without creating a draft. The choice overrides VISUAL/EDITOR. New drafts preserve the create-only publication guard and still require explicit submission unless you already published them from the harness. The configured harness can ask for a missing name; for an unnamed nvim draft, pipelines submit <draft.md> --name NAME names and publishes it. CLI pipelines new retains its existing direct-harness behavior; --no-open creates the file without launching an editor.

Press d on the dashboard or use :drafts to browse the same retained drafts as pipelines drafts. Use n/p to select, j/k to scroll source, e/Enter to reopen the existing draft, v for server validation, and s for submission confirmation. r refreshes the list. Drafts can be validated, edited, and submitted to whichever server you are connected to. Same-server submissions preserve create-only/version guards. A different destination imports the draft as a new pipeline, ignoring the origin's version number and refusing an existing destination name; the confirmation shows the destination, and opening, validating or submitting a draft whose recorded server differs from the configured one prints a notice naming both URLs so it is never silently retargeted. Retained source and origin metadata stay unchanged. For an edit draft, CLI submit --expected-version N can explicitly approve an update against version N on the destination (or --force can explicitly skip that version check). For unnamed drafts, submission asks you to type a pipeline name directly in the TUI. Enter saves the name in retained metadata and opens publication confirmation; it does not publish yet. Esc keeps the draft without saving unfinished name input. Draft selection/editor completion never publishes it. On editor exit, the connected TUI checks the authoritative server before asking to submit. If the draft source is already published (including a newer version after the draft's base that was subsequently superseded), it skips confirmation and reloads the latest pipeline/page and Home list. Local-only changes still need explicit submission. A failed check retains the draft and says publication could not be verified; it never assumes publication from an editor's exit code. Original draft conflict guards and retained draft files remain unchanged. New/edit drafts open the configured default harness; --harness opencode|pi|nvim selects another configured harness or nvim. TUI pickers show only configured harnesses, default first, plus nvim. See harnesses and runtime control for fuzzy model setup, separate defaults, shared memory and runner-owned limits.

For agents or scripts that should not open a terminal:

pipelines new "release notes" --no-open
pipelines edit "bug fix" --json
pipelines drafts --json
pipelines validate /path/to/pipeline-123.md --json
pipelines submit /path/to/pipeline-123.md --json

--json never opens an editor or picker; provide a name explicitly. It returns draft metadata including path, name, server, create_only, and base_version. --no-open prints the path and review commands. $PIPELINES_DRAFT_DIR overrides the shared cache directory. CLI and TUI drafts use private Markdown plus metadata sidecars; submit FILE can publish either. External files require --name NAME.

New drafts use create-only submissions and cannot overwrite existing names. Edits carry a latest-version guard checked atomically by the server. On conflict, inspect/export the latest version and reconcile the draft. --force explicitly bypasses the version guard only, never syntax, graph, model, tool, schedule, or notification validation. Original versions/sources remain immutable. Submitting activates included schedules, so review them before publishing.

CLI/TUI capability map

TUI action Agent-capable CLI
Browse/search pipelines list --json, show NAME --json
Search everything search QUERY --json (see Search)
Original source and history export NAME --version N, versions NAME --json
Browse templates/source templates --json, template KEY --source
Create or edit a definition new, edit, drafts
Validate and publish validate FILE, submit FILE
Start a run run NAME PROMPT --yes --json
Run history/inspection runs --json, inspect ID --json
Cancel cancel ID --json
Connection and phone settings connection --json, ping --json
Server/ntfy status popup status --json
Usage statistics/charts stats --json
Legacy operational diagnostics managed logs, doctor

The TUI does not spawn CLI processes for each request. Both front ends share internal/client's command transport and internal/draft's persisted drafts; the server owns all validation, persistence, execution, and schedules. rpc exposes the full protocol as an escape hatch, so no server command is TUI-only:

printf '%s' '{"type":"get_pipeline","pipeline_id":1,"version":1}' | pipelines rpc -

Existing protocol commands: ping, list_pipelines, get_pipeline, list_templates, validate, submit, run_pipeline, list_runs, get_run, cancel_run, retry_run, get_status, search. list_runs also returns all-history usage aggregates. See internal/protocol/message.go for JSON envelopes. Mutating RPCs have real effects; they are not previews. TUI-only animation, scrolling, and transient client logs are presentation, not separate server capabilities.

Scripted runs

On either the runs list or run inspector, press P (or use :pipeline) to open the selected run's exact pipeline version. Navigation does not cancel the run.

Initial input includes pwd, the resolved run directory (normally the directory you invoked Pipelines from), alongside the task and any custom input fields. It is recorded by the server and shown in the run's Input section.

Press r or use :retry on a failed run to restart it from the first stage. CLI equivalent: pipelines retry <run-id> (optional --yes, --json, --wait, --wait-timeout, and --request-key). The new attempt retains the initial input, directory, pinned pipeline version, and edit scope, and records retry_of. Original history is unchanged. Legacy runs gain pwd from their recorded workspace on retry. Retry replays the entire workflow; earlier edits or notifications are not undone and may happen again. A missing original directory blocks retry rather than silently using your current directory.

Retry is hidden and rejected while a run is executing or agent cleanup remains active, even if a terminal status was already persisted. Run RPCs report executing alongside status so clients can distinguish cleanup from completion.

The run inspector's upper half shows the entire graph from the run's immutable pinned version using the shared renderer and viewport. Executed stages and taken routes retain their colors; unexecuted stages, unchosen routes and unreached terminals are dimmed. Queued runs show the full graph before the first visit. Repeated visits remain individually inspectable in the stage sidebar, while the graph highlights every audited route through a loop. On small terminals details take precedence. Older servers without a graph fall back to a matching loaded definition or the recorded execution path, never a different pipeline/version. Press g to focus the graph, arrows to pan, and j/k or PgUp/PgDn to scroll. Press g or Tab to return to details. When not focused, the graph follows the latest visit or selected stage. Existing polling refreshes the graph once per second, and running durations show elapsed time rather than “in progress.”

In the run inspector, n/p select the next/previous recorded execution visit, including repeated visits in loops, from any pane. Selection stops at the last visit or overview; it does not wrap. The selected stage has a full-row highlight; the focused stage/details pane has a highlighted title and pointer. In details, the section containing the top visible row is highlighted and named in the pane title. Tab cycles stages → details → graph (Shift+Tab reverses); g switches graph focus. Each graph stage title shows live progress from its latest execution visit. New run pages open with the graph focused and using roughly three quarters of the available height. Tab expands the focused pane and contracts the others; the stage list also widens when focused, and shrinks to a sidebar for details. Pending stages are dimmed. The graph is centered horizontally/vertically when it fits; the box-and-arrow diagram is retained at every viewport size, never replaced with a stage list to fit. Oversized diagrams remain pannable with h/j/k/l or arrows while the graph is focused; c recenters the selected/current stage. P opens the pinned pipeline (lowercase p remains an alias only on the runs list).

The pipeline page also gives most height to its focused graph or stage-details pane; Tab switches focus and resizes the two panes. Focused titles use the same background highlight as the run page. Graph boxes are sized to their contents rather than the longest label anywhere in the graph. Fork columns share only their own required width, route tracks are adjacent, and branch spacing follows stacked boxes rather than tall trunk/merge nodes. All stages, outcome labels, ports, and terminal arrows remain visible in the full pannable diagram. Incoming route IDs are shown on the left and outgoing outcome labels on the right of shared rows, without separate input/output sections or divider rows. This reduces height while allowing enough width to keep both labels readable. Failure-bound error exits remain at the bottom. Pipeline and run graph panes both center horizontally and vertically when the diagram fits, and clamp scroll/pan offsets on resize or graph changes.

Press a to select the active/latest retained configured-harness agent session and open its tool evidence. Session ID is available during execution; tool transcripts appear when persisted (currently after generation). This does not attach a live harness client or stream its messages. Supporting live attachment consistently across the shared and privately owned harness backends is deliberately deferred.

Investigating runs with a configured harness

On the runs list or inspector, press D or use :debug; CLI equivalent: pipelines debug <run-id>. This opens a fresh conversation in the configured harness (Pi or OpenCode), not the implementing session. It supplies exact commands for inspect <run-id> --json (including stage history, exact outputs, diagnostics, notifications, session IDs and bounded transcript previews), show <pipeline-id> --version N --json, and runs --pipeline <pipeline-id> --json. The commands reuse the same Docker-aware CLI prefix as authoring, so a Docker-linked profile runs them in-container against the original logical endpoint. All retrieval remains available through the CLI and can be repeated while execution is active. No new model API or live attachment integration is required. The injected server-side session evidence is the latest bounded, credential-redacted evidence for up to the 3 most recent session-bearing steps; when a preview is bounded or omitted, inspect the retained session itself rather than assuming a complete tool history.

debug --json returns the context and prompt without launching the harness; --no-open prints the prompt. --harness opencode|pi|nvim selects a configured harness or nvim; --opencode selects the host executable. Debugging works for active and terminal runs and does not automatically retry, cancel, or resume them. If the recorded directory is unavailable, the harness launches from your current directory and retrieves historical evidence through the CLI.

Editor choice and draft cleanup

On the pipeline detail page, e / :edit opens an editor picker: nvim or the configured harness (Pi/OpenCode, default first). Type to fuzzy-search; Tab/Shift+Tab (or arrows) choose; Enter opens; Esc cancels without creating a draft. Explicit choice overrides VISUAL/EDITOR. Both choices edit a retained copy with the normal version guard, never the published definition. Submission remains separate and explicit.

In the draft browser, x asks to delete the selected draft; y confirms and Esc or n keeps it. The CLI equivalent, pipelines draft-delete <draft.md>, is an explicit deletion command and needs no server connection; --json reports the result. It asks for confirmation; --yes explicitly approves scripted deletion. Only managed drafts inside the configured draft cache are accepted. Source and metadata symlinks/directories and external Markdown files are rejected. Published definitions and run history are not deleted. Cleanup can also remove orphaned metadata when a draft source was already removed.

cd ~/repos/project
pipelines run "bug fix" "Fix the layout" --json
pipelines run "bug fix" "Fix the layout" --json --wait --wait-timeout 10m
pipelines run "bug fix" "Fix the layout" --json --request-key my-unique-task

Runs default to the client's current directory and latest accepted version. The server resolves that directory without a repository allowlist. --version chooses an immutable version in JSON mode. Starting a run enables its stage-declared tools without a second approval. Editing supports all file types and new files within the run directory; no file list is required. Shell commands use the repository's actual tooling. --write-paths is an optional narrower scope, not a prerequisite. --yes skips action confirmation for explicitly approved automation; --allow-tools remains a compatibility flag and is not required. --wait polls until terminal status, returns nonzero on failure/cancellation, and does not cancel a run when waiting times out. --wait-timeout bounds only the client wait after admission and is independent of the per-request --timeout; a reached wait limit leaves the admitted run active and never cancels or replays it. When it is reached, inspect read-only with pipelines inspect <run-id> --json, or keep waiting by rerunning --wait and reusing the original --request-key rather than starting a duplicate run. Reuse --request-key to prevent duplicate work. A supplied pipeline/task opens an action-confirmation popup before starting with a live inspector (unless --yes was supplied). When a task is omitted, Enter inserts a newline and Ctrl+S opens the confirmation for the composed task. There is no file-list question or typed RUN confirmation. Shell runs with your user's authority: the run directory and tool policy are not an OS sandbox.

Run, retry, cancellation, CLI publication, and draft deletion require confirmation. CLI prompts go to stderr and default to No; use --yes for non-interactive approval. The TUI's run/retry/cancel popup accepts y; Enter does not implicitly approve, and n/Esc cancels without sending the action. Retry restarts with new sessions and original input/pwd, never the failed conversation; side effects may repeat. Publication retains its explicit submission confirmation and version conflict guards. Quit with :quit, not q; Ctrl+C remains an interrupt.

JSON results go to stdout, errors to stderr (JSON error envelopes when requested), and failures exit nonzero. Export/template --source deliberately prints raw Markdown. --timeout bounds each server request; --wait-timeout bounds the optional client wait independently and, when reached, leaves admitted work active. Neither changes the stage execution timeout.

Stage limits

Model-stage configs support these optional settings:

retries=2
timeout=1h
budget_usd=1.50
budget_tokens=50000

Retries default to 2 (three total attempts); timeout defaults to one hour per attempt. Omitted budgets are unlimited; configured budgets must be positive. Each budget covers all attempts, format-only repairs, and repeated visits to that stage in one run. A new run starts new accounting. Tokens include input, output, reasoning, and cache reads/writes. USD amounts use OpenCode's reported cost. Missing or invalid required accounting stops the stage rather than counting zero. Usage is polled during execution and read again afterward; an in-flight request may exceed a limit before usage arrives. These are not provider-side hard caps.

Automatic retries cover retryable transport failures and per-attempt timeouts. An explicit read-only stage (a tools block listing neither edit nor shell) may also retry a post-admission deadline or transport failure and re-emit a malformed final result up to retries; a stage whose tools include edit/shell or that has no tools block never replays admitted tool work or invalid results. Budget failures always stop. Cleanup must be confirmed before another attempt. Existing explicit retries=0 settings remain unchanged. The old 30-minute run deadline no longer truncates stage timeouts; the 30-step and five-visits-per-stage limits still apply. Native ntfy stages also accept timeout (default 1h), but never automatically retry potentially delivered notifications and do not accept model budgets.

Pinned definitions contain the configured limits. Run transcripts retain limits, attempt counts and reported budget usage, also shown in the TUI's Evidence pane. Exact stage outputs are authoritative; a transcript over the 32 KiB retained bound is stored as a valid, credential-redacted preview envelope that preserves available stage limits and failure/format diagnostics and names the retained session instead of slicing to malformed JSON.

Result-first views

Home shows retained result previews and completed visits alongside each run and pipeline. Pipeline pages show the latest result for the selected version, not a result from a different published definition. Run opens focused on results, with output before input and the graph below. Failed or ongoing runs preserve partial output and completed stage reports without labeling them final success. These previews quote recorded output; they do not independently verify model claims. Tab and Shift+Tab toggle Details ↔ Graph only. The visit selector lives inside Details; n/p still navigate recorded visits, including repeated stages, and 0 returns to the overview. j/k scroll the report; the focused graph uses hjkl for panning and c to center. Secondary progress/input/evidence/execution sections are collapsed initially. Small inactive graph previews collapse instead of showing severed boxes. See UI/UX review for screen-by-screen feedback.

Verification scope

CLI integration tests exercise draft/validate/submit/history/conflict handling and scripted run/inspection/cancellation through a real local WebSocket server (the run test uses a controlled executor, not a model). The isolated installed CLI also exercised the actual OpenCode API's session creation and skill activation, while capturing interactive launch arguments to avoid a model turn. Full model-driven authoring and Linux execution are not tested.