Files
pipelines/README.md
T
alan a70e3aef4b Install Pipelines natively by default on macOS and Linux
Add scripts/setup-native.py and make ./install (and --native) build the server, the CLI
and ntfy from source into ~/.local/share/pipelines, detect an existing host OpenCode
and Pi instead of installing them, install the packaged authoring skill, migrate an
existing Docker database and ntfy auth store, save the host client default and start
both services. ./install --docker keeps the previous Docker server + ntfy
installation unchanged.

ntfy is compiled from its pinned Go module on every host because upstream publishes
no macOS server binary (its darwin asset is built with -tags noserver); its cgo
SQLite driver is why a C compiler is now required natively. The ntfy credential
bootstrap and the client default writer are shared by both installers,
installation.Load resolves symlinked prefixes, the Docker migration copies the
SQLite write-ahead log so the newest definitions and runs are not lost, and the
account subcommands read the generated configuration through NTFY_CONFIG_FILE.

Verified on macOS: 115 Python tests, go build/vet/test, and scratch installs that
migrate 9 pipelines and 31 runs, build ntfy v2.30.0 and start both services healthy
on spare ports before stopping them.
2026-10-13 17:20:14 +01:00

382 lines
19 KiB
Markdown

# Pipelines
Configured OpenCode/Pi harnesses, separate model defaults, runner-enforced review
limits and selected run-local context: [guide](docs/harnesses-and-control.md).
Reusable, sequential workflows for your repositories, with a Go CLI/TUI,
OpenCode model stages, retained run history, and optional phone notifications.
**First-version preview:** a single-user tool for supervised work. Start with a
disposable repository, review edits and command evidence, and expect rough edges.
See the [tester walkthrough and known limits](docs/first-version-testing.md).
Stage results include [retained, highlighted workspace diffs](docs/stage-file-diffs.md)
for newly executed visits, alongside their reports and audit evidence.
Use `:export` / `:import` to [select and transfer pipeline definitions](docs/pipeline-transfer.md)
between servers as portable ZIP bundles.
[Real-provider reference workflows](docs/reference-workflows.md) exercise structured
handoffs, file edits/verification, failure routing and reported-usage budgets with
independent acceptance checks and retained evidence. Live inference is opt-in.
## In action
Real terminal captures from a live Pipelines instance on 12 October 2026.
The execution views show retained runs from 11 October 2026.
**Workflow:** plan, implement, review, and UI review, with rejection routes back
to implementation.
![Workflow graph with planning, implementation, review, and UI review stages](docs/images/workflow.png)
**Execution:** a completed plan → implement → review run, with the review result
and the executed route highlighted.
![Successful three-stage execution with retained review output](docs/images/execution.png)
**Harnesses:** OpenCode and Pi stages in the same successful Docker-backed run.
![Successful OpenCode and Pi execution with its retained result](docs/images/opencode-pi-execution.png)
## Install
`./install` installs Pipelines **natively on the host by default** (macOS and
Linux); `./install --docker` installs the **Docker server + ntfy** explicitly.
Native setup detects (but never installs) Go, python3, git, curl and a C compiler
(`cc`/`clang`/`gcc`), plus an existing `opencode`/`pi`; it builds `pipelines-server`,
`pipelines` and ntfy from source into `~/.local/share/pipelines` (honouring
`PIPELINES_PREFIX`) and generates an ntfy configuration mirroring the Docker
settings. ntfy is compiled locally because upstream publishes no macOS server
binary, and the C compiler is required because its SQLite driver is cgo. Docker
setup needs Docker Compose and Python 3 and includes OpenCode/Pi in the persistent
server container.
For the host CLI/TUI walkthrough below, also install Go 1.27+; the Docker
installer installs/updates those clients when host Go is available.
```sh
git clone https://git.a-wan.net/alan/pipelines.git
cd pipelines
./install # native host install (Go, python3, git, curl, cc, opencode)
./install --docker # explicit Docker services + container provider login
```
Native setup reuses your existing host OpenCode login and never installs,
upgrades or downloads OpenCode/Pi. Start it with
`PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start`.
When Docker volumes are present it migrates the existing database and ntfy auth
store (never overwriting newer native data; use `--replace-data` to opt in). See
[installation](docs/install.md) for details.
The installer requires this Git checkout (including `.git`); a standalone source
archive cannot self-install. Without host Go you can still use the
Docker-contained TUI/CLI. Run Compose from the checkout with `--project-directory`
so it finds `compose.yaml`, and pass `--workdir "$PWD"` so the container keeps the
actual (disposable) workspace rather than its own image directory:
```sh
# TUI (allocate a TTY)
docker compose --project-directory /absolute/path/to/pipelines exec --workdir "$PWD" pipelines pipelines
# Plain CLI command (no TTY)
docker compose --project-directory /absolute/path/to/pipelines exec -T --workdir "$PWD" pipelines pipelines <command>
```
Point a host client at the mapped host port with
`--url ws://127.0.0.1:<actual-mapped-port>/connect`, reading the port from
`docker compose --project-directory /absolute/path/to/pipelines port pipelines 8080`.
The container listens on port **8080**; only the default host mapping is 8081.
Prefer the installer-saved configured endpoint when present; never replace it with
a guessed default.
Go uses `GOBIN` or its normal `GOPATH/bin` default; put that directory on PATH.
The Docker installer automatically detects/configures LAN networking for ntfy;
you do not need to look up your IP or supply a URL/listen address.
Both Docker services bind `0.0.0.0`; reinstall normalizes those bind settings
without replacing your URLs, timezone, ports or credentials.
The installer builds only CLI/TUI clients on the host when Go is available;
the server and stage harnesses run in Docker. It configures ntfy accounts/token,
checks publishing/health, and opens container OpenCode provider login on first setup;
existing settings, credentials and volumes are retained on reinstall.
Docker provider credentials live in its persistent container volume, separately
from your host OpenCode login; reinstalling or reconfiguring reuses the existing
container accounts and volumes rather than replacing them. Setup asks for ntfy
account passwords, then your model-provider login and models. It selects separate
saved defaults in order: first the interactive harness model, then the
new-pipeline model, in two fuzzy-searchable pickers (the saved value is
preselected and marked `(current)`; typing filters, arrow keys or Tab navigate,
Enter confirms, and Esc cancels without changing anything). Model selection
performs no inference; only the OpenCode path validates the new-pipeline model
with a small real generation (provider charges may apply). Fresh
`./install --docker --harness pi` skips OpenCode login and that generation check. Pi has no
standalone `auth login`: setup opens a fresh `/login` session (regular TUI mode)
where you choose an authentication method, reuse existing volume credentials by
exiting, or sign in with an API key. Each
setup resolves the current supported OpenCode V2 release from the single
documented complete `git/matching-refs/tags/v2` response (no pagination; never the
moving `latest` tag) and the current Pi release from npm, builds `FROM` the
OpenCode image's immutable `sha256` digest, and verifies the built image reports
exactly those executable versions (and a Node-compatible Pi engine). It stops with
a clear error instead of silently substituting a stale pin when discovery, build
or compatibility fails. Use `./install --opencode` or `./install --pi` to
reconfigure login and models inside the existing container without restarting
services: an image matching the release identity needs no rebuild, while a dirty
source always invokes the Docker build and relies on layer caching. The built
image includes python3 so the installer/verification tests can also run inside
the container.
Configuration and data remain per-user. The native default installer is described
above and in [installation](docs/install.md).
Docker setup starts the server and saves its client endpoint (the container port
is **8080**; only the default host mapping is **8081**). That saved endpoint is
authoritative: prefer it and never replace it with a guessed default. Do **not**
also start `pipelines serve` for this setup. Explicit `--url` and `PIPELINES_URL`
override the saved endpoint; restart existing TUIs after setup.
Docker mounts your home and installation checkout at their original absolute paths
and uses your UID/GID, so the checkout supplies `compose.yaml` while
`--workdir "$PWD"` preserves the actual disposable workspace. Other host
repositories need a bind mount at the same absolute path; repositories on another
client computer are not automatically transferred.
These unauthenticated server ports are for trusted networks only—do not expose
them to the public Internet.
### Developer/legacy local server (unsupported)
For developing Pipelines itself or running a previous Python-supervised legacy
host setup, an explicit foreground server remains available. This is not the
native installer and is not part of Docker setup; fresh databases contain no
pipelines:
```sh
pipelines serve # foreground developer/legacy server
# In another terminal:
pipelines
```
Optional developer/legacy autostart at login (starts the server now):
```sh
pipelines service enable
pipelines service status
```
Uses user-level systemd on Linux or launchd on macOS; no sudo. Runs use the
directory you invoke `pipelines run` from. Preview with `--print`. Stop any existing
foreground/legacy server first. Disable with `pipelines service disable` before
uninstalling. See [installation](docs/install.md) for environment and boot options.
## Your first pipeline and run
Start in the repository you want the workflow to operate on:
```sh
cd /path/to/your/repository
pipelines
```
1. **Create:** press **n** or use **:new**. Choose the configured **OpenCode** or
**Pi** harness to author a workflow together, or **nvim** to edit the Markdown
yourself. Harness authoring runs in the Docker server; only the **nvim** editor
is required on your client computer. Describe the stages, their inputs/outputs,
and the tools they need. Templates (**b** on Home) are optional; a fresh
installation has no published pipelines.
To skip authoring, use **:templates**, choose one of the five generic starters,
and press **s** or Enter on **Create pipeline**. Review the prefilled name
(edit it if needed), then press Enter to create—no separate naming or editor
step. **e Customize** edits it first. Templates include a smaller graph preview;
**Tab** switches focus and space between the description and graph. Use **hjkl**
to pan a focused graph and **n/p** to browse templates.
2. **Keep a draft:** finish/exit the editor. Nothing is published just because you
edited it. Press **d** / **:drafts** to find retained drafts; **e** edits and
**v** validates the selected draft.
3. **Create/update:** press **s** in Drafts. If it has no name, type one into the name
prompt and press Enter. Then review and explicitly confirm submission. The
action is **Create pipeline** for a new definition or **Update pipeline** for
a new version. The server validates it and retains immutable versions; the draft stays.
4. **Run:** select the published pipeline, press Enter to open it, then **:run** to
start a run. Supply your task and review the confirmation. The run uses the
directory from which you launched the client, not the installer checkout.
5. **Inspect:** watch the full graph and stage progress. **Tab** expands/focuses
panes; **n/p** select recorded stage visits; **hjkl** pan a focused graph.
On a failed run, **D** / **:debug** opens a fresh OpenCode debugging conversation;
retry requires confirmation and restarts the pinned workflow, so earlier side
effects may happen again.
Home shows only **/** search and **:** commands as action hints; use the command
palette for creation, drafts, templates, history, status, logs, and quitting.
Quit with **:quit**; **q** does not exit. For a terminal-only run after publication:
```sh
pipelines run "your pipeline name" "Describe the task"
```
### Phone notifications
Open **:status** for the server and ntfy connection details. In the ntfy phone
app, add the displayed ntfy server, sign in as **phone-reader** with the password
you set during installation, and subscribe to the displayed topic. Stages must
include a `notify` block to send notifications; see [ntfy setup](docs/ntfy.md).
Installation checks publishing, not reception on your phone.
For Tailscale, both devices must be connected; use
`http://<server-Tailscale-IP>:8090`. Binding to `0.0.0.0` does not bypass firewalls:
UFW/Docker forwarding rules can allow LAN sources but drop Tailscale sources.
See [Tailscale troubleshooting](docs/ntfy.md#tailscale-address-times-out-while-lan-works).
## Uninstall
```sh
pipelines uninstall # remove Go binaries; keep history, drafts, and ntfy data
```
## A three-stage pipeline
**Plan → Implement → Review**. The first stage receives your task prompt.
Later stages receive the previous output or fields selected by an explicit
`memory` block. One outcome selects the next stage—no
parallel branches. Definitions use Markdown `output`, `prompt`, `tools`,
`configs`, and `edges` blocks. For example:
````markdown
# stage planner
```output
{ task, plan, design, reuse_targets, acceptance_checks, research }
```
```prompt
Turn input.prompt into a focused plan. Discover repository conventions and
reuse targets, and research official sources only when external knowledge is
actually required. State why when no research is needed.
```
```tools
read
glob
grep
websearch
webfetch
```
```configs
provider=opencode-go
model=deepseek-v4.1-flash
retries=2
timeout=1h
```
```edges
success -> implementer
blocked -> #failure
error -> #failure
```
````
The [complete three-stage example](examples/three-stage.md) adds the implementer
and reviewer: the planner may search/fetch official sources, the implementer has no
web tools and edits and runs checks from the planner's research, and the reviewer
independently inspects the real changes with its own shell access, receiving the
original task and the planner-owned acceptance checks rather than the
implementer's summary. The reviewer approves only when the applicable acceptance
checks were independently observed passing and no unresolved substantive issue
remains, and otherwise rejects with concrete findings. Review and explicitly
submit it:
```sh
pipelines validate examples/three-stage.md
pipelines submit examples/three-stage.md --name "three stage" --create-only
pipelines run "three stage"
```
Every submission is validated and creates an immutable version. Examples are
optional; nothing is imported automatically. The three stages give a visible
plan → implement → review path using your own accounts and repository; no
maintainer accounts or published definitions are assumed. Run from your
repository directory; stage tools are enabled automatically. Edits support any
language/file type and new files. No file list or typed RUN confirmation is
required.
## Useful commands
```sh
pipelines # dashboard; b browses templates
pipelines list
pipelines run # fuzzy selection + multiline prompt
pipelines run "bug fix" "Fix the layout" # run in the current repository
pipelines new "daily digest" "Summarize changes"
pipelines edit "bug fix" "Add a review step"
pipelines drafts --json
pipelines validate /path/to/draft.md
pipelines submit /path/to/draft.md # explicit publication
pipelines runs
pipelines inspect 1
pipelines serve # foreground server; Ctrl+C stops it
pipelines ping
pipelines version --json # local client identity (offline)
pipelines ping --json # running server identity
pipelines help edit
```
CLI `new`/`edit` open the configured harness (**OpenCode** or **Pi**) directly with
the Pipelines skill. The installed skill asks
missing questions; `pipelines new` needs no preliminary questionnaire. Changes stay
as drafts by default; CLI and TUI share validation and draft storage. Most
commands support `--json` for agents and scripts.
In the TUI, press **d** (or use **:drafts**) to browse retained drafts, including
CLI-created ones. **e** reopens a draft, **v** validates it, and **s** opens explicit
submission confirmation, asking for a name first if needed. Drafts remain retained
after submission. Drafts are portable: submit to the currently connected server.
When that differs from the origin, submission creates a new pipeline rather than
applying the origin's version number to an existing destination pipeline.
From a run, press **P** / **:pipeline** to open the pipeline version it used.
Initial run input also records `pwd`. Press **r** / **:retry** on a failed run
(or `pipelines retry <run-id>`) to start a new attempt with the same original
input, directory, and pipeline version. Completed side effects are not undone.
Retries restart at the entry stage with fresh OpenCode sessions, never the previous
conversation or intermediate stage output. Pipelines does not automatically delete
generation, agent, or formatting-repair sessions, including on failure/cancellation.
Primary session IDs are retained in run steps (`pipelines inspect <run-id> --json`).
On Home, **r** opens global run history; **/** fuzzy-searches it. Inside a pipeline,
**r** / **:runs** opens only that pipeline's history.
Quit the TUI with **:quit**; **q** does not exit. **Ctrl+C** remains an interrupt.
Starting/retrying/canceling runs and deleting drafts require confirmation. CLI
publication also prompts; use `--yes` only for explicitly approved automation.
**n / :new** and creating from a template offer a Tab-navigable, fuzzy-searchable
picker: **nvim** opens the draft Markdown file; **OpenCode**/**Pi** author it with
you. Canceling the picker creates no draft and publishes nothing.
Wide terminals show runs and pipelines side by side; narrow terminals stack them.
The TUI uses soft light/dark-adaptive cards, muted metadata, subtle selections,
and local loading animations instead of heavy character borders and inverted bars.
The dashboard includes all-history usage, success rate, average completed duration,
per-pipeline usage, and a 14-day activity chart. CLI: `pipelines stats [--json]`.
Use **:status** for two panels: server addresses and ntfy phone connection details.
CLI: `pipelines status [--json]`. Candidate URLs are not
verified reachable; publishing endpoints and credentials are not disclosed.
Retry is offered only after the failed run and its agent have fully stopped.
Graphs use content-sized boxes/columns and tightly spaced branch lanes while
retaining every stage and labeled box-and-arrow route. Incoming route IDs (left)
and outgoing outcomes (right) share rows to reduce box height. Both pipeline and
run views center the graph horizontally and vertically when it fits.
The run inspector shows the entire pinned graph above the details, dimming stages
and routes not executed. **g** focuses the graph; **hjkl** or arrows pan it.
The path and elapsed duration update while running. **a** opens the active/latest
OpenCode session's recorded tool evidence; this does not attach a live OpenCode UI.
Press **D** / **:debug** (or `pipelines debug <run-id>`) for a fresh OpenCode
conversation about that run. It uses CLI queries rather than resuming or retrying
the implementing agent. Pipeline **Edit** offers a fuzzy-searchable **nvim /
OpenCode** picker; Tab and Shift+Tab navigate it.
In the draft browser, **x** then **y** deletes the selected retained draft and
metadata. CLI: `pipelines draft-delete <draft.md>`. Published history is untouched.
Model results use native JSON-schema output where supported, precise parsing
errors, and bounded format-only repair without replaying tools. Retained failures
distinguish final-result validation from harness transport. Optional stage
permissions narrow file/command access without imposing a language-specific
default. See [model result recovery and permissions](docs/model-results.md).
The run graph header shows start time and elapsed/final duration, plus finish
time for completed runs.
More: [CLI](docs/cli.md) · [installation](docs/install.md) · [pipeline format](internal/author/skill/references/format.md).