Files
pipelines/README.md
T
alan 476ecc781c Merge main into lane-srv3: combine the review-loop control change with main's notification and chat work
Keeps this branch's advisory review edge, bounded blocked_retryable implementer
edge and their tests alongside main's run-attention/ntfy notification work, the
split of the Android chat conversation screen, and the retention/history changes.
No conflicts: the two sides touch disjoint files.
2026-10-17 17:07:17 +01:00

419 lines
22 KiB
Markdown

# Pipelines
Configured OpenCode/Pi harnesses, separate model defaults, runner-enforced review
limits and selected run-local context: [guide](docs/harnesses-and-control.md).
Reusable, sequential workflows for your repositories, with a Go CLI/TUI,
OpenCode model stages, retained run history, and optional phone notifications.
**First-version preview:** a single-user tool for supervised work. Start with a
disposable repository, review edits and command evidence, and expect rough edges.
See the [tester walkthrough and known limits](docs/first-version-testing.md).
Stage results include [retained, highlighted workspace diffs](docs/stage-file-diffs.md)
for newly executed visits, alongside their reports and audit evidence. Stages can
also [record and serve their own artifacts](docs/stage-artifacts.md) - images,
screenshots, reports and logs - per step.
Use `:export` / `:import` to [select and transfer pipeline definitions](docs/pipeline-transfer.md)
between servers as portable ZIP bundles.
[Real-provider reference workflows](docs/reference-workflows.md) exercise structured
handoffs, file edits/verification, failure routing and reported-usage budgets with
independent acceptance checks and retained evidence. Live inference is opt-in.
## In action
Real terminal captures from a live Pipelines instance on 12 October 2026.
The execution views show retained runs from 11 October 2026.
**Workflow:** plan, implement, review, and UI review, with rejection routes back
to implementation.
![Workflow graph with planning, implementation, review, and UI review stages](docs/images/workflow.png)
**Execution:** a completed plan → implement → review run, with the review result
and the executed route highlighted.
![Successful three-stage execution with retained review output](docs/images/execution.png)
**Harnesses:** OpenCode and Pi stages in the same successful Docker-backed run.
![Successful OpenCode and Pi execution with its retained result](docs/images/opencode-pi-execution.png)
## Install
`./install` installs Pipelines **natively on the host by default** (macOS and
Linux); `./install --docker` installs the **Docker server + ntfy** explicitly.
Native setup detects (but never installs) Go, python3, git, curl and a C compiler
(`cc`/`clang`/`gcc`), plus an existing `opencode`/`pi`; it builds `pipelines-server`,
`pipelines` and ntfy from source into `~/.local/share/pipelines` (honouring
`PIPELINES_PREFIX`) and generates an ntfy configuration mirroring the Docker
settings. ntfy is compiled locally because upstream publishes no macOS server
binary, and the C compiler is required because its SQLite driver is cgo. Docker
setup needs Docker Compose and Python 3 and includes OpenCode/Pi in the persistent
server container.
For the host CLI/TUI walkthrough below, also install Go 1.27+; the Docker
installer installs/updates those clients when host Go is available.
```sh
git clone https://git.a-wan.net/alan/pipelines.git
cd pipelines
./install # native host install (Go, python3, git, curl, cc, opencode)
./install --docker # explicit Docker services + container provider login
```
Native setup reuses your existing host OpenCode login and never installs,
upgrades or downloads OpenCode/Pi. It writes host harness profiles (no Docker
link) for the harnesses it detects, so authoring works immediately, and clears a
stale Docker link when migrating a Docker install. After building, it warns if a
**different** `pipelines` resolves earlier on PATH than `<prefix>/bin/pipelines`,
naming that path and both versions, so a stale CLI is not left in front of a newer
server (it never modifies or upgrades that other binary). Start it with
`PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start`.
When Docker volumes are present it migrates the existing database and ntfy auth
store (never overwriting newer native data; use `--replace-data` to opt in). See
[installation](docs/install.md) for details.
The installer requires this Git checkout (including `.git`); a standalone source
archive cannot self-install. Without host Go you can still use the
Docker-contained TUI/CLI. Run Compose from the checkout with `--project-directory`
so it finds `compose.yaml`, and pass `--workdir "$PWD"` so the container keeps the
actual (disposable) workspace rather than its own image directory:
```sh
# TUI (allocate a TTY)
docker compose --project-directory /absolute/path/to/pipelines exec --workdir "$PWD" pipelines pipelines
# Plain CLI command (no TTY)
docker compose --project-directory /absolute/path/to/pipelines exec -T --workdir "$PWD" pipelines pipelines <command>
```
Point a host client at the mapped host port with
`--url ws://127.0.0.1:<actual-mapped-port>/connect`, reading the port from
`docker compose --project-directory /absolute/path/to/pipelines port pipelines 8080`.
The container listens on port **8080**; only the default host mapping is 8081.
Prefer the installer-saved configured endpoint when present; never replace it with
a guessed default.
Go uses `GOBIN` or its normal `GOPATH/bin` default; put that directory on PATH.
Running `pipelines` also compares its build/protocol with the server and warns (or
fails on a newer protocol) instead of dropping fields it does not model. The
Docker installer automatically detects/configures LAN networking for ntfy;
you do not need to look up your IP or supply a URL/listen address.
Both Docker services bind `0.0.0.0`; reinstall normalizes those bind settings
without replacing your URLs, timezone, ports or credentials.
The installer builds only CLI/TUI clients on the host when Go is available;
the server and stage harnesses run in Docker. It configures ntfy accounts/token,
checks publishing/health, and opens container OpenCode provider login on first setup;
existing settings, credentials and volumes are retained on reinstall.
Docker provider credentials live in its persistent container volume, separately
from your host OpenCode login; reinstalling or reconfiguring reuses the existing
container accounts and volumes rather than replacing them. Setup asks for ntfy
account passwords, then your model-provider login and models. It selects separate
saved defaults in order: first the interactive harness model, then the
new-pipeline model, in two fuzzy-searchable pickers (the saved value is
preselected and marked `(current)`; typing filters, arrow keys or Tab navigate,
Enter confirms, and Esc cancels without changing anything). Model selection
performs no inference; only the OpenCode path validates the new-pipeline model
with a small real generation (provider charges may apply). Fresh
`./install --docker --harness pi` skips OpenCode login and that generation check. Pi has no
standalone `auth login`: setup opens a fresh `/login` session (regular TUI mode)
where you choose an authentication method, reuse existing volume credentials by
exiting, or sign in with an API key. Each
setup resolves the current supported OpenCode V2 release from the single
documented complete `git/matching-refs/tags/v2` response (no pagination; never the
moving `latest` tag) and the current Pi release from npm, builds `FROM` the
OpenCode image's immutable `sha256` digest, and verifies the built image reports
exactly those executable versions (and a Node-compatible Pi engine). It stops with
a clear error instead of silently substituting a stale pin when discovery, build
or compatibility fails. Use `./install --opencode` or `./install --pi` to
reconfigure login and models inside the existing container without restarting
services: an image matching the release identity needs no rebuild, while a dirty
source always invokes the Docker build and relies on layer caching. The built
image includes python3 so the installer/verification tests can also run inside
the container.
Configuration and data remain per-user. The native default installer is described
above and in [installation](docs/install.md).
Docker setup starts the server and saves its client endpoint (the container port
is **8080**; only the default host mapping is **8081**). That saved endpoint is
authoritative: prefer it and never replace it with a guessed default. Do **not**
also start `pipelines serve` for this setup. Explicit `--url` and `PIPELINES_URL`
override the saved endpoint; restart existing TUIs after setup.
Docker mounts your home and installation checkout at their original absolute paths
and uses your UID/GID, so the checkout supplies `compose.yaml` while
`--workdir "$PWD"` preserves the actual disposable workspace. Other host
repositories need a bind mount at the same absolute path; repositories on another
client computer are not automatically transferred.
These unauthenticated server ports are for trusted networks only—do not expose
them to the public Internet.
### Developer/legacy local server (unsupported)
For developing Pipelines itself or running a previous Python-supervised legacy
host setup, an explicit foreground server remains available. This is not the
native installer and is not part of Docker setup; fresh databases contain no
pipelines:
```sh
pipelines serve # foreground developer/legacy server
# In another terminal:
pipelines
```
Optional developer/legacy autostart at login (starts the server now):
```sh
pipelines service enable
pipelines service status
```
Uses user-level systemd on Linux or launchd on macOS; no sudo. Runs use the
directory you invoke `pipelines run` from. Preview with `--print`. Stop any existing
foreground/legacy server first. Disable with `pipelines service disable` before
uninstalling. See [installation](docs/install.md) for environment and boot options.
## Your first pipeline and run
Start in the repository you want the workflow to operate on:
```sh
cd /path/to/your/repository
pipelines
```
1. **Create:** press **n** or use **:new**. Choose the configured **OpenCode** or
**Pi** harness to author a workflow together, or **nvim** to edit the Markdown
yourself. Harness authoring runs in the Docker server; only the **nvim** editor
is required on your client computer. Describe the stages, their inputs/outputs,
and the tools they need. Templates (**b** on Home) are optional; a fresh
installation has no published pipelines.
To skip authoring, use **:templates**, choose one of the four generic starters,
and press **s** or Enter on **Create pipeline**. Review the prefilled name
(edit it if needed), then press Enter to create—no separate naming or editor
step. **e Customize** edits it first. Templates include a smaller graph preview;
**Tab** switches focus and space between the description and graph. Use **hjkl**
to pan a focused graph and **n/p** to browse templates.
2. **Keep a draft:** finish/exit the editor. Nothing is published just because you
edited it. Press **d** / **:drafts** to find retained drafts; **e** edits and
**v** validates the selected draft.
3. **Create/update:** press **s** in Drafts. If it has no name, type one into the name
prompt and press Enter. Then review and explicitly confirm submission. The
action is **Create pipeline** for a new definition or **Update pipeline** for
a new version. The server validates it and retains immutable versions; the draft stays.
4. **Run:** select the published pipeline, press Enter to open it, then **:run** to
start a run. Supply your task and review the confirmation. The run uses the
directory from which you launched the client, not the installer checkout.
5. **Inspect:** watch the full graph and stage progress. **Tab** expands/focuses
panes; **n/p** select recorded stage visits; **hjkl** pan a focused graph.
On a failed run, **D** / **:debug** opens a fresh OpenCode debugging conversation;
retry requires confirmation and restarts the pinned workflow, so earlier side
effects may happen again.
Home shows only **/** search and **:** commands as action hints; **/** opens the omni
popup, which finds pipelines, versions, runs, stages/visits, artifact names and the
text of stage inputs, outputs and evidence, and Enter opens the chosen result. Use
the command palette for creation, drafts, templates, history, status, logs, and
quitting.
Quit with **:quit**; **q** does not exit. For a terminal-only run after publication:
```sh
pipelines run "your pipeline name" "Describe the task"
```
### Phone notifications
Open **:status** for the server and ntfy connection details. In the ntfy phone
app, add the displayed ntfy server, sign in as **phone-reader** with the password
you set during installation, and subscribe to the displayed topic. A run that
waits for a decision or fails notifies automatically; to notify on a stage's own
result, add a `notify` block. See [ntfy setup](docs/ntfy.md). Installation checks
publishing, not reception on your phone.
For Tailscale, both devices must be connected; use
`http://<server-Tailscale-IP>:8090`. Binding to `0.0.0.0` does not bypass firewalls:
UFW/Docker forwarding rules can allow LAN sources but drop Tailscale sources.
See [Tailscale troubleshooting](docs/ntfy.md#tailscale-address-times-out-while-lan-works).
## Uninstall
```sh
pipelines uninstall # remove Go binaries; keep history, drafts, and ntfy data
```
## A four-stage pipeline
**Plan → Implement → Review → Commit**. The first stage receives your task prompt.
Later stages receive the previous output or fields selected by an explicit
`memory` block. One outcome selects the next stage—no
parallel branches. Definitions use Markdown `output`, `prompt`, `tools`,
`configs`, and `edges` blocks. For example:
````markdown
# stage planner
```output
{ task, plan, design, reuse_targets, acceptance_checks, research }
```
```prompt
Turn input.prompt into a focused plan. Discover repository conventions and
reuse targets, and research official sources only when external knowledge is
actually required. State why when no research is needed.
```
```tools
read
glob
grep
websearch
webfetch
```
```configs
provider=opencode-go
model=deepseek-v4.1-flash
retries=2
timeout=1h
```
```edges
success -> implementer
blocked -> #failure
error -> #failure
```
````
The [complete three-stage example](examples/three-stage.md) adds the implementer,
reviewer and committer: the planner may search/fetch official sources, the
implementer has no web tools and edits and runs checks from the planner's research,
the reviewer independently inspects the real changes with its own shell access,
receiving the original task and the planner-owned acceptance checks rather than the
implementer's summary, and the committer stages the runner's captured changed paths
and commits locally without ever pushing. The reviewer approves only when the
applicable acceptance checks were independently observed passing and no unresolved
substantive issue remains, and otherwise rejects with concrete findings. Review and
explicitly submit it:
```sh
pipelines validate examples/three-stage.md
pipelines submit examples/three-stage.md --name "three stage" --create-only
pipelines run "three stage"
```
Every submission is validated and creates an immutable version. Examples are
optional; nothing is imported automatically. The four stages give a visible
plan → implement → review → commit path using your own accounts and repository; no
maintainer accounts or published definitions are assumed. Run from your
repository directory; stage tools are enabled automatically. Edits support any
language/file type and new files. No file list or typed RUN confirmation is
required.
## Useful commands
```sh
pipelines # dashboard; b browses templates
pipelines list
pipelines run # fuzzy selection + multiline prompt
pipelines run "bug fix" "Fix the layout" # run in the current repository
pipelines new "daily digest" "Summarize changes"
pipelines edit "bug fix" "Add a review step"
pipelines drafts --json
pipelines validate /path/to/draft.md
pipelines submit /path/to/draft.md # explicit publication
pipelines runs
pipelines inspect 1
pipelines serve # foreground server; Ctrl+C stops it
pipelines ping
pipelines version --json # local client identity (offline)
pipelines ping --json # running server identity
pipelines retention report --keep 200 # show what run-history retention would remove
pipelines retention apply --keep 200 --yes # remove it (artifacts go with their run)
pipelines help edit
```
Run-history retention is explicit and never automatic: nothing prunes runs or
their artifacts until you run `pipelines retention apply`, protected runs (active,
waiting on a person, or referenced by a retained retry) are kept, and pipeline
definitions and versions are never touched.
CLI `new`/`edit` open the configured harness (**OpenCode** or **Pi**, native host
by default; a profile is Docker-linked only when a checkout is explicitly linked)
directly with the Pipelines skill. The installed skill asks
missing questions; `pipelines new` needs no preliminary questionnaire. Changes stay
as drafts by default; CLI and TUI share validation and draft storage. Most
commands support `--json` for agents and scripts.
In the TUI, press **d** (or use **:drafts**) to browse retained drafts, including
CLI-created ones. **e** reopens a draft, **v** validates it, and **s** opens explicit
submission confirmation, asking for a name first if needed. Drafts remain retained
after submission. Drafts are portable: submit to the currently connected server.
When that differs from the origin, submission creates a new pipeline rather than
applying the origin's version number to an existing destination pipeline, and a
notice names both server URLs instead of silently retargeting the draft.
From a run, press **P** / **:pipeline** to open the pipeline version it used.
Initial run input also records `pwd`. Press **r** / **:retry** on a failed run
(or `pipelines retry <run-id>`) to start a new attempt with the same original
input, directory, and pipeline version. Completed side effects are not undone.
Retries restart at the entry stage with fresh OpenCode sessions, never the previous
conversation or intermediate stage output. Pipelines does not automatically delete
generation, agent, or formatting-repair sessions, including on failure/cancellation.
Primary session IDs are retained in run steps (`pipelines inspect <run-id> --json`).
Every run also records a short, single-line title: `pipelines run ... --title "…"`
sets one explicitly, otherwise it is generated from the prompt. `pipelines runs`
and `pipelines inspect` show it beside the pipeline name and version, and
`pipelines title <run-id> "…"` renames it afterwards.
A pipeline may include a `type=human` decision stage: the run pauses as
**waiting** with the stage's instructions and options visible in `inspect` and the
run inspector. Answer it with `pipelines decide <run-id> [stage] --option KEY
[--feedback "…"]` or by pressing **c** in the run inspector to pick an option and
type feedback. The choice and feedback are recorded as the step's evidence and the
run resumes along that edge. The wait is persisted, so it survives a server
restart; an unknown option is refused with the valid list, a stage that is not
waiting is refused, and a repeated decision does not double-apply. **cancel** and
**retry** keep working while waiting and after a decision.
On Home, **r** opens global run history; **/** opens the omni search popup over
pipelines, versions, runs, stages/visits, artifacts and stage input/output/evidence
text. Inside a pipeline, **r** / **:runs** opens only that pipeline's history.
Quit the TUI with **:quit**; **q** does not exit. **Ctrl+C** remains an interrupt.
Starting/retrying/canceling runs and deleting drafts require confirmation. CLI
publication also prompts; use `--yes` only for explicitly approved automation.
**n / :new** and creating from a template offer a Tab-navigable, fuzzy-searchable
picker: **nvim** opens the draft Markdown file; **OpenCode**/**Pi** author it with
you. Canceling the picker creates no draft and publishes nothing.
Wide terminals show runs and pipelines side by side; narrow terminals stack them.
The TUI uses soft light/dark-adaptive cards, muted metadata, subtle selections,
and local loading animations instead of heavy character borders and inverted bars.
The dashboard includes all-history usage, success rate, average completed duration,
per-pipeline usage, and a 14-day activity chart. CLI: `pipelines stats [--json]`.
Use **:status** for two panels: server addresses and ntfy phone connection details.
CLI: `pipelines status [--json]`. Candidate URLs are not
verified reachable; publishing endpoints and credentials are not disclosed.
Retry is offered only after the failed run and its agent have fully stopped.
Graphs use content-sized boxes/columns and tightly spaced branch lanes while
retaining every stage and labeled box-and-arrow route. Incoming route IDs (left)
and outgoing outcomes (right) share rows to reduce box height. Both pipeline and
run views center the graph horizontally and vertically when it fits.
The run inspector shows the entire pinned graph above the details, dimming stages
and routes not executed. **g** focuses the graph; **hjkl** or arrows pan it.
The path and elapsed duration update while running. **a** opens the active/latest
OpenCode session's recorded tool evidence; this does not attach a live OpenCode UI.
Press **D** / **:debug** (or `pipelines debug <run-id>`) for a fresh OpenCode
conversation about that run. It uses CLI queries rather than resuming or retrying
the implementing agent. Pipeline **Edit** offers a fuzzy-searchable **nvim /
OpenCode** picker; Tab and Shift+Tab navigate it.
In the draft browser, **x** then **y** deletes the selected retained draft and
metadata. CLI: `pipelines draft-delete <draft.md>`. Published history is untouched.
Model results use native JSON-schema output where supported, precise parsing
errors, and bounded format-only repair without replaying tools. Retained failures
distinguish final-result validation from harness transport. Optional stage
permissions narrow file/command access without imposing a language-specific
default. See [model result recovery and permissions](docs/model-results.md).
The run graph header shows start time and elapsed/final duration, plus finish
time for completed runs.
More: [CLI](docs/cli.md) · [installation](docs/install.md) · [pipeline format](internal/author/skill/references/format.md).