Files
pipelines/docs/install.md
T
alan 8c1eea53a9 Report a stale client and refuse an unreadable newer server
Exchange a build/protocol identity so an older CLI can no longer talk to a
newer native server silently:

- protocol carries ProtocolVersion and every server frame (including the
  unsolicited welcome) is stamped with the server build and protocol, so a
  client that never pings still learns the server identity.
- buildinfo.Compare classifies the relationship as InSync/Warn/Refuse and
  names both builds; the client Watch reports one warning per connection and
  returns a hard error when the server speaks a newer, uninterpretable
  protocol so the caller never acts on the frame.
- client.ReceiveNotice decodes the raw frame so an unmodeled top-level field
  produces a one-time "unrecognized field(s)" notice instead of being dropped
  silently (the case that hid the artifacts field).
- CLI routes warnings to stderr, keeps machine JSON stdout clean and fails
  non-zero on a refused mismatch; ping/connection print the client/server
  relationship. The TUI surfaces the same notices through the shared log.
- scripts/setup-native.py now warns when an older `pipelines` earlier on PATH
  shadows prefix/bin/pipelines, naming both paths and versions, without
  installing or modifying anything else.
- Docs, the packaged debugging skill reference and README no longer imply the
  installed CLI always matches the server.

Tests cover the comparison table, unknown-field detection and caching, the
server identity stamping on pong and welcome, the CLI/TUI reporting paths and
the installer PATH shadowing warning.
2026-10-16 18:37:01 +01:00

437 lines
26 KiB
Markdown

# Installation
`./install` installs **natively on the host by default** (macOS and Linux);
`./install --docker` runs the existing Docker server + ntfy installation.
To reconfigure login and separate interactive/new-pipeline models inside an
existing Docker installation without a service reinstall: `./install --opencode`
or `./install --pi`.
See [configured harnesses](harnesses-and-control.md) for scope and verification limits.
```sh
git clone https://git.a-wan.net/alan/pipelines.git
cd pipelines
./install # native: build server, CLI and ntfy on this host
./install --docker # explicit Docker server + ntfy
```
## Native installation (default)
`./install`, `./install --native` and `./install native` run
`scripts/setup-native.py`. It only detects the tools it needs and never installs
or upgrades them: **Go, python3, git, curl and a C compiler (`cc`/`clang`/`gcc`)**
must already be on PATH. The C compiler is required because ntfy stores its cache
and auth database in SQLite through the cgo driver `mattn/go-sqlite3`. It also
detects an existing `opencode` (required) and `pi` (optional) on PATH and records
their absolute paths; it never downloads, installs or upgrades OpenCode or Pi
(harness installation stays Docker-only).
Setup resolves `PIPELINES_PREFIX` or defaults to `~/.local/share/pipelines`,
creates `bin/`, `lib/`, `data/`, `logs/` and `secrets/` (`secrets` and `data` at
0700), copies `scripts/manage.py` to `<prefix>/lib/manage.py`, builds exactly
`./cmd/pipelines-server` and `./cmd/pipelines` with `go build -trimpath` into
`<prefix>/bin`, and runs `<prefix>/bin/pipelines skill install` so the packaged
authoring skill lands in your OpenCode skills directory. No `pipelines-tui` is
built. It writes `<prefix>/pipelines-install.json` atomically at 0600; that
manifest is what `scripts/manage.py` and the Go CLI load. Provider/model defaults
are never invented: your existing host OpenCode configuration is reused (an
explicit `--model` is recorded if supplied), and setup never starts a picker.
After the build, setup checks whether a **different** `pipelines` resolves earlier
on PATH than `<prefix>/bin/pipelines` (the common cause of a stale CLI talking to a
newer server). When it does, setup prints a `WARNING` naming that absolute path and
both versions (`version --json` on each binary, read-only) and tells you to put
`<prefix>/bin` first on PATH. It never modifies, moves, deletes or upgrades that
other binary, and it installs no unrelated tool; the manifest keeps
`<prefix>/bin` first on the recorded PATH.
The native install also writes **host** harness profiles for the harnesses it
detects into XDG config home's `pipelines/harnesses.json` (no
`docker_directory`), defaulting to the detected OpenCode profile rather than a
stale `pi` default and taking models from `--model` or the existing valid profile.
When migrating a Docker install it clears the recorded Docker link instead of
leaving the config Docker-linked. It never installs, upgrades or downloads a
harness. If no valid model can be resolved it writes nothing and points you at
`pipelines harness setup <opencode|pi> --host`.
ntfy is built from source with no Homebrew or package manager. Setup resolves the
exact stable `binwiederhier/ntfy` tag (never the moving `latest` tag and never an
older fallback), then compiles the server from the pinned `heckel.io/ntfy/v2` Go
module at that tag into `<prefix>/bin/ntfy`. A source build is required because
upstream publishes **no macOS server binary** (its `darwin_all` asset is built
with `-tags noserver` and only provides the client commands), so both macOS and
Linux take the same path; the Go toolchain verifies the module against its public
checksum database. The build uses
`-tags sqlite_omit_load_extension,osusergo,netgo` and creates the same empty web
placeholders the upstream Makefile uses for local builds, so the bundled ntfy web
UI is not populated (notifications and the API are unaffected). A binary for the
same resolved version is reused instead of rebuilt. Setup generates
`<prefix>/data/ntfy/server.yml` mirroring `compose.yaml` (`listen-http`,
`base-url`, `cache-file`, `auth-file`, `auth-default-access: deny-all`,
`enable-login`, attachment cache directory, expiry and size limits), creates the
`pipeline-publisher` (write-only on `pipelines`) and `phone-reader` (read-only)
accounts with ntfy's own CLI (`--ignore-exists`, so existing passwords are kept;
the account subcommands read the generated config through `NTFY_CONFIG_FILE`),
and writes the publisher token only to `<prefix>/secrets/publisher-token` at 0600.
When Docker volumes `pipelines_pipelines-data` or `pipelines_ntfy-data` exist,
setup migrates the pipeline database to `<prefix>/data/pipelines.db` and the ntfy
auth store to `<prefix>/data/ntfy/auth.db` using the local Pipelines image
(never an implicit pull). Existing native data is never overwritten unless you
pass `--replace-data`; a requested migration that fails aborts the install
instead of starting empty. Absent volumes are a no-op. If the Docker CLI is
present but its volumes cannot be inspected (for example a stopped daemon), setup
fails clearly instead of silently skipping migration, so an existing Docker
database is never shadowed by an empty native start.
Start and manage it with the manifest's prefix on PATH:
```sh
PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start
PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines status
```
`start`, `stop`, `restart`, `doctor`, `logs` and `uninstall` reach
`<prefix>/lib/manage.py`. Ports default to 8080 (server) and 8090 (ntfy) and can
be changed with `--port`/`--ntfy-port`; setup starts both services itself, so
those ports must be free (a running Docker installation holds them — stop it
first; the migration still needs its volumes and image). ntfy runs from
`<prefix>/bin/ntfy` with the generated config; per-user launchd (macOS) or
`systemd --user` (Linux) registration and the supervised fallback both remain.
Dated evidence (2026-10-13, macOS arm64): a scratch-prefix run of
`scripts/setup-native.py` resolved ntfy v2.30.0, compiled it from source (`ntfy
--version` = v2.30.0 with the `Server commands` present), built the server and
CLI, installed the skill, migrated 8 pipeline definitions and 28 runs plus both
ntfy accounts from the live Docker volumes, and wrote the manifest.
`pipelines start` then brought up ntfy and the server through the supervisor
(`{"healthy":true}` from both `/v1/health` and `/healthz` on spare ports) and
`pipelines stop` released both ports. launchd/systemd registration and real phone
delivery are **not tested**.
## Docker installation (explicit)
`./install --docker` (or `./install docker`) runs the unchanged
`scripts/setup-docker.py`. It requires Docker Compose and Python 3 and includes
OpenCode/Pi in the persistent server container. Host Go builds the CLI/TUI
clients, not a native server.
The installer builds host `pipelines` and `pipelines-tui` clients when Go is
available, using `GOBIN` or `GOPATH/bin`; put that directory on PATH. It does not
install a host server. Reconfiguration uses `./install --opencode` or `./install --pi`
inside the existing Docker installation and never falls back to host execution. It
resolves and verifies the same current releases and refuses to configure against an
image that does not match them (rebuild with `./install` first); it does not restart
or redeploy services.
Native Go builds embed VCS identity; Docker setup passes the checkout revision and
dirty state as compile-time build arguments (without copying `.git` into the image).
For a build-only refresh use `python3 scripts/build_metadata.py`; it resolves the
current OpenCode V2/Pi releases and runs `docker compose build` with the identity
arguments plus the resolved version/digest arguments, without restarting services.
A plain `docker compose build` cannot resolve the OpenCode image digest, so it must
be given `OPENCODE_VERSION`, `OPENCODE_DIGEST` and `PI_VERSION` through Docker
`--build-arg` flags (and `PIPELINES_VERSION`, `PIPELINES_COMMIT` and
`PIPELINES_DIRTY` outside a Git checkout); `./install` supplies all of them. After deployment,
`pipelines --url ws://HOST:PORT/connect ping --json` identifies the running server.
Docker setup checks that the started server's pong matches the source identity
passed to the build and fails setup if metadata is missing or mismatched.
Each setup resolves the current releases before touching settings or services: the
newest supported **OpenCode V2** release from the single documented complete
`GET .../git/matching-refs/tags/v2` response (a bounded complete array with no
pagination parameters; never the moving `latest` tag or the registry's insertion
order) and the current Pi release from npm. The image is built `FROM` the OpenCode
image's immutable `sha256` digest, resolved from GHCR, and labeled with the
resolved OpenCode/Pi versions, digest and source identity; the moving `latest` tag
and version-only references are never used. Setup verifies the built image reports
exactly those executable versions and a Node-compatible Pi engine, and stops with
a clear error (keeping the last verified Dockerfile fallback pins) when discovery,
resolution, build or compatibility fails rather than guessing or silently
installing stale or unchecked copies. A full install skips a rebuild only when the
release receipt **and** the actual built image's release and source identity labels
already match; a dirty source always invokes the Docker build and relies on layer
caching. A model/login-only reconfiguration compares only the verified release
image identity, so a changed source alone does not rebuild on reconfigure.
Setup writes that release receipt to the machine-local, generated root file
`.pipelines-releases.json` (with atomic `.pipelines-releases-*` temporary files)
and both are git-ignored, so a fresh clean checkout stays clean after setup and
reinstall still reuses the verified release/image identity. The actual built
image's labels stay authoritative, clean-checkout identities/reinstall reuse are
unchanged, and a dirty source still rebuilds.
Docker mode detects the computer's LAN IPv4 address and timezone automatically,
configures `http://LAN_IP:8090`, and binds both Docker services to `0.0.0.0` without asking
for networking settings (timezone falls back to UTC). On reinstall it updates only
the bind settings to `0.0.0.0`, refreshes the detected host LAN IP for Status,
and retains existing URLs/timezone/ports/credentials. It bootstraps
the publishing/phone accounts and publishing token, builds the server, installs
the skill inside the container, and opens **container OpenCode provider login**
on first setup. Choose any supported provider; signup is handled by that provider.
Provider credentials remain in the `opencode-home` volume. Reinstalls skip login
after completed setup; `./install --relogin` opens it again without
deleting accounts. Setup then chooses **separate saved defaults** in order: first the
interactive OpenCode harness model, then the new-pipeline model, in two
fuzzy-searchable pickers: type to filter, arrow keys or Tab navigate, Enter confirms,
the saved value is preselected and marked `(current)`, and Esc cancels with nothing
changed. `--model PROVIDER/MODEL` and `--harness-model
PROVIDER/MODEL` set them explicitly; reinstall retests the saved selection. Container
credentials and accounts stay separate from host credentials and are reused on
reinstall/reconfigure. Picker
selection performs no inference; the OpenCode path then requires a real, small
**tool-free model response** for the new-pipeline model before marking setup complete
(provider charges may apply). Login, discovery and validation prime
OpenCode's integration catalog in their own container/service. Discovery retries empty model
reads in the same container, avoiding a cold-start empty-catalog response.
An empty catalog is not proof of an invalid key; saved credentials are retained.
New drafts/templates inherit the new-pipeline selection;
published definitions and retry versions stay unchanged. Failed checks print
bounded, credential-redacted evidence and retain their OpenCode session.
The installer then starts/recreates the Compose services,
checks ntfy health through Pipelines, and sends an authenticated test notification.
It does not register an OS service, delete volumes, or seed pipeline definitions.
Phone reception is not verified by installation.
Docker setup mounts the host user's home and the installation checkout at their
original absolute paths, and runs Pipelines/OpenCode using that host user's UID
and GID. This makes a client invocation directory such as
`/home/tank/repos/pipelines` refer to the actual host checkout rather than an
unavailable container path or the unrelated image checkout at `/workspace`.
Stage permissions/edit scopes still apply; this is a filesystem mount, not a
workspace allowlist or automatic file transfer from another client computer.
For repositories outside those mounts, add a Compose bind at the same absolute
path. Reinstallation stops the old Pipelines service before changing ownership
of its managed data/auth/workspace volumes. It never recursively chowns the host
home/repository, deletes credentials, or removes volumes. Existing data/auth
volumes use `nocopy` so Docker cannot reset a freshly prepared empty volume's
ownership to the image UID. Setup verifies that the server can enter the host
checkout before reporting completion.
For fresh configuration you can supply `--ntfy-url https://ntfy.example.com
--bind-address 127.0.0.1 --timezone Europe/London --port 8081` directly to `./install`.
The defaults are ready for direct phone access on your trusted LAN. Only use
plain HTTP on a trusted network. Other existing `.env` settings are not replaced.
These Docker flags apply to `./install --docker`; the explicit Docker mode fails
rather than silently falling back to native binaries if the LAN address cannot be
detected. Use an interactive terminal for first Docker/provider setup.
The Docker Pipelines API is unauthenticated: keep its port on a trusted network
and do not forward it to the Internet. For a manual loopback-only deployment,
set `PIPELINES_BIND_ADDRESS=127.0.0.1` after installation and use Compose directly.
Ctrl+C cancels cleanly without deleting configuration, credentials or volumes.
If Go is available, Docker setup also updates standard host CLI/TUI binaries and
the skill. Restart an already-running TUI to load the changes. Without Go, the
installer explicitly warns that the host client was not updated; Docker still runs
the server, and you can build the client later with host Go. See [Docker setup](docker.md).
After ntfy verification, setup also checks the published Pipelines endpoint from
the host (when Go or a host CLI is available) and saves it in
`$XDG_CONFIG_HOME/pipelines/client.json`, or `~/.config/pipelines/client.json`.
Only `server_url` is updated; unrelated settings are preserved. CLI/TUI precedence
is `--url`, then `PIPELINES_URL`, then the saved default, then localhost:8080.
Docker's default host mapping is port 8081 (the container listens on 8080), so an
old native server on 8080 must not accidentally become the client target. Setup warns if an environment override still selects
another address; it does not stop or delete that other server.
If Status says ntfy is **not configured**, check which Pipelines server the client
is reaching before reinstalling. On the Docker server computer:
```sh
docker compose port pipelines 8080
pipelines --url ws://127.0.0.1:8081/connect status --json
pipelines --url ws://127.0.0.1:8081/connect tui
```
Use the actual mapped port instead of 8081 if customized. On a different computer,
use the Docker host's LAN address instead of 127.0.0.1. The saved default applies
only on the computer/user that ran the installer. Phone reception remains untested
by these checks.
## Failure evidence and confirmations
TUI run/retry/cancel requests cannot be sent before their confirmation modal.
Draft creation/editing and opening a debugging session also require approval.
CLI mutations prompt with a default of No; `--yes` is explicit noninteractive
approval. The standalone TUI always keeps its modals, including with `-yes`.
`pipelines debug RUN_ID --no-open` retrieves the retained run plus read-only
OpenCode session evidence from the **server**, not the client's unrelated service.
Debug includes the exact pinned pipeline definition and full run snapshot directly.
Large context packets are retained in a private client cache file which OpenCode
is instructed to read, rather than discarding evidence or exceeding argument limits.
New failed operations retain operation, provider/model, executable, directory,
timing, exit/HTTP status and bounded stdout/stderr under `opencode_failures` in
the step transcript. Credentials are redacted, but project/provider messages may
still contain private context: review before sharing. Older failures cannot gain
discarded subprocess output retroactively; Debug attempts retained session reads.
Native OpenCode schema support is checked with a retained, deny-all, no-inference
POST/readback probe. OpenCode 1.18.35 accepts explicit formats but rejects their
stored messages on readback; affected backends omit `format` and require the same
strict local `{outcome, output}` validation. Compatible backends retain schema
mode with no automatic tool-replaying schema retries. The compatibility decision
and probe session are recorded, without rewriting published definitions/history.
For old format-bearing sessions whose HTTP readback fails, Debug uses read-only
`opencode export` with an exact session-ID check to recover original messages.
`StructuredOutputError` is surfaced directly; DSML/tool-call markup and flattened
results are rejected and retained, never silently interpreted as successful tool
execution or wrapped into a guessed result. Native HTTP evidence distinguishes
adapter/HTTP failures from actual subprocess exit codes.
To validate a model on the execution computer, run:
```sh
pipelines provider-check --model PROVIDER/MODEL
# Docker credentials are separate from host credentials (plain CLI: no TTY):
docker compose exec -T pipelines pipelines provider-check --model PROVIDER/MODEL
```
For a developer or legacy host build, a private repository uses your normal Git
credentials and `GOPRIVATE` to compile the client/host binary (this is not a
supported server installation):
```sh
GOPRIVATE=git.a-wan.net go install git.a-wan.net/alan/pipelines/cmd/...@latest
pipelines skill install
```
For a system-wide command, build as your normal user and install the three binaries
into an administrator-selected bin directory using your normal system permissions.
Services and OpenCode accounts still run as the logged-in user, never as root.
The installer does not hardcode a system destination or run sudo.
## Developer/legacy host server (unsupported)
This explicit foreground server is for developing Pipelines itself or running a
previous Python-supervised legacy host setup. It is **not** the native installer;
a fresh database starts empty.
```sh
pipelines serve # foreground developer/legacy server; Ctrl+C stops it
# In another terminal:
pipelines
```
The server binds to `0.0.0.0` by default (the LAN, so the phone client can reach
it), matching the Docker install; record `127.0.0.1` on the manifest with
`./install --bind-address 127.0.0.1` for a loopback-only server. The Pipelines API
has no authentication, so keep that port on a trusted LAN/VPN. Change into your
repository and run a pipeline there; the client sends its current directory to the
server. The server's own working directory does not restrict which repositories
you can use. Model tool permissions and approved edit paths remain separate,
per-run/stage controls.
The SQLite database uses the standard user data location, honoring `XDG_DATA_HOME`
when set. On macOS it uses the user Application Support directory. `-db FILE`
selects another database, including an existing one; nothing is imported or moved
automatically. `pipelines serve --help` prints the actual defaults on your machine.
Drafts use the OS user cache; `PIPELINES_DRAFT_DIR` overrides that location.
## Developer/legacy autostart (unsupported)
Stop any existing foreground or legacy server first, then explicitly configure
the OS user service:
```sh
pipelines service enable
pipelines service status
```
This starts the server now and at subsequent logins. Linux uses
`systemctl --user`; macOS uses a LaunchAgent in your user `Library/LaunchAgents`.
There is no sudo, Docker dependency, or second supervisor. `--db`
and `--addr` select the database/listen address, just like foreground startup.
Re-run enable to update the definition and restart with the new settings.
```sh
pipelines service enable --print # preview only
pipelines service disable # stop and remove
```
Setup records the current PATH, HOME, XDG directory overrides, selected OpenCode
executable, and supported ntfy URL/token-file settings in a private service file.
It does not copy provider credentials or the rest of the shell environment.
Use `PIPELINES_NTFY_TOKEN_FILE`, not an inline token. Re-run enable after changing
these settings or moving binaries. Relative database/token-file paths are resolved
at configuration time. Services do not read your shell startup files.
On Linux this is **login autostart**, not automatically boot-without-login.
If needed, explicitly enable lingering using your system's normal administration
policy (`loginctl enable-linger "$USER"`). Pipelines does not change that policy.
macOS LaunchAgents likewise require a user login. Windows autostart is unsupported.
Use `journalctl --user -u pipelines.service` for Linux logs. On macOS inspect
`pipelines service status`; run `pipelines serve` in the foreground for diagnostics.
## OpenCode runtime (developer/legacy host)
For a developer or legacy host server, use your existing `opencode` on PATH, or set
`PIPELINES_OPENCODE` to an explicit executable. No OpenCode version is required just
to install binaries. At runtime:
- The standard/native API release uses an owned, authenticated loopback
`opencode serve` process, scoped to the requested repository and using the
existing user's configuration/login. The API, not a version number, is checked.
- A CLI advertising `api` uses the V2 command interface and its normal discovery.
- Both adapters select the pipeline's configured model explicitly and preserve
stage tool permissions. Native `shell` permissions map to OpenCode's `bash`.
- Audited native sessions record their backend and a private service reference
so recovery addresses the same service, rather than a new unrelated instance.
- `new`/`edit` open the normal interface directly with `@pipelines` and a draft
context-file reference. The skill handles questions and draft validation inside
OpenCode; no preliminary session/API calls or instruction preload are performed.
The global skill is installed in `$XDG_CONFIG_HOME/opencode/skills/pipelines`,
or the normal `~/.config/opencode/skills/pipelines` default. Only unchanged files
previously installed by Pipelines are updated. A user-managed/modified skill is
never overwritten; preserve or rename it before running `pipelines skill install`.
Installation does not require OpenCode itself. Uninstall retains the skill and drafts.
The packaged format, notification, operations, transfer and verification references
are self-contained; agents read only those relevant to the task. Authoring
does not require Pipelines source in the current repository. Without a task prompt,
OpenCode is instructed to ask what to create/change before investigating files or
running CLI commands. See [authoring session review](authoring-session-review.md)
for the reasoning and verification limits.
New features and confirmed limitations are maintained in the packaged skill alongside
the topic docs. Skill tests check that every embedded reference is installed and
its links resolve. Refresh the global managed skill from current source with
`go run ./cmd/pipelines skill install`; this does not deploy server/runtime fixes.
The bundled examples retain `opencode-go/deepseek-v4.1-flash`. Availability and
authentication for that model are runtime requirements, not installation checks.
No login or provider credentials are copied or exported by Pipelines.
## Optional notifications
Installation configures this automatically via `./install`.
For a developer/legacy host server, use an existing ntfy server or the Docker setup in [docs/ntfy.md](ntfy.md).
Configure `PIPELINES_NTFY_URL`, `PIPELINES_NTFY_PUBLIC_URL`, and a publishing token
via `PIPELINES_NTFY_TOKEN_FILE` in the server's environment. Keep secrets out of
pipeline definitions. Use HTTPS or an encrypted VPN for phone access; do not
expose an unauthenticated pipeline server or HTTP credentials to untrusted networks.
## Uninstall
Disable autostart with `pipelines service disable` if configured, or stop the
foreground server first, then:
```sh
pipelines uninstall
# or pipelines uninstall --yes
```
Only Go binaries whose embedded build metadata matches this repository are
removed from the current command's bin directory. History, drafts, OpenCode,
and optional ntfy data remain untouched. Use the permissions appropriate to your
chosen bin directory. Custom database paths cannot be inferred, so uninstallation
does not purge data.
## Existing managed installations
The previous Python-supervised setup is retained only for explicit legacy use.
Set `PIPELINES_PREFIX` to its existing directory for its lifecycle commands;
no command infers that directory from an executable location. The old
`--system-command`, provisioning, and autostart installer flags have been removed.
To migrate, stop the old server first, install normal Go binaries, and start with
`pipelines serve -db EXISTING_DATABASE`. Keep the old
data and credentials until you have verified the new setup. There is no automatic
purge, database move, login migration, or shared OpenCode service modification.