Files
pipelines/docs/install.md
T

327 lines
16 KiB
Markdown

# Installation
`./install` installs Pipelines natively for your user on macOS or Linux. It
builds the server, the CLI and the web app together, installs them under
`~/.local/share/pipelines`, and starts them under a login service. No sudo.
```sh
git clone https://git.a-wan.net/alan/pipelines.git
cd pipelines
./install
```
## Prerequisites
The installer only detects tools. It never installs, upgrades or downloads
them, and it stops naming whatever is missing.
| Tool | Why |
| --- | --- |
| Go (version in `go.mod`, currently 1.27) | builds `pipelines-server`, `pipelines` and ntfy |
| Node.js 18+ | runs the authenticated web proxy (`web/serve.mjs`) |
| Python 3.9+ | installer and the lifecycle manager (`manage.py`) |
| git, curl | source checkout, updates, release lookup |
| C compiler (`cc`, `clang` or `gcc`) | ntfy's SQLite driver is cgo (Xcode Command Line Tools on macOS) |
| **Docker, running** | builds the Flutter web app and the APK |
| OpenCode on `PATH` | required; default stage and chat harness (`curl -fsSL https://opencode.ai/install \| bash`) |
| Pi, Claude Code, Codex | optional; set up after install with `pipelines harness setup <name>` |
| tailscale | optional; tailnet HTTPS access |
The install is **all or nothing**: the web app and the server are built before
either is installed. If Docker is not running, `./install` stops with
"Docker is not running, so the web app cannot be built" and leaves any existing
installation running unchanged.
Run as your normal logged-in user. macOS needs `launchctl` in a GUI login;
Linux needs `systemctl --user`. Other platforms are refused.
## What `./install` does
1. Checks tools, port settings, the login service manager and the OpenCode
executable.
2. Builds the Flutter web app in Docker (pinned to `app/.flutter-version`).
Builds are cached in `~/.cache/pipelines/app-builds` by a digest of the app
sources, so an unchanged app is reused.
3. Builds `pipelines-server`, `pipelines`, `pipelines-claude-host` and
`pipelines-codex-host` (retain Claude processes and Codex interaction callbacks
across server restarts) with `go build -trimpath`.
4. Installs into the prefix: binaries in `bin/`, web app in `lib/app-web`, web
proxy in `lib/web`, `lib/manage.py`, and an empty `lib/app-updates` for APKs.
5. Runs `pipelines skill install` (the authoring skill for OpenCode and Claude
Code skill roots).
6. Writes host harness profiles to `~/.config/pipelines/harnesses.json`. If no
usable OpenCode default model is found, it opens the model picker in the
terminal (or fails without a terminal; pass `--model`).
7. Builds ntfy from its pinned Go module, writes its config and creates the
`pipeline-publisher` and `phone-reader` accounts.
8. Migrates data from an earlier Docker installation, if its volumes exist.
9. Registers the login service, writes the manifest
`<prefix>/pipelines-install.json`, saves `ws://127.0.0.1:<port>/connect` as
the CLI default in `~/.config/pipelines/client.json`, and starts everything.
10. Retires the legacy standalone web agent `net.awan.pipelines.web` on macOS
(stopped, plist renamed to `.disabled`), so only one web app is served.
11. If `PIPELINES_WEB_PUBLIC_URL` is an `https://…ts.net` address, runs
`tailscale serve --bg --https=<port> http://127.0.0.1:<web-port>`.
12. Prints the browser URLs and the next step, setting up harnesses.
## Set up harnesses
One command per harness, the same on native and Docker installations:
```sh
pipelines harness setup opencode # stages + chat (required; ./install already configured it)
pipelines harness setup pi # stages
pipelines harness setup claude # chat
pipelines harness setup codex # chat
pipelines harness list # status of all four
```
Each run checks the CLI is installed where the server finds it (printing the
install command if not), signs in with the harness's own login when needed,
picks OpenCode/Pi default models, and verifies with the server's check. No
restart is needed. See [Harnesses](harnesses.md#set-up-harnesses).
It also warns when a different `pipelines` earlier on `PATH` shadows the new
one; it does not touch that binary.
### Addresses
| Service | Default | Flag |
| --- | --- | --- |
| Web proxy (sign-in, web app, `/updates/`) | `0.0.0.0:8082` | `--web-port`, `--web-host` |
| Server API (no auth) | `127.0.0.1:8080` | `--port`, `--bind-address` |
| ntfy | `:8090` | `--ntfy-port`, `--ntfy-url` |
Expose only the web proxy. A reinstall keeps the saved ports and bind
addresses unless you pass the flag again, so an installation set up with
`--web-port 8083` stays on 8083.
The server advertises itself on the LAN with mDNS/DNS-SD as `_pipelines._tcp`,
so the phone app's "Find servers on this network" lists it. The advertised port
is the web proxy's (when it listens beyond loopback; the supervisor passes
`-mdns-port`), or the server's own `-addr` port when that is a LAN address.
TXT records: `v` (version), `name` (host), `path=/connect`, `tls=0`,
`auth=password|none`. Set `PIPELINES_MDNS=off` (or `-mdns=false`) to disable
it; a failure to advertise is logged and never stops the server.
`PIPELINES_WEB_PUBLIC_URL` is the address phones and notification links use.
Set it once; reinstalls keep it:
```sh
PIPELINES_WEB_PUBLIC_URL=https://mymac.tailnet-name.ts.net ./install
```
Without it the installer records `http://<LAN IP>:<web-port>`. Use HTTPS or a
VPN for remote access; plain HTTP does not protect the password.
## Flags
From `arguments()` in `scripts/setup-native.py`:
| Flag | Effect |
| --- | --- |
| `--prefix PATH` | install location (default `~/.local/share/pipelines`, or `PIPELINES_PREFIX`) |
| `--port N` | server API port (default 8080) |
| `--bind-address IP` | server API bind (default `127.0.0.1`) |
| `--web-port N` | web proxy port (default 8082) |
| `--web-host IP` | web proxy bind (default `0.0.0.0`) |
| `--ntfy-port N` | ntfy port (default 8090) |
| `--ntfy-url URL` | ntfy address given to phones (default `http://<LAN IP>:<ntfy-port>`) |
| `--timezone TZ` | override the detected timezone (fallback UTC) |
| `--model PROVIDER/MODEL` | default model for new starters and missing OpenCode profile fields |
| `--allow-authoring` | allow creating and publishing pipelines (fresh installs default to this) |
| `--no-authoring` | execution-only server |
| `--fresh` | start empty even if earlier Docker volumes exist |
| `--replace-data` | let Docker migration overwrite existing native data |
Reinstalls keep the recorded authoring role unless you pass one of the two
role flags.
## Other install modes
```sh
./install --web # rebuild and install only the web app (Docker), then reload the page
./install --apk # build the Android APK (Docker, slower) and offer it to phones
./install --docker # server and ntfy in Docker Compose instead of native
```
`./install --apk` builds a release APK for this installation's web address,
copies it to `<prefix>/lib/app-updates/pipelines.apk` with a `version.json`,
and serves it at `<web>/updates/` behind the sign-in. On the phone:
**Settings → App updates → Check for updates**, then **Download and install**.
The first install can use `<web>/updates/pipelines.apk` directly. Every APK is
signed with the key in `~/.config/pipelines/android/` (seeded from
`~/.android/debug.keystore` if present), so each build installs over the last.
Options: `--server-url ws(s)://host:port/connect`, `--notes TEXT`, `--prefix`.
`./install --docker` is the optional Compose installation with its own
container harness credentials; see [Docker](docker.md). Do not run it
alongside a native install on the same ports. Its harnesses are set up with
the same `pipelines harness setup opencode|pi` (run from the checkout), which
configures them inside the container. The old `./install --opencode` and
`./install --pi` only print that command and exit.
## Lifecycle commands
Put the installed CLI on `PATH`:
```sh
export PATH="$HOME/.local/share/pipelines/bin:$PATH"
```
The installed `pipelines` finds its own manifest. `PIPELINES_PREFIX` selects
another installation.
| Command | Effect |
| --- | --- |
| `pipelines start` | start server, web proxy and ntfy; print the startup report |
| `pipelines stop` | stop all three; harness sessions keep running and the next start reattaches them (see below) |
| `pipelines restart` | stop then start, keeping data; chat turns and stage sessions continue |
| `pipelines doctor` | startup report, health of server and ntfy, OpenCode API check (no inference) |
| `pipelines status` | server, host and ntfy publishing health, from the running server |
| `pipelines logs` | follow `logs/server.log` |
| `pipelines update` | pull and reinstall the same way, then restart (see below) |
| `pipelines uninstall [--purge] [--yes]` | remove the service and binaries; `--purge` also deletes data |
| `pipelines service enable [--print]` | enable and start login autostart (`--print` previews the definition) |
| `pipelines service disable` | disable autostart and stop the supervisor |
| `pipelines service status` | autostart and supervisor state |
### Restarts keep sessions running
A server restart, `pipelines stop` and `pipelines update` stop the Pipelines
processes, not the work running on harnesses. On SIGTERM the server leaves
every OpenCode and Codex session running, records what it needs to reattach,
and exits. It does not interrupt or abort anything. On the next start:
- **Runs**: a stage whose OpenCode V2 session is still alive is re-adopted. The
runner waits for that same session (no new prompt), or reads the result it
reached while the server was down, and then continues the graph. A stage
that had not yet handed work to a harness starts again. A run between stages
continues with its next stage. The stage timeout restarts when the stage is
reattached. Its file-change list is marked partial, because the snapshot
taken before the restart is gone.
- **Chats**: turns, pending questions/approvals, background tasks and queued
prompts are picked up again, and a reply that arrived during the downtime
still notifies. See [Chat backends](chat-backends.md#restarts-keep-sessions-running).
If the harness is gone, the old behaviour applies and the run is interrupted
with the reason in its diagnostic, for example "Server restarted; the OpenCode
service running stage … was gone". This happens when the OpenCode service
process died, or the session belongs to a Pi or OpenCode V1 process that ended
with the server.
Only explicit operator actions stop sessions:
- `pipelines cancel <run>` (or Cancel in the app) interrupts a run's stage.
- Closing a chat, or Stop on its turn, interrupts that chat.
To stop everything: cancel the active runs and close or stop the busy chats,
then run `pipelines stop`. The owned OpenCode stage service stops itself at
shutdown when idle. The owned chat service, used only without an attached
endpoint, keeps running and is reused. Stop it by killing the `pid` in
`~/Library/Caches/pipelines/private/opencode-chat-service.json`
(`~/.cache/...` on Linux). To make every shutdown interrupt runs and close chat
clients as before, add `"PIPELINES_SHUTDOWN_SESSIONS": "stop"` to
`opencode_env` in `<prefix>/pipelines-install.json` and restart.
The OpenCode services the server starts (stage service, owned chat service)
run in their own process session, so macOS launchd leaves them running. The
systemd unit uses `KillMode=process`, so Linux stops only the supervisor, which
stops its own children; the detached services survive. An attached OpenCode
endpoint and the Codex App Server are separate processes anyway.
The login service is `~/Library/LaunchAgents/net.a-wan.pipelines.plist` on
macOS and `~/.config/systemd/user/net.a-wan.pipelines.service` on Linux. It
runs `manage.py supervise`, which owns all three processes. This is login
autostart, not boot-without-login.
## Where things live
| Path | Contents |
| --- | --- |
| `<prefix>/pipelines-install.json` | manifest (ports, paths, install args, environment) |
| `<prefix>/bin/` | `pipelines`, `pipelines-server`, `pipelines-claude-host`, `pipelines-codex-host`, `ntfy` |
| `<prefix>/data/pipelines.db` | SQLite database (definitions, runs, chats, settings) |
| `<prefix>/data/artifacts/` | run artifacts and session exports |
| `<prefix>/data/ntfy/` | ntfy config, cache and accounts |
| `<prefix>/secrets/publisher-token` | ntfy publishing token |
| `<prefix>/logs/` | `server.log`, `web.log`, `ntfy.log`, `supervisor.log` |
| `~/.config/pipelines/web-auth.json` | web password and cookie secret (`PIPELINES_WEB_AUTH_FILE`) |
| `~/.config/pipelines/harnesses.json` | harness profiles (`PIPELINES_HARNESS_CONFIG`) |
| `~/.config/pipelines/client.json` | default server URL for the CLI/TUI |
| `~/.config/pipelines/claude-token` | optional Claude Code service token |
| `~/.config/pipelines/android/` | APK signing key |
| `~/.cache/pipelines/app-builds/` | cached web and APK builds |
`<prefix>` is `~/.local/share/pipelines` unless changed. On Linux the
supervisor's output also reaches `journalctl --user -u net.a-wan.pipelines.service`.
## Web sign-in password
On a new computer `./install` asks for the password (8+ characters, typed
twice). Without a terminal, set `PIPELINES_WEB_PASSWORD`. It is stored in
`~/.config/pipelines/web-auth.json` (mode 0600) with a generated cookie secret,
and reused unchanged on every reinstall. An invalid or non-private file stops
the installer rather than being reset.
The same password is given to the ntfy `phone-reader` account when that account
is first created; existing ntfy accounts keep their passwords.
To change the web password, edit `password` in `web-auth.json` (keep it 0600)
and run `pipelines restart`.
## Upgrading
```sh
pipelines update
```
`update` refuses while runs are executing (finish or cancel them first). It
then runs `git pull --ff-only` in the checkout it was installed from (skipped
when the branch has no upstream), re-runs `./install` with the recorded
arguments, restarts, and prints the startup report. Data, passwords, ntfy
accounts and the signing key are kept. If the installer fails, the previous
version keeps running.
Running `./install` again from the checkout does the same without the pull.
Run `./install --apk` again to offer a new APK to phones.
## Uninstall
```sh
pipelines uninstall # keeps data and credentials in <prefix>
pipelines uninstall --purge # also deletes <prefix>, including the database
```
Neither touches OpenCode, Claude Code, Codex, Pi or files in
`~/.config/pipelines`.
## Developer server
For working on Pipelines itself, run a server in the foreground on spare
ports. This is not the supported installation and does not serve the web app:
```sh
go run ./cmd/pipelines-server -db /tmp/dev.db -addr 127.0.0.1:8081
go run ./cmd/pipelines --url ws://127.0.0.1:8081/connect tui
```
`go run ./cmd/pipelines-server -help` lists the server flags (`-addr`, `-db`,
`-allow-authoring`, `-opencode-endpoint`, `-mdns`, `-mdns-port`). Without `-db` it uses the standard
per-user data directory (`XDG_DATA_HOME`, or Application Support on macOS).
## Troubleshooting
- **"Docker is not running"**: start Docker Desktop (or the daemon) and rerun.
- **Ports busy**: a Docker installation or an old server may hold 8080, 8082 or
8090. Stop it, or choose other ports with the flags above.
- **"No usable configured harness model"**: rerun `./install` in a terminal to
pick models, or pass `--model PROVIDER/MODEL`.
- **CLI talks to the wrong server**: `PIPELINES_URL` overrides the saved
default; unset it. Check `pipelines connection`.
- **Web app missing**: the server refuses to start without it. Run
`./install --web`.
- **A harness works in the terminal but not in the service**: the service uses
the `PATH` recorded at install time; `pipelines harness setup <name>` says
so. Rerun `./install` from a shell where the harness is on `PATH`. See [Harnesses](harnesses.md#troubleshooting).