Install Pipelines natively by default on macOS and Linux

Add scripts/setup-native.py and make ./install (and --native) build the server, the CLI
and ntfy from source into ~/.local/share/pipelines, detect an existing host OpenCode
and Pi instead of installing them, install the packaged authoring skill, migrate an
existing Docker database and ntfy auth store, save the host client default and start
both services. ./install --docker keeps the previous Docker server + ntfy
installation unchanged.

ntfy is compiled from its pinned Go module on every host because upstream publishes
no macOS server binary (its darwin asset is built with -tags noserver); its cgo
SQLite driver is why a C compiler is now required natively. The ntfy credential
bootstrap and the client default writer are shared by both installers,
installation.Load resolves symlinked prefixes, the Docker migration copies the
SQLite write-ahead log so the newest definitions and runs are not lost, and the
account subcommands read the generated configuration through NTFY_CONFIG_FILE.

Verified on macOS: 115 Python tests, go build/vet/test, and scratch installs that
migrate 9 pipelines and 31 runs, build ntfy v2.30.0 and start both services healthy
on spare ports before stopping them.
This commit is contained in:
2026-10-13 17:20:14 +01:00
parent 9b305df725
commit a70e3aef4b
21 changed files with 1474 additions and 153 deletions
+12 -10
View File
@@ -2,11 +2,12 @@
## Complete installer (recommended for both services in Docker)
Run `./install` from an interactive terminal (Docker is the default;
`./install.sh` remains a compatibility alias). It handles Compose
Run `./install --docker` from an interactive terminal (Docker is the explicit
`--docker`/`docker` mode; `./install.sh` remains a compatibility alias, and a bare
`./install` now performs the native host installation). It handles Compose
configuration, ntfy users/permissions/token, image build, container skill install,
first-time container harness provider login (OpenCode by default; `./install
--harness pi` configures Pi and skips OpenCode login), startup, and an
--docker --harness pi` configures Pi and skips OpenCode login), startup, and an
authenticated test notification. Choose your own provider in the configured
harness; signup/login credentials stay in the Docker volume, not your host
account. Re-run to update while
@@ -80,8 +81,9 @@ are sent until it is filled. `.env`, `secrets/`, and `data/` are git-ignored and
excluded from image builds. The container publishes port **8080**; the default
host mapping is **8081**, bound to host `0.0.0.0`, avoiding the existing native
server on 8080. Query the actual mapped port with
`docker compose port pipelines 8080`. Android does not need access to the
unauthenticated pipeline API.
`docker compose port pipelines 8080`. The unauthenticated pipeline WebSocket API
shares that port, so expose it only on a trusted LAN/VPN and never to the public
internet.
Use HTTP only on a trusted LAN or encrypted VPN. For public access, use HTTPS
with a reverse proxy/VPN; configure the external URL in `NTFY_BASE_URL`. The
@@ -186,8 +188,8 @@ npm, then builds `FROM` the OpenCode image's immutable `sha256` digest and label
the built image with the resolved versions, digest and source identity. It fails
closed (keeping the last verified Dockerfile fallback pins) rather than guessing or
silently substituting a stale pin when discovery, digest resolution, build or
compatibility verification fails. A full `./install` always invokes the Docker build
for a dirty source and relies on layer caching; a model/login-only reconfiguration
rebuilds only when the verified release image identity (versions/digest) no longer
matches the resolved releases, so a source change alone does not rebuild on
reconfigure.
compatibility verification fails. A full `./install --docker` always invokes the
Docker build for a dirty source and relies on layer caching; a model/login-only
reconfiguration rebuilds only when the verified release image identity
(versions/digest) no longer matches the resolved releases, so a source change
alone does not rebuild on reconfigure.
+8 -6
View File
@@ -7,9 +7,10 @@ history. It is not a guarantee that a model completes every task correctly.
## Before installing
- Use a Git checkout (including `.git`) so Docker setup can stamp its build identity.
- **Docker is the only supported runtime installation.** The
[recommended installer](install.md) needs Docker Compose and Python 3; it builds
the Go CLI/TUI clients on the host only when Go 1.27+ is available.
- **Native host installation is the default** (`./install`/`--native`); the
[Docker installation](install.md) is explicit with `./install --docker` and needs
Docker Compose and Python 3. Docker setup builds the Go CLI/TUI clients on the host
only when Go 1.27+ is available.
- Docker setup starts services, asks for ntfy passwords, opens provider sign-in,
and then saves **two separate model defaults** (the interactive harness model and
the new-pipeline model) in two fuzzy pickers. Signing in to a provider is
@@ -25,9 +26,10 @@ history. It is not a guarantee that a model completes every task correctly.
## Install and identify the build
From the checkout, run `./install`, then restart any existing TUI. Docker is the
only supported runtime installation; no mode flag is needed. The installer
requires a Git checkout; a standalone source archive cannot self-install.
From the checkout, run `./install` for the native host installation, or
`./install --docker` for the Docker server + ntfy installation, then restart any
existing TUI. The installer requires a Git checkout; a standalone source archive
cannot self-install.
Follow [installation](install.md) for PATH and provider setup.
Do not start a native server alongside your Docker server.
+6 -4
View File
@@ -3,7 +3,7 @@
Configure two independent model defaults:
```sh
./install --harness pi # Docker setup without OpenCode login
./install --docker --harness pi # Docker setup without OpenCode login
./install --opencode --default
./install --pi
pipelines harness
@@ -34,9 +34,11 @@ On existing Docker setups, the reconfiguration commands
resolve and verify the current OpenCode V2/Pi releases, then operate inside the
container without restarting it. A rebuild is required only when the verified
release image identity (versions/digest) differs from the resolved releases; a
model/login-only change does not rebuild. A full `./install` always invokes the
Docker build for a dirty source and relies on layer caching. Installation is
Docker-only; reconfiguration never falls back to host harness execution.
model/login-only change does not rebuild. A full `./install --docker` always
invokes the Docker build for a dirty source and relies on layer caching. Harness
installation stays Docker-only: the default native setup only detects an existing
host OpenCode/Pi and never installs, upgrades or downloads them, and
reconfiguration never falls back to host harness execution.
Container credentials and accounts are separate from host credentials and are
reused on reinstall/reconfigure.
Defaults affect only new drafts/starters, not existing immutable versions.
+96 -14
View File
@@ -1,21 +1,103 @@
# Installation
To reconfigure login and separate interactive/new-pipeline models without a
service reinstall: `./install --opencode` or `./install --pi`.
`./install` installs **natively on the host by default** (macOS and Linux);
`./install --docker` runs the existing Docker server + ntfy installation.
To reconfigure login and separate interactive/new-pipeline models inside an
existing Docker installation without a service reinstall: `./install --opencode`
or `./install --pi`.
See [configured harnesses](harnesses-and-control.md) for scope and verification limits.
`./install` sets up **Docker server + ntfy**, the only supported runtime installation.
It requires Docker Compose and Python 3 and includes OpenCode/Pi in the persistent
server container. Host Go builds the CLI/TUI clients, not a native server.
```sh
git clone https://git.a-wan.net/alan/pipelines.git
cd pipelines
./install
# Optional: choose Pi rather than OpenCode for setup:
./install --harness pi
./install # native: build server, CLI and ntfy on this host
./install --docker # explicit Docker server + ntfy
```
## Native installation (default)
`./install`, `./install --native` and `./install native` run
`scripts/setup-native.py`. It only detects the tools it needs and never installs
or upgrades them: **Go, python3, git, curl and a C compiler (`cc`/`clang`/`gcc`)**
must already be on PATH. The C compiler is required because ntfy stores its cache
and auth database in SQLite through the cgo driver `mattn/go-sqlite3`. It also
detects an existing `opencode` (required) and `pi` (optional) on PATH and records
their absolute paths; it never downloads, installs or upgrades OpenCode or Pi
(harness installation stays Docker-only).
Setup resolves `PIPELINES_PREFIX` or defaults to `~/.local/share/pipelines`,
creates `bin/`, `lib/`, `data/`, `logs/` and `secrets/` (`secrets` and `data` at
0700), copies `scripts/manage.py` to `<prefix>/lib/manage.py`, builds exactly
`./cmd/pipelines-server` and `./cmd/pipelines` with `go build -trimpath` into
`<prefix>/bin`, and runs `<prefix>/bin/pipelines skill install` so the packaged
authoring skill lands in your OpenCode skills directory. No `pipelines-tui` is
built. It writes `<prefix>/pipelines-install.json` atomically at 0600; that
manifest is what `scripts/manage.py` and the Go CLI load. Provider/model defaults
are never invented: your existing host OpenCode configuration is reused (an
explicit `--model` is recorded if supplied), and setup never starts a picker.
ntfy is built from source with no Homebrew or package manager. Setup resolves the
exact stable `binwiederhier/ntfy` tag (never the moving `latest` tag and never an
older fallback), then compiles the server from the pinned `heckel.io/ntfy/v2` Go
module at that tag into `<prefix>/bin/ntfy`. A source build is required because
upstream publishes **no macOS server binary** (its `darwin_all` asset is built
with `-tags noserver` and only provides the client commands), so both macOS and
Linux take the same path; the Go toolchain verifies the module against its public
checksum database. The build uses
`-tags sqlite_omit_load_extension,osusergo,netgo` and creates the same empty web
placeholders the upstream Makefile uses for local builds, so the bundled ntfy web
UI is not populated (notifications and the API are unaffected). A binary for the
same resolved version is reused instead of rebuilt. Setup generates
`<prefix>/data/ntfy/server.yml` mirroring `compose.yaml` (`listen-http`,
`base-url`, `cache-file`, `auth-file`, `auth-default-access: deny-all`,
`enable-login`, attachment cache directory, expiry and size limits), creates the
`pipeline-publisher` (write-only on `pipelines`) and `phone-reader` (read-only)
accounts with ntfy's own CLI (`--ignore-exists`, so existing passwords are kept;
the account subcommands read the generated config through `NTFY_CONFIG_FILE`),
and writes the publisher token only to `<prefix>/secrets/publisher-token` at 0600.
When Docker volumes `pipelines_pipelines-data` or `pipelines_ntfy-data` exist,
setup migrates the pipeline database to `<prefix>/data/pipelines.db` and the ntfy
auth store to `<prefix>/data/ntfy/auth.db` using the local Pipelines image
(never an implicit pull). Existing native data is never overwritten unless you
pass `--replace-data`; a requested migration that fails aborts the install
instead of starting empty. Absent volumes are a no-op. If the Docker CLI is
present but its volumes cannot be inspected (for example a stopped daemon), setup
fails clearly instead of silently skipping migration, so an existing Docker
database is never shadowed by an empty native start.
Start and manage it with the manifest's prefix on PATH:
```sh
PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start
PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines status
```
`start`, `stop`, `restart`, `doctor`, `logs` and `uninstall` reach
`<prefix>/lib/manage.py`. Ports default to 8080 (server) and 8090 (ntfy) and can
be changed with `--port`/`--ntfy-port`; setup starts both services itself, so
those ports must be free (a running Docker installation holds them — stop it
first; the migration still needs its volumes and image). ntfy runs from
`<prefix>/bin/ntfy` with the generated config; per-user launchd (macOS) or
`systemd --user` (Linux) registration and the supervised fallback both remain.
Dated evidence (2026-10-13, macOS arm64): a scratch-prefix run of
`scripts/setup-native.py` resolved ntfy v2.30.0, compiled it from source (`ntfy
--version` = v2.30.0 with the `Server commands` present), built the server and
CLI, installed the skill, migrated 8 pipeline definitions and 28 runs plus both
ntfy accounts from the live Docker volumes, and wrote the manifest.
`pipelines start` then brought up ntfy and the server through the supervisor
(`{"healthy":true}` from both `/v1/health` and `/healthz` on spare ports) and
`pipelines stop` released both ports. launchd/systemd registration and real phone
delivery are **not tested**.
## Docker installation (explicit)
`./install --docker` (or `./install docker`) runs the unchanged
`scripts/setup-docker.py`. It requires Docker Compose and Python 3 and includes
OpenCode/Pi in the persistent server container. Host Go builds the CLI/TUI
clients, not a native server.
The installer builds host `pipelines` and `pipelines-tui` clients when Go is
available, using `GOBIN` or `GOPATH/bin`; put that directory on PATH. It does not
install a host server. Reconfiguration uses `./install --opencode` or `./install --pi`
@@ -111,9 +193,9 @@ For fresh configuration you can supply `--ntfy-url https://ntfy.example.com
--bind-address 127.0.0.1 --timezone Europe/London --port 8081` directly to `./install`.
The defaults are ready for direct phone access on your trusted LAN. Only use
plain HTTP on a trusted network. Other existing `.env` settings are not replaced.
Docker is also the default without a terminal; setup fails rather than
silently falling back to native binaries if the LAN address cannot be detected.
Use an interactive terminal for first Docker/provider setup.
These Docker flags apply to `./install --docker`; the explicit Docker mode fails
rather than silently falling back to native binaries if the LAN address cannot be
detected. Use an interactive terminal for first Docker/provider setup.
The Docker Pipelines API is unauthenticated: keep its port on a trusted network
and do not forward it to the Internet. For a manual loopback-only deployment,
set `PIPELINES_BIND_ADDRESS=127.0.0.1` after installation and use Compose directly.
@@ -202,8 +284,8 @@ The installer does not hardcode a system destination or run sudo.
## Developer/legacy host server (unsupported)
This explicit foreground server is for developing Pipelines itself or running a
previous native setup. It is **not** a supported installation; a fresh database
starts empty.
previous Python-supervised legacy host setup. It is **not** the native installer;
a fresh database starts empty.
```sh
pipelines serve # foreground developer/legacy server; Ctrl+C stops it