Install Pipelines natively by default on macOS and Linux
Add scripts/setup-native.py and make ./install (and --native) build the server, the CLI and ntfy from source into ~/.local/share/pipelines, detect an existing host OpenCode and Pi instead of installing them, install the packaged authoring skill, migrate an existing Docker database and ntfy auth store, save the host client default and start both services. ./install --docker keeps the previous Docker server + ntfy installation unchanged. ntfy is compiled from its pinned Go module on every host because upstream publishes no macOS server binary (its darwin asset is built with -tags noserver); its cgo SQLite driver is why a C compiler is now required natively. The ntfy credential bootstrap and the client default writer are shared by both installers, installation.Load resolves symlinked prefixes, the Docker migration copies the SQLite write-ahead log so the newest definitions and runs are not lost, and the account subcommands read the generated configuration through NTFY_CONFIG_FILE. Verified on macOS: 115 Python tests, go build/vet/test, and scratch installs that migrate 9 pipelines and 31 runs, build ntfy v2.30.0 and start both services healthy on spare ports before stopping them.
This commit is contained in:
@@ -41,17 +41,33 @@ and the executed route highlighted.
|
||||
|
||||
## Install
|
||||
|
||||
`./install` installs **Docker server + ntfy**. Docker is the only supported runtime
|
||||
installation. Setup needs Docker Compose and Python 3.
|
||||
`./install` installs Pipelines **natively on the host by default** (macOS and
|
||||
Linux); `./install --docker` installs the **Docker server + ntfy** explicitly.
|
||||
Native setup detects (but never installs) Go, python3, git, curl and a C compiler
|
||||
(`cc`/`clang`/`gcc`), plus an existing `opencode`/`pi`; it builds `pipelines-server`,
|
||||
`pipelines` and ntfy from source into `~/.local/share/pipelines` (honouring
|
||||
`PIPELINES_PREFIX`) and generates an ntfy configuration mirroring the Docker
|
||||
settings. ntfy is compiled locally because upstream publishes no macOS server
|
||||
binary, and the C compiler is required because its SQLite driver is cgo. Docker
|
||||
setup needs Docker Compose and Python 3 and includes OpenCode/Pi in the persistent
|
||||
server container.
|
||||
For the host CLI/TUI walkthrough below, also install Go 1.27+; the Docker
|
||||
installer installs/updates those clients when host Go is available.
|
||||
|
||||
```sh
|
||||
git clone https://git.a-wan.net/alan/pipelines.git
|
||||
cd pipelines
|
||||
./install # configure/start Docker services and connect a provider
|
||||
./install # native host install (Go, python3, git, curl, cc, opencode)
|
||||
./install --docker # explicit Docker services + container provider login
|
||||
```
|
||||
|
||||
Native setup reuses your existing host OpenCode login and never installs,
|
||||
upgrades or downloads OpenCode/Pi. Start it with
|
||||
`PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start`.
|
||||
When Docker volumes are present it migrates the existing database and ntfy auth
|
||||
store (never overwriting newer native data; use `--replace-data` to opt in). See
|
||||
[installation](docs/install.md) for details.
|
||||
|
||||
The installer requires this Git checkout (including `.git`); a standalone source
|
||||
archive cannot self-install. Without host Go you can still use the
|
||||
Docker-contained TUI/CLI. Run Compose from the checkout with `--project-directory`
|
||||
@@ -91,7 +107,7 @@ preselected and marked `(current)`; typing filters, arrow keys or Tab navigate,
|
||||
Enter confirms, and Esc cancels without changing anything). Model selection
|
||||
performs no inference; only the OpenCode path validates the new-pipeline model
|
||||
with a small real generation (provider charges may apply). Fresh
|
||||
`./install --harness pi` skips OpenCode login and that generation check. Pi has no
|
||||
`./install --docker --harness pi` skips OpenCode login and that generation check. Pi has no
|
||||
standalone `auth login`: setup opens a fresh `/login` session (regular TUI mode)
|
||||
where you choose an authentication method, reuse existing volume credentials by
|
||||
exiting, or sign in with an API key. Each
|
||||
@@ -107,7 +123,8 @@ services: an image matching the release identity needs no rebuild, while a dirty
|
||||
source always invokes the Docker build and relies on layer caching. The built
|
||||
image includes python3 so the installer/verification tests can also run inside
|
||||
the container.
|
||||
Configuration and data remain per-user. No native server installer is offered.
|
||||
Configuration and data remain per-user. The native default installer is described
|
||||
above and in [installation](docs/install.md).
|
||||
|
||||
Docker setup starts the server and saves its client endpoint (the container port
|
||||
is **8080**; only the default host mapping is **8081**). That saved endpoint is
|
||||
@@ -124,9 +141,10 @@ them to the public Internet.
|
||||
|
||||
### Developer/legacy local server (unsupported)
|
||||
|
||||
For developing Pipelines itself or running a previous native setup, an explicit
|
||||
foreground server remains available. This is **not** a supported end-user
|
||||
installation and is not part of Docker setup; fresh databases contain no pipelines:
|
||||
For developing Pipelines itself or running a previous Python-supervised legacy
|
||||
host setup, an explicit foreground server remains available. This is not the
|
||||
native installer and is not part of Docker setup; fresh databases contain no
|
||||
pipelines:
|
||||
|
||||
```sh
|
||||
pipelines serve # foreground developer/legacy server
|
||||
|
||||
+12
-10
@@ -2,11 +2,12 @@
|
||||
|
||||
## Complete installer (recommended for both services in Docker)
|
||||
|
||||
Run `./install` from an interactive terminal (Docker is the default;
|
||||
`./install.sh` remains a compatibility alias). It handles Compose
|
||||
Run `./install --docker` from an interactive terminal (Docker is the explicit
|
||||
`--docker`/`docker` mode; `./install.sh` remains a compatibility alias, and a bare
|
||||
`./install` now performs the native host installation). It handles Compose
|
||||
configuration, ntfy users/permissions/token, image build, container skill install,
|
||||
first-time container harness provider login (OpenCode by default; `./install
|
||||
--harness pi` configures Pi and skips OpenCode login), startup, and an
|
||||
--docker --harness pi` configures Pi and skips OpenCode login), startup, and an
|
||||
authenticated test notification. Choose your own provider in the configured
|
||||
harness; signup/login credentials stay in the Docker volume, not your host
|
||||
account. Re-run to update while
|
||||
@@ -80,8 +81,9 @@ are sent until it is filled. `.env`, `secrets/`, and `data/` are git-ignored and
|
||||
excluded from image builds. The container publishes port **8080**; the default
|
||||
host mapping is **8081**, bound to host `0.0.0.0`, avoiding the existing native
|
||||
server on 8080. Query the actual mapped port with
|
||||
`docker compose port pipelines 8080`. Android does not need access to the
|
||||
unauthenticated pipeline API.
|
||||
`docker compose port pipelines 8080`. The unauthenticated pipeline WebSocket API
|
||||
shares that port, so expose it only on a trusted LAN/VPN and never to the public
|
||||
internet.
|
||||
|
||||
Use HTTP only on a trusted LAN or encrypted VPN. For public access, use HTTPS
|
||||
with a reverse proxy/VPN; configure the external URL in `NTFY_BASE_URL`. The
|
||||
@@ -186,8 +188,8 @@ npm, then builds `FROM` the OpenCode image's immutable `sha256` digest and label
|
||||
the built image with the resolved versions, digest and source identity. It fails
|
||||
closed (keeping the last verified Dockerfile fallback pins) rather than guessing or
|
||||
silently substituting a stale pin when discovery, digest resolution, build or
|
||||
compatibility verification fails. A full `./install` always invokes the Docker build
|
||||
for a dirty source and relies on layer caching; a model/login-only reconfiguration
|
||||
rebuilds only when the verified release image identity (versions/digest) no longer
|
||||
matches the resolved releases, so a source change alone does not rebuild on
|
||||
reconfigure.
|
||||
compatibility verification fails. A full `./install --docker` always invokes the
|
||||
Docker build for a dirty source and relies on layer caching; a model/login-only
|
||||
reconfiguration rebuilds only when the verified release image identity
|
||||
(versions/digest) no longer matches the resolved releases, so a source change
|
||||
alone does not rebuild on reconfigure.
|
||||
|
||||
@@ -7,9 +7,10 @@ history. It is not a guarantee that a model completes every task correctly.
|
||||
## Before installing
|
||||
|
||||
- Use a Git checkout (including `.git`) so Docker setup can stamp its build identity.
|
||||
- **Docker is the only supported runtime installation.** The
|
||||
[recommended installer](install.md) needs Docker Compose and Python 3; it builds
|
||||
the Go CLI/TUI clients on the host only when Go 1.27+ is available.
|
||||
- **Native host installation is the default** (`./install`/`--native`); the
|
||||
[Docker installation](install.md) is explicit with `./install --docker` and needs
|
||||
Docker Compose and Python 3. Docker setup builds the Go CLI/TUI clients on the host
|
||||
only when Go 1.27+ is available.
|
||||
- Docker setup starts services, asks for ntfy passwords, opens provider sign-in,
|
||||
and then saves **two separate model defaults** (the interactive harness model and
|
||||
the new-pipeline model) in two fuzzy pickers. Signing in to a provider is
|
||||
@@ -25,9 +26,10 @@ history. It is not a guarantee that a model completes every task correctly.
|
||||
|
||||
## Install and identify the build
|
||||
|
||||
From the checkout, run `./install`, then restart any existing TUI. Docker is the
|
||||
only supported runtime installation; no mode flag is needed. The installer
|
||||
requires a Git checkout; a standalone source archive cannot self-install.
|
||||
From the checkout, run `./install` for the native host installation, or
|
||||
`./install --docker` for the Docker server + ntfy installation, then restart any
|
||||
existing TUI. The installer requires a Git checkout; a standalone source archive
|
||||
cannot self-install.
|
||||
Follow [installation](install.md) for PATH and provider setup.
|
||||
Do not start a native server alongside your Docker server.
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Configure two independent model defaults:
|
||||
|
||||
```sh
|
||||
./install --harness pi # Docker setup without OpenCode login
|
||||
./install --docker --harness pi # Docker setup without OpenCode login
|
||||
./install --opencode --default
|
||||
./install --pi
|
||||
pipelines harness
|
||||
@@ -34,9 +34,11 @@ On existing Docker setups, the reconfiguration commands
|
||||
resolve and verify the current OpenCode V2/Pi releases, then operate inside the
|
||||
container without restarting it. A rebuild is required only when the verified
|
||||
release image identity (versions/digest) differs from the resolved releases; a
|
||||
model/login-only change does not rebuild. A full `./install` always invokes the
|
||||
Docker build for a dirty source and relies on layer caching. Installation is
|
||||
Docker-only; reconfiguration never falls back to host harness execution.
|
||||
model/login-only change does not rebuild. A full `./install --docker` always
|
||||
invokes the Docker build for a dirty source and relies on layer caching. Harness
|
||||
installation stays Docker-only: the default native setup only detects an existing
|
||||
host OpenCode/Pi and never installs, upgrades or downloads them, and
|
||||
reconfiguration never falls back to host harness execution.
|
||||
Container credentials and accounts are separate from host credentials and are
|
||||
reused on reinstall/reconfigure.
|
||||
Defaults affect only new drafts/starters, not existing immutable versions.
|
||||
|
||||
+96
-14
@@ -1,21 +1,103 @@
|
||||
# Installation
|
||||
|
||||
To reconfigure login and separate interactive/new-pipeline models without a
|
||||
service reinstall: `./install --opencode` or `./install --pi`.
|
||||
`./install` installs **natively on the host by default** (macOS and Linux);
|
||||
`./install --docker` runs the existing Docker server + ntfy installation.
|
||||
To reconfigure login and separate interactive/new-pipeline models inside an
|
||||
existing Docker installation without a service reinstall: `./install --opencode`
|
||||
or `./install --pi`.
|
||||
See [configured harnesses](harnesses-and-control.md) for scope and verification limits.
|
||||
|
||||
`./install` sets up **Docker server + ntfy**, the only supported runtime installation.
|
||||
It requires Docker Compose and Python 3 and includes OpenCode/Pi in the persistent
|
||||
server container. Host Go builds the CLI/TUI clients, not a native server.
|
||||
|
||||
```sh
|
||||
git clone https://git.a-wan.net/alan/pipelines.git
|
||||
cd pipelines
|
||||
./install
|
||||
# Optional: choose Pi rather than OpenCode for setup:
|
||||
./install --harness pi
|
||||
./install # native: build server, CLI and ntfy on this host
|
||||
./install --docker # explicit Docker server + ntfy
|
||||
```
|
||||
|
||||
## Native installation (default)
|
||||
|
||||
`./install`, `./install --native` and `./install native` run
|
||||
`scripts/setup-native.py`. It only detects the tools it needs and never installs
|
||||
or upgrades them: **Go, python3, git, curl and a C compiler (`cc`/`clang`/`gcc`)**
|
||||
must already be on PATH. The C compiler is required because ntfy stores its cache
|
||||
and auth database in SQLite through the cgo driver `mattn/go-sqlite3`. It also
|
||||
detects an existing `opencode` (required) and `pi` (optional) on PATH and records
|
||||
their absolute paths; it never downloads, installs or upgrades OpenCode or Pi
|
||||
(harness installation stays Docker-only).
|
||||
|
||||
Setup resolves `PIPELINES_PREFIX` or defaults to `~/.local/share/pipelines`,
|
||||
creates `bin/`, `lib/`, `data/`, `logs/` and `secrets/` (`secrets` and `data` at
|
||||
0700), copies `scripts/manage.py` to `<prefix>/lib/manage.py`, builds exactly
|
||||
`./cmd/pipelines-server` and `./cmd/pipelines` with `go build -trimpath` into
|
||||
`<prefix>/bin`, and runs `<prefix>/bin/pipelines skill install` so the packaged
|
||||
authoring skill lands in your OpenCode skills directory. No `pipelines-tui` is
|
||||
built. It writes `<prefix>/pipelines-install.json` atomically at 0600; that
|
||||
manifest is what `scripts/manage.py` and the Go CLI load. Provider/model defaults
|
||||
are never invented: your existing host OpenCode configuration is reused (an
|
||||
explicit `--model` is recorded if supplied), and setup never starts a picker.
|
||||
|
||||
ntfy is built from source with no Homebrew or package manager. Setup resolves the
|
||||
exact stable `binwiederhier/ntfy` tag (never the moving `latest` tag and never an
|
||||
older fallback), then compiles the server from the pinned `heckel.io/ntfy/v2` Go
|
||||
module at that tag into `<prefix>/bin/ntfy`. A source build is required because
|
||||
upstream publishes **no macOS server binary** (its `darwin_all` asset is built
|
||||
with `-tags noserver` and only provides the client commands), so both macOS and
|
||||
Linux take the same path; the Go toolchain verifies the module against its public
|
||||
checksum database. The build uses
|
||||
`-tags sqlite_omit_load_extension,osusergo,netgo` and creates the same empty web
|
||||
placeholders the upstream Makefile uses for local builds, so the bundled ntfy web
|
||||
UI is not populated (notifications and the API are unaffected). A binary for the
|
||||
same resolved version is reused instead of rebuilt. Setup generates
|
||||
`<prefix>/data/ntfy/server.yml` mirroring `compose.yaml` (`listen-http`,
|
||||
`base-url`, `cache-file`, `auth-file`, `auth-default-access: deny-all`,
|
||||
`enable-login`, attachment cache directory, expiry and size limits), creates the
|
||||
`pipeline-publisher` (write-only on `pipelines`) and `phone-reader` (read-only)
|
||||
accounts with ntfy's own CLI (`--ignore-exists`, so existing passwords are kept;
|
||||
the account subcommands read the generated config through `NTFY_CONFIG_FILE`),
|
||||
and writes the publisher token only to `<prefix>/secrets/publisher-token` at 0600.
|
||||
|
||||
When Docker volumes `pipelines_pipelines-data` or `pipelines_ntfy-data` exist,
|
||||
setup migrates the pipeline database to `<prefix>/data/pipelines.db` and the ntfy
|
||||
auth store to `<prefix>/data/ntfy/auth.db` using the local Pipelines image
|
||||
(never an implicit pull). Existing native data is never overwritten unless you
|
||||
pass `--replace-data`; a requested migration that fails aborts the install
|
||||
instead of starting empty. Absent volumes are a no-op. If the Docker CLI is
|
||||
present but its volumes cannot be inspected (for example a stopped daemon), setup
|
||||
fails clearly instead of silently skipping migration, so an existing Docker
|
||||
database is never shadowed by an empty native start.
|
||||
|
||||
Start and manage it with the manifest's prefix on PATH:
|
||||
|
||||
```sh
|
||||
PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start
|
||||
PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines status
|
||||
```
|
||||
|
||||
`start`, `stop`, `restart`, `doctor`, `logs` and `uninstall` reach
|
||||
`<prefix>/lib/manage.py`. Ports default to 8080 (server) and 8090 (ntfy) and can
|
||||
be changed with `--port`/`--ntfy-port`; setup starts both services itself, so
|
||||
those ports must be free (a running Docker installation holds them — stop it
|
||||
first; the migration still needs its volumes and image). ntfy runs from
|
||||
`<prefix>/bin/ntfy` with the generated config; per-user launchd (macOS) or
|
||||
`systemd --user` (Linux) registration and the supervised fallback both remain.
|
||||
|
||||
Dated evidence (2026-10-13, macOS arm64): a scratch-prefix run of
|
||||
`scripts/setup-native.py` resolved ntfy v2.30.0, compiled it from source (`ntfy
|
||||
--version` = v2.30.0 with the `Server commands` present), built the server and
|
||||
CLI, installed the skill, migrated 8 pipeline definitions and 28 runs plus both
|
||||
ntfy accounts from the live Docker volumes, and wrote the manifest.
|
||||
`pipelines start` then brought up ntfy and the server through the supervisor
|
||||
(`{"healthy":true}` from both `/v1/health` and `/healthz` on spare ports) and
|
||||
`pipelines stop` released both ports. launchd/systemd registration and real phone
|
||||
delivery are **not tested**.
|
||||
|
||||
## Docker installation (explicit)
|
||||
|
||||
`./install --docker` (or `./install docker`) runs the unchanged
|
||||
`scripts/setup-docker.py`. It requires Docker Compose and Python 3 and includes
|
||||
OpenCode/Pi in the persistent server container. Host Go builds the CLI/TUI
|
||||
clients, not a native server.
|
||||
|
||||
The installer builds host `pipelines` and `pipelines-tui` clients when Go is
|
||||
available, using `GOBIN` or `GOPATH/bin`; put that directory on PATH. It does not
|
||||
install a host server. Reconfiguration uses `./install --opencode` or `./install --pi`
|
||||
@@ -111,9 +193,9 @@ For fresh configuration you can supply `--ntfy-url https://ntfy.example.com
|
||||
--bind-address 127.0.0.1 --timezone Europe/London --port 8081` directly to `./install`.
|
||||
The defaults are ready for direct phone access on your trusted LAN. Only use
|
||||
plain HTTP on a trusted network. Other existing `.env` settings are not replaced.
|
||||
Docker is also the default without a terminal; setup fails rather than
|
||||
silently falling back to native binaries if the LAN address cannot be detected.
|
||||
Use an interactive terminal for first Docker/provider setup.
|
||||
These Docker flags apply to `./install --docker`; the explicit Docker mode fails
|
||||
rather than silently falling back to native binaries if the LAN address cannot be
|
||||
detected. Use an interactive terminal for first Docker/provider setup.
|
||||
The Docker Pipelines API is unauthenticated: keep its port on a trusted network
|
||||
and do not forward it to the Internet. For a manual loopback-only deployment,
|
||||
set `PIPELINES_BIND_ADDRESS=127.0.0.1` after installation and use Compose directly.
|
||||
@@ -202,8 +284,8 @@ The installer does not hardcode a system destination or run sudo.
|
||||
## Developer/legacy host server (unsupported)
|
||||
|
||||
This explicit foreground server is for developing Pipelines itself or running a
|
||||
previous native setup. It is **not** a supported installation; a fresh database
|
||||
starts empty.
|
||||
previous Python-supervised legacy host setup. It is **not** the native installer;
|
||||
a fresh database starts empty.
|
||||
|
||||
```sh
|
||||
pipelines serve # foreground developer/legacy server; Ctrl+C stops it
|
||||
|
||||
+3
-3
@@ -13,8 +13,8 @@ case "$mode" in
|
||||
exec python3 scripts/setup-docker.py "$@"
|
||||
;;
|
||||
--native|native)
|
||||
echo 'Native installation is not supported. Run ./install for Docker setup.' >&2
|
||||
exit 2
|
||||
shift
|
||||
exec python3 scripts/setup-native.py "$@"
|
||||
;;
|
||||
*) exec python3 scripts/setup-docker.py "$@" ;;
|
||||
*) exec python3 scripts/setup-native.py "$@" ;;
|
||||
esac
|
||||
|
||||
@@ -63,10 +63,13 @@ stay authoritative and a dirty source still rebuilds. If discovery, digest
|
||||
resolution or completeness cannot be established they stop with an error (keeping
|
||||
the last verified Dockerfile fallback pins); they never guess, fall back to a host
|
||||
runtime or install a stale pin.
|
||||
Without Docker setup, these commands stop and require `./install`. Container/host
|
||||
credentials and accounts remain independent and are reused on reinstall/reconfigure.
|
||||
Docker is the only supported runtime installation: run `./install`, with no mode flag.
|
||||
Fresh setup accepts `./install --harness pi` to skip OpenCode
|
||||
Without a Docker installation, `./install --opencode`/`--pi` stop and require an
|
||||
existing Docker setup; the native installation reuses the host OpenCode
|
||||
configuration. Container and host credentials and accounts remain independent and
|
||||
are reused on reinstall/reconfigure. `./install` installs natively by default
|
||||
(`--native` is equivalent); explicit `./install --docker` selects the Docker
|
||||
server + ntfy installation.
|
||||
Fresh setup accepts `./install --docker --harness pi` to skip OpenCode
|
||||
login. Existing setup can configure either harness independently. Pi setup does
|
||||
not run a generation check; `provider-check --harness pi --model PROVIDER/MODEL`
|
||||
is a separate confirmed, potentially chargeable check.
|
||||
|
||||
@@ -102,8 +102,10 @@ raw credential/session databases or publish private inspect output as diagnostic
|
||||
In a source checkout, `python3 scripts/smoke-first-install.py` checks a disposable
|
||||
source-built test binaries, draft/publication/transfer and restart persistence without
|
||||
inference, notifications or deployment. It is not a live-provider or phone test.
|
||||
That is a developer test fixture, not a native installation option. End-user
|
||||
installation is Docker-only through `./install`; no `--docker` flag is needed.
|
||||
That is a developer test fixture, not the native installer. End-user installation
|
||||
runs natively through `./install` by default (`scripts/setup-native.py`, with
|
||||
`--native` equivalent); the Docker server + ntfy installation is explicit with
|
||||
`./install --docker`.
|
||||
|
||||
In the TUI, Home advertises `/` search and `:` commands. Run Tab switches only
|
||||
Details ↔ Graph. `n/p` select recorded visits, including repeated stages; `d`
|
||||
|
||||
@@ -29,6 +29,12 @@ func Load(executable string) (*Config, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// The native installer and scripts/manage.py record the symlink-resolved
|
||||
// prefix, so resolve the same way here instead of rejecting a symlinked
|
||||
// HOME or prefix as "invalid".
|
||||
if resolved, err := filepath.EvalSymlinks(prefix); err == nil {
|
||||
prefix = resolved
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(prefix, "pipelines-install.json"))
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Create local Compose settings without overwriting settings or credentials."""
|
||||
"""Create local Compose settings and the host client default."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
from urllib.parse import urlparse
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
@@ -42,6 +44,36 @@ def host_workspaces(directory):
|
||||
return {"PIPELINES_HOST_HOME": "'"+home+"'", "PIPELINES_HOST_REPOSITORY": "'"+repository+"'", "PIPELINES_UID": str(os.getuid()), "PIPELINES_GID": str(os.getgid())}
|
||||
|
||||
|
||||
def configure_client(endpoint, environ=None):
|
||||
"""Save the server address the host CLI/TUI dials by default.
|
||||
|
||||
Atomic, 0600, and it preserves unrelated keys. Shared by both installers so
|
||||
the saved default always matches the installation that just ran.
|
||||
"""
|
||||
environ = os.environ if environ is None else environ
|
||||
root = Path(environ.get("XDG_CONFIG_HOME") or Path.home() / ".config")
|
||||
path = root / "pipelines" / "client.json"
|
||||
if path.is_symlink():
|
||||
raise ValueError("Refusing to modify symlinked client.json; use the printed --url explicitly")
|
||||
existing = json.loads(path.read_text()) if path.exists() else {}
|
||||
if not isinstance(existing, dict):
|
||||
raise ValueError("client.json must contain an object; existing configuration was not replaced")
|
||||
existing["server_url"] = endpoint
|
||||
path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
temporary = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(mode="w", dir=str(path.parent), delete=False) as file:
|
||||
temporary = Path(file.name)
|
||||
os.chmod(temporary, 0o600)
|
||||
json.dump(existing, file, indent=2)
|
||||
file.write("\n")
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
if temporary is not None:
|
||||
temporary.unlink(missing_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def configure(directory, ntfy_url, bind_address, timezone=None, port=8081):
|
||||
directory = Path(directory)
|
||||
url = urlparse(ntfy_url)
|
||||
|
||||
+90
-16
@@ -17,6 +17,15 @@ DEFAULT_PREFIX = Path.home() / ".local/share/pipelines"
|
||||
MARKER = "pipelines-install.json"
|
||||
|
||||
|
||||
def ntfy_mode(config):
|
||||
"""Native ntfy is selected by the manifest; absent/explicit docker uses Compose."""
|
||||
return (config.get("ntfy") or {}).get("mode", "docker")
|
||||
|
||||
|
||||
def ntfy_health_url(config):
|
||||
return f"http://127.0.0.1:{config['ntfy_port']}/v1/health"
|
||||
|
||||
|
||||
def load(prefix):
|
||||
prefix = Path(prefix).expanduser().resolve()
|
||||
path = prefix / MARKER
|
||||
@@ -60,19 +69,27 @@ def wait_health(url, timeout=40):
|
||||
raise RuntimeError(f"Service did not become healthy: {url}; see pipelines logs")
|
||||
|
||||
|
||||
def process_command(pid):
|
||||
"""Return the OS command line for a live PID, or None when it cannot be read."""
|
||||
try:
|
||||
return subprocess.check_output(["ps", "-p", str(pid), "-o", "command="], text=True)
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def managed_pid(config):
|
||||
path = Path(config["prefix"]) / "supervisor.pid"
|
||||
try:
|
||||
pid = int(path.read_text())
|
||||
if pid <= 1:
|
||||
return None
|
||||
command = subprocess.check_output(["ps", "-p", str(pid), "-o", "command="], text=True)
|
||||
# Never signal an arbitrary/reused PID.
|
||||
if str(Path(config["prefix"]) / "lib/manage.py") not in command or "supervise" not in command:
|
||||
return None
|
||||
return pid
|
||||
except (OSError, ValueError, subprocess.CalledProcessError):
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
if pid <= 1:
|
||||
return None
|
||||
command = process_command(pid)
|
||||
# Never signal an arbitrary/reused PID.
|
||||
if command is None or str(Path(config["prefix"]) / "lib/manage.py") not in command or "supervise" not in command:
|
||||
return None
|
||||
return pid
|
||||
|
||||
|
||||
def registration(config):
|
||||
@@ -112,7 +129,48 @@ def stop(config):
|
||||
time.sleep(0.2)
|
||||
if managed_pid(config) is not None:
|
||||
raise RuntimeError("Supervisor did not stop; inspect logs before trying again")
|
||||
compose(config, "stop", "ntfy", quiet=True)
|
||||
if ntfy_mode(config) == "native":
|
||||
ntfy_stop(config)
|
||||
else:
|
||||
compose(config, "stop", "ntfy", quiet=True)
|
||||
|
||||
|
||||
def ntfy_start(config):
|
||||
info = config["ntfy"]
|
||||
log = (Path(config["prefix"]) / "logs/ntfy.log").open("a")
|
||||
child = subprocess.Popen([info["bin"], "serve", "--config", info["config"]],
|
||||
stdin=subprocess.DEVNULL, stdout=log, stderr=log, start_new_session=True)
|
||||
log.close()
|
||||
(Path(config["prefix"]) / "ntfy.pid").write_text(str(child.pid))
|
||||
return child
|
||||
|
||||
|
||||
def ntfy_stop(config):
|
||||
path = Path(config["prefix"]) / "ntfy.pid"
|
||||
try:
|
||||
pid = int(path.read_text())
|
||||
except (OSError, ValueError):
|
||||
return
|
||||
if pid <= 1:
|
||||
return
|
||||
command = process_command(pid)
|
||||
# Never signal an arbitrary/reused PID: only the managed ntfy serve process.
|
||||
# Accept the recorded path or its symlink-resolved form so a prefix reached
|
||||
# through a symlinked HOME (macOS /var → /private/var) is still matched.
|
||||
binary = str(config["ntfy"]["bin"])
|
||||
candidates = {binary, str(Path(binary).resolve())}
|
||||
if command is None or "serve" not in command or not any(candidate in command for candidate in candidates):
|
||||
path.unlink(missing_ok=True)
|
||||
return
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
deadline = time.monotonic() + 10
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def supervise(config):
|
||||
@@ -126,18 +184,23 @@ def supervise(config):
|
||||
pidfile.write_text(str(os.getpid()))
|
||||
stopped = False
|
||||
child = None
|
||||
ntfy_child = None
|
||||
|
||||
def stopping(*_):
|
||||
nonlocal stopped
|
||||
stopped = True
|
||||
if child is not None and child.poll() is None:
|
||||
child.terminate()
|
||||
for process in (child, ntfy_child):
|
||||
if process is not None and process.poll() is None:
|
||||
process.terminate()
|
||||
|
||||
signal.signal(signal.SIGTERM, stopping)
|
||||
signal.signal(signal.SIGINT, stopping)
|
||||
try:
|
||||
compose(config, "up", "-d", "ntfy", quiet=True)
|
||||
wait_health(f"http://127.0.0.1:{config['ntfy_port']}/v1/health")
|
||||
if ntfy_mode(config) == "native":
|
||||
ntfy_child = ntfy_start(config)
|
||||
else:
|
||||
compose(config, "up", "-d", "ntfy", quiet=True)
|
||||
wait_health(ntfy_health_url(config))
|
||||
if stopped:
|
||||
return
|
||||
env = dict(os.environ)
|
||||
@@ -163,7 +226,17 @@ def supervise(config):
|
||||
except subprocess.TimeoutExpired:
|
||||
child.kill()
|
||||
child.wait()
|
||||
compose(config, "stop", "ntfy", quiet=True, check=False)
|
||||
if ntfy_mode(config) == "native":
|
||||
if ntfy_child is not None and ntfy_child.poll() is None:
|
||||
ntfy_child.terminate()
|
||||
try:
|
||||
ntfy_child.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
ntfy_child.kill()
|
||||
ntfy_child.wait()
|
||||
(root / "ntfy.pid").unlink(missing_ok=True)
|
||||
else:
|
||||
compose(config, "stop", "ntfy", quiet=True, check=False)
|
||||
pidfile.unlink(missing_ok=True)
|
||||
|
||||
|
||||
@@ -178,7 +251,8 @@ def uninstall(config, purge=False, yes=False):
|
||||
if platform.system() != "Darwin":
|
||||
run(["systemctl", "--user", "disable", config["label"] + ".service"], quiet=True, check=False)
|
||||
Path(registration(config)).unlink(missing_ok=True)
|
||||
compose(config, "down", *(["--volumes"] if purge else []), quiet=True)
|
||||
if ntfy_mode(config) != "native":
|
||||
compose(config, "down", *(["--volumes"] if purge else []), quiet=True)
|
||||
link = Path(config["launcher"])
|
||||
if link.is_symlink() and link.resolve() == (root / "bin/pipelines").resolve():
|
||||
if str(link)=="/usr/local/bin/pipelines" and not os.access(link.parent,os.W_OK):
|
||||
@@ -231,7 +305,7 @@ def main():
|
||||
stop(config); start(config)
|
||||
elif args.action in ("status", "doctor"):
|
||||
print("Pipeline server:", "healthy" if healthy(f"http://127.0.0.1:{config['server_port']}/healthz") else "stopped/unhealthy")
|
||||
print("ntfy:", "healthy" if healthy(f"http://127.0.0.1:{config['ntfy_port']}/v1/health") else "stopped/unhealthy")
|
||||
print("ntfy:", "healthy" if healthy(ntfy_health_url(config)) else "stopped/unhealthy")
|
||||
print("Phone server:", config["public_url"], "· topic:", config["topic"], "· username:", config["username"])
|
||||
if args.action == "doctor":
|
||||
run([config["opencode"], "api", "get", "/api/info"], quiet=True)
|
||||
|
||||
+62
-13
@@ -1,10 +1,10 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bounded resolution of current harness releases for the Docker image.
|
||||
"""Bounded resolution of current harness and ntfy releases for installation.
|
||||
|
||||
Network I/O is isolated behind an injectable ``fetch`` callable so unit tests
|
||||
never touch the network. The resolver:
|
||||
|
||||
* discovers supported V2 releases from one bounded GitHub request to
|
||||
* discovers supported V2 OpenCode releases from one bounded GitHub request to
|
||||
``git/matching-refs/tags/v2``, which the official REST docs
|
||||
(https://docs.github.com/en/rest/git/refs#list-matching-references) document
|
||||
as a complete array with no pagination parameters. There is no page/per_page
|
||||
@@ -15,9 +15,14 @@ never touch the network. The resolver:
|
||||
ones; it never uses the moving ``releases/latest`` anchor;
|
||||
* resolves the selected tag's immutable ``sha256`` digest from GHCR and fails
|
||||
closed when it is missing or invalid, never falling back to an older version;
|
||||
* resolves the exact Pi package version and validates its Node engine
|
||||
constraint fail-closed;
|
||||
* resolves the exact ntfy release tag that the native installer builds its
|
||||
server from, because upstream publishes only a client-only binary for macOS
|
||||
and the server is therefore compiled from the pinned Go module on every host;
|
||||
* validates untrusted metadata, versions, digests and fixture values;
|
||||
* refuses non-registry hosts and unexpected redirects;
|
||||
* validates the Pi Node engine constraint fail-closed.
|
||||
* refuses any host outside the explicit allowlist (the container registry, the
|
||||
GitHub API and the npm registry) and refuses any redirect that leaves it.
|
||||
|
||||
The selected versions/digests are data, not evergreen hardcoded answers.
|
||||
"""
|
||||
@@ -38,10 +43,18 @@ NPM = "https://registry.npmjs.org"
|
||||
OPENCODE_REPOSITORY = "anomalyco/opencode"
|
||||
OPENCODE_IMAGE = "ghcr.io/anomalyco/opencode"
|
||||
PI_PACKAGE = "@earendil-works/pi-coding-agent"
|
||||
NTFY_REPOSITORY = "binwiederhier/ntfy"
|
||||
# The ntfy Go module the native installer builds a server from. Upstream only
|
||||
# publishes a client-only binary for macOS, so both platforms compile the server
|
||||
# from this module at the resolved tag (Go verifies the module via its checksum
|
||||
# database).
|
||||
NTFY_MODULE = "heckel.io/ntfy/v2"
|
||||
MAX_REFS = 1000 # Explicit ceiling on the documented complete matching-refs array.
|
||||
MAX_BODY = 1 << 20 # Generous ceiling for tag lists and manifest metadata.
|
||||
SUPPORTED_MAJOR = 2
|
||||
HTTP_TIMEOUT = 5.0
|
||||
# Registry, GitHub API and npm metadata hosts only. The native installer builds
|
||||
# ntfy from its Go module, so no release-asset download hosts are needed.
|
||||
ALLOWED_HOSTS = frozenset({"ghcr.io", "api.github.com", "registry.npmjs.org"})
|
||||
GITHUB_HEADERS = {"Accept": "application/vnd.github+json", "User-Agent": "pipelines-setup"}
|
||||
MANIFEST_ACCEPT = ", ".join([
|
||||
@@ -72,19 +85,26 @@ def parse_version(text):
|
||||
return tuple(int(part) for part in match.groups())
|
||||
|
||||
|
||||
class _NoRedirect(urllib.request.HTTPRedirectHandler):
|
||||
"""Registry metadata lookups must never be silently redirected elsewhere."""
|
||||
class _AllowlistRedirect(urllib.request.HTTPRedirectHandler):
|
||||
"""Follow only redirects that stay on an explicitly allowed HTTPS host."""
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
raise ReleaseError(f"refusing unexpected redirect from {req.full_url!r} to {newurl!r}")
|
||||
if not is_allowlisted(newurl):
|
||||
raise ReleaseError(f"refusing unexpected redirect from {req.full_url!r} to {newurl!r}")
|
||||
return super().redirect_request(req, fp, code, msg, headers, newurl)
|
||||
|
||||
|
||||
_OPENER = urllib.request.build_opener(_NoRedirect)
|
||||
_OPENER = urllib.request.build_opener(_AllowlistRedirect)
|
||||
|
||||
|
||||
def is_allowlisted(url):
|
||||
"""True only for HTTPS URLs on an explicitly allowed host (shared guard)."""
|
||||
parsed = urllib.parse.urlsplit(url)
|
||||
return parsed.scheme == "https" and parsed.hostname in ALLOWED_HOSTS
|
||||
|
||||
|
||||
def _default_fetch(url, headers=None):
|
||||
parsed = urllib.parse.urlsplit(url)
|
||||
if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS:
|
||||
if not is_allowlisted(url):
|
||||
raise ReleaseError(f"refusing non-registry host {url!r}")
|
||||
request = urllib.request.Request(url, headers=headers or {})
|
||||
try:
|
||||
@@ -118,15 +138,16 @@ def _json(fetch, url, headers=None):
|
||||
return data
|
||||
|
||||
|
||||
def _github_v2_tags(fetch):
|
||||
def _github_matching_refs(fetch, repository, tag_prefix, normalize=lambda tag: tag):
|
||||
"""One bounded request for the documented complete matching-refs array.
|
||||
|
||||
GitHub's "List matching references" endpoint documents a complete array and
|
||||
no pagination parameters, so a single ``MAX_BODY``/``MAX_REFS``-bounded GET
|
||||
is sufficient. Exact stable versions become candidates; version-shaped tags
|
||||
with a prerelease/build suffix are ignored; other malformed tags fail closed.
|
||||
``normalize`` maps a raw tag to the candidate key (OpenCode strips the ``v``).
|
||||
"""
|
||||
url = f"{GITHUB}/repos/{OPENCODE_REPOSITORY}/git/matching-refs/tags/v{SUPPORTED_MAJOR}"
|
||||
url = f"{GITHUB}/repos/{repository}/git/matching-refs/tags/{tag_prefix}"
|
||||
data, _ = _response_json(fetch, url, GITHUB_HEADERS)
|
||||
if not isinstance(data, list):
|
||||
raise ReleaseError("GitHub matching-refs payload is not a list")
|
||||
@@ -141,7 +162,7 @@ def _github_v2_tags(fetch):
|
||||
ref = entry.get("ref")
|
||||
if not isinstance(ref, str) or not ref.startswith("refs/tags/"):
|
||||
raise ReleaseError(f"GitHub matching-refs entry has an unexpected ref {ref!r}")
|
||||
tag = registry_tag(ref[len("refs/tags/"):])
|
||||
tag = normalize(ref[len("refs/tags/"):])
|
||||
version = parse_version(tag)
|
||||
if version is None:
|
||||
if VERSION_SUFFIX.fullmatch(tag):
|
||||
@@ -152,6 +173,10 @@ def _github_v2_tags(fetch):
|
||||
return candidates
|
||||
|
||||
|
||||
def _github_v2_tags(fetch):
|
||||
return _github_matching_refs(fetch, OPENCODE_REPOSITORY, f"v{SUPPORTED_MAJOR}", registry_tag)
|
||||
|
||||
|
||||
def _ghcr_token(fetch):
|
||||
data = _json(fetch, f"{GHCR}/token?scope=repository:{OPENCODE_REPOSITORY}:pull")
|
||||
if not isinstance(data, dict):
|
||||
@@ -247,6 +272,30 @@ def resolve_pi(fetch=_default_fetch, node_version=None):
|
||||
return {"package": PI_PACKAGE, "version": version, "engine": engine}
|
||||
|
||||
|
||||
def _github_ntfy_tags(fetch):
|
||||
"""One bounded request for the complete ntfy vX.Y.Z matching-refs array."""
|
||||
return _github_matching_refs(fetch, NTFY_REPOSITORY, "v")
|
||||
|
||||
|
||||
def resolve_ntfy(fetch=_default_fetch):
|
||||
"""Resolve the exact ntfy release the native installer builds from source.
|
||||
|
||||
The selected version comes from the complete official matching-refs tag list
|
||||
(never the moving ``latest`` release and never an older fallback); a missing
|
||||
or malformed tag list fails closed. The native installer then compiles the
|
||||
server from the pinned ``heckel.io/ntfy/v2`` module, so no release assets or
|
||||
their checksums are downloaded here.
|
||||
"""
|
||||
candidates = _github_ntfy_tags(fetch)
|
||||
if not candidates:
|
||||
raise ReleaseError("no stable ntfy release found in the official tag metadata")
|
||||
tag = max(candidates, key=lambda name: candidates[name])
|
||||
version = registry_tag(tag)
|
||||
if parse_version(version) is None:
|
||||
raise ReleaseError(f"resolved ntfy tag is not an exact version: {tag!r}")
|
||||
return {"repository": NTFY_REPOSITORY, "module": NTFY_MODULE, "version": version, "tag": tag}
|
||||
|
||||
|
||||
def resolve(fetch=_default_fetch, node_version=None):
|
||||
return {"opencode": resolve_opencode(fetch), "pi": resolve_pi(fetch, node_version)}
|
||||
|
||||
|
||||
+45
-12
@@ -1,33 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract only the token from ntfy's CLI response; never print the credential."""
|
||||
"""Shared ntfy credential helpers: token extraction and account bootstrap.
|
||||
|
||||
Both installers (native and Docker) create the same publisher/reader accounts on
|
||||
ntfy, so the extraction, symlink/non-empty guard and writing live here once.
|
||||
Never print a credential.
|
||||
"""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
|
||||
TOKEN_PATTERN = re.compile(r"\btk_[A-Za-z0-9]{29}\b")
|
||||
|
||||
def save_token(output, directory):
|
||||
tokens = re.findall(r"\btk_[A-Za-z0-9]{29}\b", output)
|
||||
|
||||
def save_token(output, path, mode=0o444):
|
||||
"""Write the single token found in ``output`` to ``path``.
|
||||
|
||||
``mode`` is 0444 for Compose's file secret (read by container uid 10001) and
|
||||
0600 for the native install, whose server runs as the installing user.
|
||||
"""
|
||||
tokens = TOKEN_PATTERN.findall(output or "")
|
||||
if len(tokens) != 1:
|
||||
raise ValueError("ntfy did not return exactly one token; existing credentials were not changed")
|
||||
directory = Path(directory)
|
||||
secrets = directory / "secrets"
|
||||
secrets.mkdir(mode=0o700, exist_ok=True)
|
||||
secrets.chmod(0o700)
|
||||
path = secrets / "ntfy-token.txt"
|
||||
path = Path(path)
|
||||
path.parent.mkdir(mode=0o700, exist_ok=True)
|
||||
path.parent.chmod(0o700)
|
||||
if path.is_symlink() or (path.exists() and path.stat().st_size != 0):
|
||||
raise ValueError("Token file already contains credentials; refusing to overwrite")
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w") as file:
|
||||
file.write(tokens[0] + "\n")
|
||||
# Compose's file secret is a bind mount. Container uid 10001 needs read access;
|
||||
# the private host parent directory restricts access by other host users.
|
||||
path.chmod(0o444)
|
||||
path.chmod(mode)
|
||||
return path
|
||||
|
||||
|
||||
def bootstrap_accounts(run, token_path, mode=0o444, log=print):
|
||||
"""Create the pipeline-publisher and phone-reader accounts on an ntfy server.
|
||||
|
||||
``run(args, capture=False, interactive=False)`` executes the ntfy CLI (directly
|
||||
or through Compose) and returns the completed process. Existing accounts and an
|
||||
existing token file are preserved on reinstall.
|
||||
"""
|
||||
token_path = Path(token_path)
|
||||
if token_path.is_symlink():
|
||||
raise ValueError("Refusing a symlink for the publishing token")
|
||||
if token_path.exists() and token_path.stat().st_size:
|
||||
log("Keeping existing ntfy publishing token.")
|
||||
else:
|
||||
log("Set a password for ntfy's pipeline-publisher account (not a model-provider account).")
|
||||
run(["user", "add", "--ignore-exists", "pipeline-publisher"], interactive=True)
|
||||
run(["access", "pipeline-publisher", "pipelines", "write-only"])
|
||||
result = run(["token", "add", "pipeline-publisher"], capture=True)
|
||||
save_token(result.stdout + result.stderr, token_path, mode)
|
||||
# --ignore-exists preserves the phone account's password on reinstall.
|
||||
log("Set the phone-reader password if this is a new ntfy account; existing accounts are preserved.")
|
||||
run(["user", "add", "--ignore-exists", "phone-reader"], interactive=True)
|
||||
run(["access", "phone-reader", "pipelines", "read-only"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
save_token(sys.stdin.read(), Path(__file__).resolve().parents[1])
|
||||
save_token(sys.stdin.read(), Path(__file__).resolve().parents[1] / "secrets/ntfy-token.txt", 0o444)
|
||||
except ValueError as error:
|
||||
sys.exit(str(error))
|
||||
print("Saved ntfy publishing token; restart pipelines if it was already running.")
|
||||
|
||||
+8
-36
@@ -105,21 +105,13 @@ class Setup:
|
||||
print("Use this address on your phone on the same trusted LAN; do not forward this HTTP port to the Internet.")
|
||||
|
||||
def bootstrap_ntfy(self):
|
||||
token = self.directory / "secrets/ntfy-token.txt"
|
||||
if token.is_symlink():
|
||||
raise ValueError("Refusing a symlink for the publishing token")
|
||||
if token.exists() and token.stat().st_size:
|
||||
print("Keeping existing ntfy publishing token.")
|
||||
else:
|
||||
print("Set a password for ntfy's pipeline-publisher account (not a model-provider account).")
|
||||
self.compose("exec", "ntfy", "ntfy", "user", "add", "--ignore-exists", "pipeline-publisher")
|
||||
self.compose("exec", "-T", "ntfy", "ntfy", "access", "pipeline-publisher", "pipelines", "write-only")
|
||||
result = self.compose("exec", "-T", "ntfy", "ntfy", "token", "add", "pipeline-publisher", capture=True)
|
||||
tokens.save_token(result.stdout + result.stderr, self.directory)
|
||||
# --ignore-exists preserves the phone account's password on re-install.
|
||||
print("Set the phone-reader password if this is a new ntfy account; existing accounts are preserved.")
|
||||
self.compose("exec", "ntfy", "ntfy", "user", "add", "--ignore-exists", "phone-reader")
|
||||
self.compose("exec", "-T", "ntfy", "ntfy", "access", "phone-reader", "pipelines", "read-only")
|
||||
def run_ntfy(args, capture=False, interactive=False):
|
||||
# -T is required for the non-interactive calls; the interactive
|
||||
# `user add` password prompts need the allocated TTY.
|
||||
exec_args = ["exec"] + ([] if interactive else ["-T"]) + ["ntfy", "ntfy", *args]
|
||||
return self.compose(*exec_args, capture=capture)
|
||||
|
||||
tokens.bootstrap_accounts(run_ntfy, self.directory / "secrets/ntfy-token.txt", 0o444)
|
||||
|
||||
def authenticate(self, relogin=False):
|
||||
def connected():
|
||||
@@ -554,27 +546,7 @@ printf 'header = "Authorization: Bearer %s"\\n' "$(cat "$PIPELINES_NTFY_TOKEN_FI
|
||||
print("Verified: host client reaches the ntfy-enabled Docker server at " + endpoint)
|
||||
|
||||
def configure_client(self, endpoint):
|
||||
root = Path(os.environ.get("XDG_CONFIG_HOME") or Path.home()/".config")
|
||||
directory = root/"pipelines"
|
||||
path = directory/"client.json"
|
||||
if path.is_symlink():
|
||||
raise ValueError("Refusing to modify symlinked client.json; use the printed --url explicitly")
|
||||
existing = json.loads(path.read_text()) if path.exists() else {}
|
||||
if not isinstance(existing, dict):
|
||||
raise ValueError("client.json must contain an object; existing configuration was not replaced")
|
||||
existing["server_url"] = endpoint
|
||||
directory.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
temporary = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(mode="w", dir=directory, delete=False) as file:
|
||||
temporary = Path(file.name)
|
||||
os.chmod(temporary, 0o600)
|
||||
json.dump(existing, file, indent=2)
|
||||
file.write("\n")
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
if temporary is not None:
|
||||
temporary.unlink(missing_ok=True)
|
||||
path = configure.configure_client(endpoint)
|
||||
print(f"Saved Docker client default: {endpoint} ({path}). Restart the TUI.")
|
||||
override = os.environ.get("PIPELINES_URL", "")
|
||||
if override and override != endpoint:
|
||||
|
||||
@@ -0,0 +1,465 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Install Pipelines natively on the host (macOS and Linux).
|
||||
|
||||
The installer only detects the tools it needs (go, python3, git, curl and a C
|
||||
compiler for ntfy's SQLite driver); it never installs or upgrades them, and it
|
||||
never installs, upgrades or downloads OpenCode or Pi. It builds the server, CLI
|
||||
and ntfy from source into a per-user prefix, generates an ntfy configuration that
|
||||
mirrors compose.yaml, creates the publishing/phone accounts and writes the
|
||||
manifest scripts/manage.py loads. ntfy is compiled from its pinned Go module on
|
||||
every host because upstream publishes no macOS server binary. It reuses existing
|
||||
host OpenCode configuration; model selection stays at runtime and is never
|
||||
invented here.
|
||||
"""
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from zoneinfo import ZoneInfoNotFoundError
|
||||
|
||||
|
||||
def module(name, file):
|
||||
spec = importlib.util.spec_from_file_location(name, Path(__file__).with_name(file))
|
||||
result = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(result)
|
||||
return result
|
||||
|
||||
|
||||
configure = module("configure_compose", "configure-compose.py")
|
||||
lan = module("lan_address", "lan-address.py")
|
||||
release_lookup = module("resolve_releases", "resolve-releases.py")
|
||||
tokens = module("save_ntfy_token", "save-ntfy-token.py")
|
||||
|
||||
DEFAULT_PREFIX = Path.home() / ".local/share/pipelines"
|
||||
MARKER = "pipelines-install.json"
|
||||
LABEL = "net.a-wan.pipelines"
|
||||
REQUIRED_TOOLS = ("go", "python3", "git", "curl")
|
||||
# ntfy's SQLite driver (mattn/go-sqlite3) is cgo, so a C compiler is required to
|
||||
# build the server. Upstream's macOS release asset is client-only, so both
|
||||
# platforms compile ntfy from the pinned Go module.
|
||||
C_COMPILERS = ("cc", "clang", "gcc")
|
||||
NTFY_BUILD_TAGS = "sqlite_omit_load_extension,osusergo,netgo"
|
||||
DEFAULT_NTFY_PORT = 8090
|
||||
DEFAULT_SERVER_PORT = 8080
|
||||
|
||||
|
||||
class Setup:
|
||||
def __init__(self, directory, prefix=None, runner=subprocess.run, resolver=None, environ=None):
|
||||
self.directory = Path(directory)
|
||||
self.prefix_argument = prefix
|
||||
self.runner = runner
|
||||
self.resolver = resolver or release_lookup.resolve_ntfy
|
||||
self.environ = environ if environ is not None else os.environ
|
||||
|
||||
# --- detection -------------------------------------------------------
|
||||
|
||||
def require_compiler(self):
|
||||
"""ntfy is built locally, so a C toolchain must already be present."""
|
||||
for compiler in C_COMPILERS:
|
||||
path = shutil.which(compiler)
|
||||
if path:
|
||||
return str(Path(path).resolve())
|
||||
raise ValueError(
|
||||
"No C compiler found (tried " + ", ".join(C_COMPILERS)
|
||||
+ "). ntfy's SQLite driver is cgo, so the server is compiled locally; "
|
||||
"install a C toolchain (Xcode Command Line Tools on macOS, gcc or clang on Linux) and rerun.")
|
||||
|
||||
def require_tools(self):
|
||||
found, missing = {}, []
|
||||
for tool in REQUIRED_TOOLS:
|
||||
path = shutil.which(tool)
|
||||
if path:
|
||||
found[tool] = str(Path(path).resolve())
|
||||
else:
|
||||
missing.append(tool)
|
||||
if missing:
|
||||
raise ValueError(
|
||||
"Missing required tool(s): " + ", ".join(missing)
|
||||
+ ". Install them yourself and rerun; this installer never installs tools or uses a package manager.")
|
||||
return found["python3"]
|
||||
|
||||
def harness(self):
|
||||
"""Detect existing harnesses only; never install, upgrade or download them."""
|
||||
opencode = shutil.which("opencode")
|
||||
if not opencode:
|
||||
raise ValueError(
|
||||
"OpenCode was not found on PATH. Install OpenCode yourself (this installer never installs, "
|
||||
"upgrades or downloads it) and rerun; model selection happens in your existing OpenCode configuration.")
|
||||
pi = shutil.which("pi")
|
||||
return {"opencode": str(Path(opencode).resolve()), "pi": str(Path(pi).resolve()) if pi else ""}
|
||||
|
||||
def resolve_prefix(self):
|
||||
default = Path.home() / ".local/share/pipelines"
|
||||
raw = self.prefix_argument or self.environ.get("PIPELINES_PREFIX") or default
|
||||
return Path(raw).expanduser().resolve()
|
||||
|
||||
# --- prefix layout ---------------------------------------------------
|
||||
|
||||
def prepare_layout(self, prefix):
|
||||
prefix.mkdir(parents=True, exist_ok=True)
|
||||
for name in ("bin", "lib", "data", "logs", "secrets"):
|
||||
(prefix / name).mkdir(exist_ok=True)
|
||||
(prefix / "secrets").chmod(0o700)
|
||||
(prefix / "data").chmod(0o700)
|
||||
source = Path(__file__).with_name("manage.py")
|
||||
target = prefix / "lib/manage.py"
|
||||
if target.is_symlink():
|
||||
raise ValueError("Refusing to replace a symlinked lib/manage.py")
|
||||
shutil.copyfile(source, target)
|
||||
os.chmod(target, 0o700)
|
||||
return target
|
||||
|
||||
# --- build and skill -------------------------------------------------
|
||||
|
||||
def build(self, prefix):
|
||||
for command in ("pipelines-server", "pipelines"):
|
||||
result = self.runner(
|
||||
["go", "build", "-trimpath", "-o", str(prefix / "bin" / command), "./cmd/" + command],
|
||||
cwd=str(self.directory))
|
||||
if result.returncode:
|
||||
raise ValueError(f"go build failed for ./cmd/{command}; the native install is incomplete")
|
||||
|
||||
def install_skill(self, prefix):
|
||||
result = self.runner([str(prefix / "bin/pipelines"), "skill", "install"], cwd=str(self.directory))
|
||||
if result.returncode:
|
||||
raise ValueError("pipelines skill install failed; the native install is incomplete")
|
||||
|
||||
# --- ntfy server build and configuration ----------------------------
|
||||
|
||||
def resolve_ntfy(self):
|
||||
try:
|
||||
return self.resolver()
|
||||
except release_lookup.ReleaseError as error:
|
||||
raise ValueError(
|
||||
f"Could not resolve the ntfy release ({error}); refusing to build an unverified version") from None
|
||||
|
||||
def build_ntfy(self, prefix, resolution):
|
||||
"""Compile the ntfy server from the pinned Go module on this host.
|
||||
|
||||
Upstream publishes no macOS server binary (its darwin asset is built with
|
||||
``-tags noserver``), so every platform builds here. The module is fetched
|
||||
through the Go toolchain, which verifies it against the public checksum
|
||||
database, and cgo is required for ntfy's SQLite driver.
|
||||
"""
|
||||
path = prefix / "bin/ntfy"
|
||||
if path.is_symlink():
|
||||
raise ValueError("Refusing to replace a symlinked bin/ntfy")
|
||||
version = resolution.get("tag") or resolution.get("version")
|
||||
module = resolution.get("module") or release_lookup.NTFY_MODULE
|
||||
# A previously built binary for the same resolved version is reused.
|
||||
if path.is_file() and self._existing_ntfy_version(prefix) == version:
|
||||
return path
|
||||
self.require_compiler()
|
||||
pinned = f"{module}@{version}"
|
||||
result = self.runner(["go", "mod", "download", pinned], cwd=str(self.directory))
|
||||
if result.returncode:
|
||||
raise ValueError(f"go mod download {pinned} failed; the native install is incomplete")
|
||||
cache = self.runner(["go", "env", "GOMODCACHE"], cwd=str(self.directory),
|
||||
capture_output=True, text=True)
|
||||
if cache.returncode or not (cache.stdout or "").strip():
|
||||
raise ValueError("Could not locate the Go module cache to build ntfy from")
|
||||
source = Path(cache.stdout.strip()) / f"{module}@{version}"
|
||||
if not source.is_dir():
|
||||
raise ValueError(f"the Go module cache has no source for {pinned}")
|
||||
with tempfile.TemporaryDirectory(prefix="pipelines-ntfy-") as temporary:
|
||||
checkout = Path(temporary) / "src"
|
||||
shutil.copytree(source, checkout, symlinks=True)
|
||||
# The module cache is read-only and carries no built web assets; the
|
||||
# upstream Makefile creates the same placeholders for local builds.
|
||||
self.runner(["chmod", "-R", "u+w", str(checkout)])
|
||||
(checkout / "server/docs").mkdir(parents=True, exist_ok=True)
|
||||
(checkout / "server/site").mkdir(parents=True, exist_ok=True)
|
||||
(checkout / "server/docs/index.html").touch()
|
||||
(checkout / "server/site/app.html").touch()
|
||||
built = Path(temporary) / "ntfy"
|
||||
env = dict(self.environ)
|
||||
env["CGO_ENABLED"] = "1"
|
||||
build = self.runner(
|
||||
["go", "build", "-trimpath", "-tags", NTFY_BUILD_TAGS,
|
||||
"-ldflags", f"-s -w -X main.version={version}", "-o", str(built), "."],
|
||||
cwd=str(checkout), env=env)
|
||||
if build.returncode:
|
||||
raise ValueError(
|
||||
"ntfy server build failed; a working C compiler is required for its SQLite driver")
|
||||
if not built.is_file():
|
||||
raise ValueError("ntfy server build produced no binary")
|
||||
os.chmod(built, 0o755)
|
||||
os.replace(built, path)
|
||||
return path
|
||||
|
||||
def _existing_ntfy_version(self, prefix):
|
||||
manifest = prefix / MARKER
|
||||
if manifest.is_symlink() or not manifest.is_file():
|
||||
return None
|
||||
try:
|
||||
data = json.loads(manifest.read_text())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return (data.get("ntfy") or {}).get("version") if isinstance(data, dict) else None
|
||||
|
||||
def write_ntfy_config(self, prefix, base_url, ntfy_port):
|
||||
directory = prefix / "data/ntfy"
|
||||
directory.mkdir(mode=0o700, exist_ok=True)
|
||||
directory.chmod(0o700)
|
||||
path = directory / "server.yml"
|
||||
if path.is_symlink():
|
||||
raise ValueError("Refusing to replace a symlinked ntfy configuration")
|
||||
text = (
|
||||
f'base-url: "{base_url}"\n'
|
||||
f'listen-http: ":{ntfy_port}"\n'
|
||||
f'cache-file: "{directory}/cache.db"\n'
|
||||
'cache-duration: "168h"\n'
|
||||
f'auth-file: "{directory}/auth.db"\n'
|
||||
'auth-default-access: "deny-all"\n'
|
||||
'enable-login: true\n'
|
||||
f'attachment-cache-dir: "{directory}/attachments"\n'
|
||||
'attachment-expiry-duration: "168h"\n'
|
||||
'attachment-total-size-limit: "1G"\n'
|
||||
'attachment-file-size-limit: "1M"\n'
|
||||
)
|
||||
self._write_atomic(path, text.encode(), 0o600)
|
||||
return path
|
||||
|
||||
# --- ntfy users and publishing token --------------------------------
|
||||
|
||||
def ntfy(self, prefix, config_path, args, capture=False):
|
||||
# The account subcommands locate the server configuration through
|
||||
# NTFY_CONFIG_FILE; --config is a `serve`-only flag.
|
||||
env = dict(self.environ)
|
||||
env["NTFY_CONFIG_FILE"] = str(config_path)
|
||||
result = self.runner([str(prefix / "bin/ntfy"), *args], capture_output=capture, text=True, env=env)
|
||||
if result.returncode:
|
||||
step = " ".join(args[:2])
|
||||
raise ValueError(f"ntfy {step} failed; the native install is incomplete")
|
||||
return result
|
||||
|
||||
def bootstrap_ntfy(self, prefix, config_path):
|
||||
def run_ntfy(args, capture=False, interactive=False):
|
||||
return self.ntfy(prefix, config_path, list(args), capture=capture)
|
||||
|
||||
tokens.bootstrap_accounts(run_ntfy, prefix / "secrets/publisher-token", 0o600)
|
||||
|
||||
# --- Docker migration ------------------------------------------------
|
||||
|
||||
def docker_volumes(self):
|
||||
try:
|
||||
result = self.runner(["docker", "volume", "ls", "--format", "{{.Name}}"],
|
||||
capture_output=True, text=True)
|
||||
except FileNotFoundError:
|
||||
# No Docker CLI at all: there is no Docker installation to migrate.
|
||||
return ()
|
||||
except OSError as error:
|
||||
raise ValueError(
|
||||
f"Could not inspect Docker volumes for migration ({error}); refusing to start empty") from None
|
||||
if result.returncode:
|
||||
# A present-but-unusable Docker daemon must not silently skip migration:
|
||||
# Docker data may still exist, so fail instead of starting empty.
|
||||
raise ValueError(
|
||||
"Could not inspect Docker volumes for migration; refusing a possibly empty start")
|
||||
names = set(result.stdout.split())
|
||||
return tuple(name for name in ("pipelines_pipelines-data", "pipelines_ntfy-data") if name in names)
|
||||
|
||||
def _migration_image(self):
|
||||
result = self.runner(["docker", "image", "ls", "--format", "{{.Repository}}:{{.Tag}}"],
|
||||
capture_output=True, text=True)
|
||||
if result.returncode:
|
||||
return None
|
||||
for line in result.stdout.splitlines():
|
||||
name = line.strip()
|
||||
repository = name.rsplit(":", 1)[0]
|
||||
if repository == "pipelines" or repository.endswith("/pipelines") or repository.endswith("-pipelines"):
|
||||
return name
|
||||
return None
|
||||
|
||||
def _docker_copy_out(self, image, volume, inner, destination, replace):
|
||||
if destination.exists() and not replace:
|
||||
print(f"Keeping existing native {destination.name}; not overwriting it during migration.")
|
||||
return False
|
||||
name = f"pipelines-migrate-{os.getpid()}"
|
||||
created = self.runner(["docker", "create", "--name", name, "--volume", f"{volume}:/mnt:ro", image],
|
||||
capture_output=True, text=True)
|
||||
if created.returncode:
|
||||
raise ValueError(f"Could not open Docker volume {volume} for migration; install aborted")
|
||||
try:
|
||||
result = self.runner(["docker", "cp", f"{name}:/mnt/{inner}", str(destination)],
|
||||
capture_output=True, text=True)
|
||||
if result.returncode:
|
||||
raise ValueError(f"Could not copy {inner} from Docker volume {volume}; install aborted")
|
||||
# The live server keeps recent commits in the SQLite write-ahead log,
|
||||
# so copy the -wal sidecar too or the newest definitions and runs are
|
||||
# lost. SQLite recovers it on next open and rebuilds -shm itself.
|
||||
for suffix in ("-wal",):
|
||||
sidecar = Path(str(destination) + suffix)
|
||||
sidecar.unlink(missing_ok=True)
|
||||
copy = self.runner(["docker", "cp", f"{name}:/mnt/{inner}{suffix}", str(sidecar)],
|
||||
capture_output=True, text=True)
|
||||
if copy.returncode or not sidecar.exists() or sidecar.stat().st_size == 0:
|
||||
sidecar.unlink(missing_ok=True)
|
||||
finally:
|
||||
self.runner(["docker", "rm", "-f", name], capture_output=True, text=True, check=False)
|
||||
return True
|
||||
|
||||
def migrate_docker(self, prefix, replace=False):
|
||||
volumes = self.docker_volumes()
|
||||
if not volumes:
|
||||
return False
|
||||
image = self._migration_image()
|
||||
if not image:
|
||||
raise ValueError(
|
||||
"Docker volumes exist but no local Pipelines image is available to read them; "
|
||||
"refusing to pull an image. Build the Docker image or rerun without migration.")
|
||||
(prefix / "data").mkdir(exist_ok=True)
|
||||
if "pipelines_pipelines-data" in volumes:
|
||||
self._docker_copy_out(image, "pipelines_pipelines-data", "pipelines.db",
|
||||
prefix / "data/pipelines.db", replace)
|
||||
if "pipelines_ntfy-data" in volumes:
|
||||
directory = prefix / "data/ntfy"
|
||||
directory.mkdir(mode=0o700, exist_ok=True)
|
||||
self._docker_copy_out(image, "pipelines_ntfy-data", "auth.db",
|
||||
directory / "auth.db", replace)
|
||||
print("Existing Docker data was migrated where it did not already exist.")
|
||||
return True
|
||||
|
||||
# --- manifest --------------------------------------------------------
|
||||
|
||||
def write_manifest(self, prefix, values):
|
||||
path = prefix / MARKER
|
||||
if path.is_symlink():
|
||||
raise ValueError("Refusing to replace a symlinked installation manifest")
|
||||
self._write_atomic(path, (json.dumps(values, indent=2, sort_keys=True) + "\n").encode(), 0o600)
|
||||
return path
|
||||
|
||||
def _write_atomic(self, path, data, mode):
|
||||
handle, temporary = tempfile.mkstemp(dir=str(path.parent), prefix=".pipelines-")
|
||||
try:
|
||||
os.chmod(temporary, mode)
|
||||
with os.fdopen(handle, "wb") as file:
|
||||
file.write(data)
|
||||
if path.is_symlink():
|
||||
raise ValueError(f"Refusing to replace a symlinked file {path}")
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
if os.path.exists(temporary):
|
||||
os.unlink(temporary)
|
||||
|
||||
# --- run -------------------------------------------------------------
|
||||
|
||||
def run(self, args):
|
||||
python = self.require_tools()
|
||||
if args.prefix:
|
||||
self.prefix_argument = args.prefix
|
||||
if not 1024 <= args.port <= 65535:
|
||||
raise ValueError("Invalid server port; choose a value between 1024 and 65535")
|
||||
if not 1024 <= args.ntfy_port <= 65535:
|
||||
raise ValueError("Invalid ntfy port; choose a value between 1024 and 65535")
|
||||
if args.ntfy_port == args.port:
|
||||
raise ValueError("The server and ntfy ports must differ")
|
||||
prefix = self.resolve_prefix()
|
||||
self.prepare_layout(prefix)
|
||||
harness = self.harness()
|
||||
# Reuse an already built ntfy binary and skip the release lookup, so a
|
||||
# reinstall with the same version makes no network requests at all.
|
||||
installed = self._existing_ntfy_version(prefix)
|
||||
if (prefix / "bin/ntfy").is_file() and installed:
|
||||
resolution = {"module": release_lookup.NTFY_MODULE, "version": installed, "tag": installed}
|
||||
else:
|
||||
resolution = self.resolve_ntfy()
|
||||
self.build(prefix)
|
||||
self.install_skill(prefix)
|
||||
base_url = args.ntfy_url or f"http://{lan.detect_lan()}:{args.ntfy_port}"
|
||||
config_path = self.write_ntfy_config(prefix, base_url, args.ntfy_port)
|
||||
self.build_ntfy(prefix, resolution)
|
||||
# Migrate existing Docker data before creating native ntfy accounts so a
|
||||
# migrated auth store (the phone's credentials) is preserved rather than
|
||||
# shadowed by a freshly bootstrapped one.
|
||||
self.migrate_docker(prefix, args.replace_data)
|
||||
self.bootstrap_ntfy(prefix, config_path)
|
||||
timezone = args.timezone
|
||||
if not timezone:
|
||||
try:
|
||||
timezone = configure.host_timezone()
|
||||
except (ValueError, ZoneInfoNotFoundError):
|
||||
timezone = "UTC"
|
||||
opencode_env = {}
|
||||
for key in ("XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME", "XDG_CACHE_HOME",
|
||||
"OPENCODE_SERVER", "OPENCODE_DB"):
|
||||
if self.environ.get(key):
|
||||
opencode_env[key] = self.environ[key]
|
||||
if args.model:
|
||||
opencode_env["PIPELINES_DEFAULT_MODEL"] = args.model
|
||||
path_value = str(prefix / "bin") + os.pathsep + self.environ.get("PATH", "")
|
||||
manifest = {
|
||||
"format": 1,
|
||||
"prefix": str(prefix),
|
||||
"python": python,
|
||||
"server_port": args.port,
|
||||
"ntfy_port": args.ntfy_port,
|
||||
"opencode": harness["opencode"],
|
||||
"pi": harness["pi"],
|
||||
"opencode_env": opencode_env,
|
||||
"path": path_value,
|
||||
"topic": "pipelines",
|
||||
"public_url": base_url,
|
||||
"username": "phone-reader",
|
||||
"workspace": str(Path.home()),
|
||||
"launcher": str(prefix / "bin/pipelines"),
|
||||
"registration": None,
|
||||
"label": LABEL,
|
||||
"docker": "docker",
|
||||
"project": "pipelines",
|
||||
"timezone": timezone,
|
||||
"ntfy": {"mode": "native", "bin": str(prefix / "bin/ntfy"),
|
||||
"config": str(config_path), "base_url": base_url,
|
||||
"version": resolution.get("tag") or resolution.get("version")},
|
||||
}
|
||||
self.write_manifest(prefix, manifest)
|
||||
# Save the address the host CLI/TUI dials by default, so a plain
|
||||
# `pipelines` (or its TUI) reaches this installation instead of a stale
|
||||
# Docker address.
|
||||
endpoint = f"ws://127.0.0.1:{args.port}/connect"
|
||||
client_path = configure.configure_client(endpoint, self.environ)
|
||||
print(f"Saved native client default: {endpoint} ({client_path}). Restart the TUI.")
|
||||
override = self.environ.get("PIPELINES_URL", "")
|
||||
if override and override != endpoint:
|
||||
print("WARNING: PIPELINES_URL overrides the saved native address; unset it or pass --url explicitly.")
|
||||
print(f"Native Pipelines installed at {prefix}."
|
||||
f" ntfy {resolution.get('tag') or resolution.get('version')} was built from source.")
|
||||
# Parity with the Docker installer: leave the server and ntfy running.
|
||||
started = self.runner([python, str(prefix / "lib/manage.py"), "start", "--prefix", str(prefix)])
|
||||
if started.returncode:
|
||||
raise ValueError(
|
||||
"The native services were installed but could not be started; run "
|
||||
f"'{prefix}/bin/pipelines start' to see the error. Check that ports "
|
||||
f"{args.port} and {args.ntfy_port} are free (a running Docker installation uses them).")
|
||||
print(f"ntfy is configured for {base_url} and both services are running.")
|
||||
print("Create the phone account password if it is a new account, then subscribe to the pipelines topic.")
|
||||
|
||||
|
||||
def arguments(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--prefix", type=Path, help="Installation prefix (default: ~/.local/share/pipelines)")
|
||||
parser.add_argument("--ntfy-url", help="Override the auto-detected LAN ntfy address")
|
||||
parser.add_argument("--timezone", help="Override the detected timezone (fallback: UTC)")
|
||||
parser.add_argument("--port", type=int, default=DEFAULT_SERVER_PORT, help="Pipelines server port")
|
||||
parser.add_argument("--ntfy-port", type=int, default=DEFAULT_NTFY_PORT, help="ntfy listen port")
|
||||
parser.add_argument("--model", help="Record an explicit default model; otherwise host OpenCode configuration is reused")
|
||||
parser.add_argument("--replace-data", action="store_true",
|
||||
help="Allow Docker migration to replace an existing native database/auth store")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
Setup(Path(__file__).resolve().parents[1]).run(arguments())
|
||||
except KeyboardInterrupt:
|
||||
print("\nNative setup canceled; existing configuration and data were retained.", file=sys.stderr)
|
||||
sys.exit(130)
|
||||
except (ValueError, OSError, json.JSONDecodeError, subprocess.SubprocessError,
|
||||
ZoneInfoNotFoundError) as error:
|
||||
sys.exit(f"Native setup incomplete: {error}")
|
||||
@@ -1,4 +1,5 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
@@ -52,6 +53,35 @@ class ConfigureTests(unittest.TestCase):
|
||||
with tempfile.TemporaryDirectory() as directory, self.assertRaises(ValueError):
|
||||
configure.configure(directory, "https://example.com", "0.0.0.0\nEVIL=yes", "Europe/London")
|
||||
|
||||
def test_client_default_is_saved_privately_and_preserves_other_keys(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
environ = {"XDG_CONFIG_HOME": directory}
|
||||
path = configure.configure_client("ws://127.0.0.1:8080/connect", environ)
|
||||
self.assertEqual(path, Path(directory) / "pipelines/client.json")
|
||||
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
|
||||
self.assertEqual(json.loads(path.read_text())["server_url"], "ws://127.0.0.1:8080/connect")
|
||||
# Unrelated keys survive a later reinstall with a different address.
|
||||
path.write_text(json.dumps({"server_url": "ws://old/connect", "theme": "dark"}))
|
||||
configure.configure_client("ws://127.0.0.1:9090/connect", environ)
|
||||
saved = json.loads(path.read_text())
|
||||
self.assertEqual(saved["server_url"], "ws://127.0.0.1:9090/connect")
|
||||
self.assertEqual(saved["theme"], "dark")
|
||||
|
||||
def test_client_default_refuses_symlink_and_non_object(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
environ = {"XDG_CONFIG_HOME": directory}
|
||||
target = Path(directory) / "real.json"
|
||||
target.write_text("{}")
|
||||
directory_path = Path(directory) / "pipelines"
|
||||
directory_path.mkdir()
|
||||
(directory_path / "client.json").symlink_to(target)
|
||||
with self.assertRaisesRegex(ValueError, "symlinked client.json"):
|
||||
configure.configure_client("ws://127.0.0.1:8080/connect", environ)
|
||||
(directory_path / "client.json").unlink()
|
||||
(directory_path / "client.json").write_text("[]")
|
||||
with self.assertRaisesRegex(ValueError, "must contain an object"):
|
||||
configure.configure_client("ws://127.0.0.1:8080/connect", environ)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -7,13 +7,19 @@ import manage
|
||||
|
||||
|
||||
class InstallationTests(unittest.TestCase):
|
||||
def test_docker_installer_is_available_through_main_entrypoint(self):
|
||||
def test_native_installer_is_the_default_entrypoint(self):
|
||||
result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),"--help"],capture_output=True,text=True)
|
||||
self.assertEqual(result.returncode,0)
|
||||
self.assertIn("--prefix",result.stdout)
|
||||
self.assertIn("--ntfy-url",result.stdout)
|
||||
|
||||
def test_docker_installer_is_available_explicitly(self):
|
||||
result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),"--docker","--help"],capture_output=True,text=True)
|
||||
self.assertEqual(result.returncode,0)
|
||||
self.assertIn("--ntfy-url",result.stdout)
|
||||
self.assertIn("--relogin",result.stdout)
|
||||
|
||||
def test_default_installer_delegates_to_docker_without_terminal(self):
|
||||
def dispatch(self,*args):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory)
|
||||
capture=root/"args"
|
||||
@@ -21,27 +27,21 @@ class InstallationTests(unittest.TestCase):
|
||||
executable.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > "'+str(capture)+'"\n')
|
||||
executable.chmod(0o700)
|
||||
env=dict(os.environ,PATH=str(root)+os.pathsep+os.environ["PATH"])
|
||||
subprocess.run([str(Path(__file__).resolve().parents[1]/"install")],env=env,stdin=subprocess.DEVNULL,check=True)
|
||||
self.assertEqual(capture.read_text().splitlines(),["scripts/setup-docker.py"])
|
||||
subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),*args],env=env,stdin=subprocess.DEVNULL,check=True)
|
||||
return capture.read_text().splitlines()
|
||||
|
||||
def test_default_installer_forwards_docker_options(self):
|
||||
result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install.sh"),"--help"],capture_output=True,text=True)
|
||||
self.assertEqual(result.returncode,0)
|
||||
self.assertIn("--ntfy-url",result.stdout)
|
||||
def test_default_installer_delegates_to_native_without_terminal(self):
|
||||
self.assertEqual(self.dispatch(),["scripts/setup-native.py"])
|
||||
|
||||
def test_native_installer_is_rejected_without_running_go(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory)
|
||||
capture=root/"args"
|
||||
executable=root/"go"
|
||||
executable.write_text('#!/bin/sh\nprintf "%s\\n" "$GOBIN" "$@" >> "'+str(capture)+'"\n')
|
||||
executable.chmod(0o700)
|
||||
env=dict(os.environ,PATH=str(root)+os.pathsep+os.environ["PATH"],GOBIN=str(root/"bin"))
|
||||
result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),"--native"],env=env,capture_output=True,text=True)
|
||||
self.assertEqual(result.returncode,2)
|
||||
self.assertIn("Run ./install",result.stderr)
|
||||
self.assertFalse(capture.exists())
|
||||
self.assertFalse((root/"bin").exists())
|
||||
def test_explicit_modes_dispatch_to_the_right_installer(self):
|
||||
self.assertEqual(self.dispatch("--native"),["scripts/setup-native.py"])
|
||||
self.assertEqual(self.dispatch("native"),["scripts/setup-native.py"])
|
||||
self.assertEqual(self.dispatch("--docker"),["scripts/setup-docker.py"])
|
||||
self.assertEqual(self.dispatch("docker"),["scripts/setup-docker.py"])
|
||||
|
||||
def test_docker_mode_forwards_docker_options(self):
|
||||
self.assertEqual(self.dispatch("--docker","--ntfy-url","http://192.168.1.20:8090"),
|
||||
["scripts/setup-docker.py","--ntfy-url","http://192.168.1.20:8090"])
|
||||
|
||||
def test_standard_installer_rejects_legacy_provisioning_flags(self):
|
||||
result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install.sh"),"--system-command"],capture_output=True,text=True)
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
spec = importlib.util.spec_from_file_location("manage_native", Path(__file__).with_name("manage.py"))
|
||||
manage = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(manage)
|
||||
|
||||
|
||||
class FakeProcess:
|
||||
def __init__(self, pid=4321):
|
||||
self.pid = pid
|
||||
|
||||
def poll(self):
|
||||
return None
|
||||
|
||||
|
||||
class ManageNativeTests(unittest.TestCase):
|
||||
def config(self, root, mode="native"):
|
||||
return {
|
||||
"format": 1, "prefix": str(root), "python": "python3", "server_port": 8080,
|
||||
"ntfy_port": 8090, "opencode": "opencode", "opencode_env": {}, "path": "",
|
||||
"topic": "pipelines", "public_url": "http://192.168.1.20:8090", "username": "phone-reader",
|
||||
"workspace": str(root), "launcher": str(root / "bin/pipelines"), "registration": None,
|
||||
"label": "net.a-wan.pipelines", "docker": "docker", "project": "pipelines", "timezone": "UTC",
|
||||
"ntfy": {"mode": mode, "bin": str(root / "bin/ntfy"),
|
||||
"config": str(root / "data/ntfy/server.yml"), "base_url": "http://192.168.1.20:8090"},
|
||||
}
|
||||
|
||||
def test_mode_selection_defaults_to_docker(self):
|
||||
self.assertEqual(manage.ntfy_mode({}), "docker")
|
||||
self.assertEqual(manage.ntfy_mode({"ntfy": {}}), "docker")
|
||||
self.assertEqual(manage.ntfy_mode({"ntfy": {"mode": "native"}}), "native")
|
||||
self.assertEqual(manage.ntfy_health_url({"ntfy_port": 8090}), "http://127.0.0.1:8090/v1/health")
|
||||
|
||||
def test_ntfy_start_launches_binary_with_generated_config(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
(root / "bin").mkdir()
|
||||
(root / "logs").mkdir()
|
||||
config = self.config(root)
|
||||
with patch.object(manage.subprocess, "Popen", return_value=FakeProcess(pid=4321)) as popen:
|
||||
child = manage.ntfy_start(config)
|
||||
self.assertEqual(popen.call_args[0][0], [config["ntfy"]["bin"], "serve", "--config", config["ntfy"]["config"]])
|
||||
self.assertEqual(child.pid, 4321)
|
||||
self.assertEqual((root / "ntfy.pid").read_text(), "4321")
|
||||
|
||||
def test_ntfy_stop_terminates_managed_process_and_removes_pid(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
(root / "bin").mkdir()
|
||||
(root / "bin/ntfy").write_text("")
|
||||
(root / "ntfy.pid").write_text(str(os.getpid()))
|
||||
config = self.config(root)
|
||||
killed = []
|
||||
|
||||
def fake_kill(pid, sig):
|
||||
if sig == 0:
|
||||
raise ProcessLookupError
|
||||
killed.append((pid, sig))
|
||||
|
||||
with patch.object(manage.subprocess, "check_output",
|
||||
return_value=str(root / "bin/ntfy") + " serve --config x"), \
|
||||
patch.object(manage.os, "kill", side_effect=fake_kill), \
|
||||
patch.object(manage, "compose", side_effect=AssertionError("compose used for native ntfy")):
|
||||
manage.ntfy_stop(config)
|
||||
self.assertIn((os.getpid(), signal.SIGTERM), killed)
|
||||
self.assertFalse((root / "ntfy.pid").exists())
|
||||
|
||||
def test_native_stop_never_calls_compose(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
config = self.config(Path(directory))
|
||||
with patch.object(manage, "registration", return_value=None), \
|
||||
patch.object(manage, "managed_pid", return_value=None), \
|
||||
patch.object(manage, "ntfy_stop") as stop, \
|
||||
patch.object(manage, "compose", side_effect=AssertionError("compose used for native ntfy")):
|
||||
manage.stop(config)
|
||||
stop.assert_called_once_with(config)
|
||||
|
||||
def test_uninstall_native_skips_compose_down(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
(root / "bin").mkdir()
|
||||
config = self.config(root)
|
||||
with patch.object(manage, "stop"), \
|
||||
patch.object(manage, "registration", return_value=None), \
|
||||
patch.object(manage, "compose", side_effect=AssertionError("compose used for native ntfy")):
|
||||
manage.uninstall(config, purge=False, yes=True)
|
||||
self.assertTrue((root / manage.MARKER).is_file())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -147,6 +147,14 @@ class ReleaseResolutionTests(unittest.TestCase):
|
||||
with self.assertRaisesRegex(resolve_releases.ReleaseError, "non-registry host"):
|
||||
resolve_releases._default_fetch("https://evil.example.com/tags/list")
|
||||
|
||||
def test_shared_allowlist_guard_requires_https_allowlisted_host(self):
|
||||
self.assertTrue(resolve_releases.is_allowlisted("https://api.github.com/x"))
|
||||
self.assertTrue(resolve_releases.is_allowlisted("https://ghcr.io/x"))
|
||||
self.assertTrue(resolve_releases.is_allowlisted("https://registry.npmjs.org/x"))
|
||||
self.assertFalse(resolve_releases.is_allowlisted("http://api.github.com/x"))
|
||||
self.assertFalse(resolve_releases.is_allowlisted("https://evil.example.com/x"))
|
||||
self.assertFalse(resolve_releases.is_allowlisted("https://api.github.com.evil.example/x"))
|
||||
|
||||
def test_malformed_pi_metadata_is_rejected(self):
|
||||
for pi in ({"version": "not-semver"}, {"version": "1.1.0", "engines": "node"}, 42):
|
||||
with self.assertRaises(resolve_releases.ReleaseError):
|
||||
@@ -178,5 +186,51 @@ class ReleaseResolutionTests(unittest.TestCase):
|
||||
resolve_releases.load_fixture(path)
|
||||
|
||||
|
||||
class FakeNtfy:
|
||||
"""Deterministic ntfy tag fixture served through the same fetch seam."""
|
||||
|
||||
VERSION = "2.11.0"
|
||||
|
||||
def __init__(self, tags=None, tags_response=None):
|
||||
self.tags = tags if tags is not None else ["v" + self.VERSION]
|
||||
self.tags_response = tags_response
|
||||
self.calls = []
|
||||
|
||||
def __call__(self, url, headers=None):
|
||||
self.calls.append(url)
|
||||
if "/git/matching-refs/" in url:
|
||||
if self.tags_response is not None:
|
||||
return 200, {}, json.dumps(self.tags_response).encode()
|
||||
return 200, {}, json.dumps([{"ref": "refs/tags/" + tag} for tag in self.tags]).encode()
|
||||
return 404, {}, b""
|
||||
|
||||
|
||||
class NtfyResolutionTests(unittest.TestCase):
|
||||
def test_resolves_exact_version_from_official_tags_only(self):
|
||||
registry = FakeNtfy(tags=["v2.10.0", "v2.11.0"])
|
||||
result = resolve_releases.resolve_ntfy(registry)
|
||||
self.assertEqual(result["version"], "2.11.0")
|
||||
self.assertEqual(result["tag"], "v2.11.0")
|
||||
self.assertEqual(result["module"], resolve_releases.NTFY_MODULE)
|
||||
self.assertEqual(result["repository"], resolve_releases.NTFY_REPOSITORY)
|
||||
# No release assets or checksums are fetched: the native installer builds
|
||||
# the server from the pinned module instead.
|
||||
self.assertNotIn("assets", result)
|
||||
self.assertEqual(len(registry.calls), 1)
|
||||
self.assertTrue(registry.calls[0].startswith(
|
||||
"https://api.github.com/repos/binwiederhier/ntfy/git/matching-refs/"))
|
||||
self.assertFalse(any("releases/latest" in url for url in registry.calls))
|
||||
|
||||
def test_malformed_tag_metadata_is_rejected(self):
|
||||
for tags in ([42], [{"ref": "refs/heads/main"}], [{"ref": "refs/tags/not-a-version"}],
|
||||
[{"noref": "refs/tags/v2.11.0"}]):
|
||||
with self.assertRaises(resolve_releases.ReleaseError):
|
||||
resolve_releases.resolve_ntfy(FakeNtfy(tags_response=tags))
|
||||
|
||||
def test_no_stable_release_fails_closed(self):
|
||||
with self.assertRaisesRegex(resolve_releases.ReleaseError, "no stable ntfy release"):
|
||||
resolve_releases.resolve_ntfy(FakeNtfy(tags=[]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -12,15 +12,22 @@ class TokenTests(unittest.TestCase):
|
||||
def test_extract_prose_and_protect_source_directory(self):
|
||||
token = "tk_" + "a" * 29
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
save_token.save_token(f"token {token} created for user pipeline-publisher, never expires\n", directory)
|
||||
path = Path(directory, "secrets/ntfy-token.txt")
|
||||
save_token.save_token(f"token {token} created for user pipeline-publisher, never expires\n", path, 0o444)
|
||||
self.assertEqual(path.read_text(), token + "\n")
|
||||
self.assertEqual(path.stat().st_mode & 0o777, 0o444)
|
||||
self.assertEqual(path.parent.stat().st_mode & 0o777, 0o700)
|
||||
with self.assertRaises(ValueError):
|
||||
save_token.save_token(f"token {token} created", directory)
|
||||
save_token.save_token(f"token {token} created", path, 0o444)
|
||||
self.assertEqual(path.read_text(), token + "\n")
|
||||
|
||||
def test_native_mode_is_private_to_the_installing_user(self):
|
||||
token = "tk_" + "b" * 29
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory, "secrets/publisher-token")
|
||||
save_token.save_token(f"token {token} created", path, 0o600)
|
||||
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
|
||||
|
||||
def test_failures_do_not_replace_empty_placeholder(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory, "secrets/ntfy-token.txt")
|
||||
@@ -28,9 +35,38 @@ class TokenTests(unittest.TestCase):
|
||||
path.write_text("")
|
||||
for output in ["", "connection failed", "tk_" + "a" * 29 + " tk_" + "b" * 29]:
|
||||
with self.assertRaises(ValueError):
|
||||
save_token.save_token(output, directory)
|
||||
save_token.save_token(output, path, 0o444)
|
||||
self.assertEqual(path.read_text(), "")
|
||||
|
||||
def test_accounts_are_bootstrapped_once_with_the_shared_sequence(self):
|
||||
token = "tk_" + "c" * 29
|
||||
|
||||
class FakeCompleted:
|
||||
returncode = 0
|
||||
stdout = f"token {token} created for pipeline-publisher\n"
|
||||
stderr = ""
|
||||
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory, "secrets/publisher-token")
|
||||
calls = []
|
||||
|
||||
def run(args, capture=False, interactive=False):
|
||||
calls.append((tuple(args), capture, interactive))
|
||||
return FakeCompleted()
|
||||
|
||||
save_token.bootstrap_accounts(run, path, 0o600, log=lambda _message: None)
|
||||
self.assertEqual([args for args, _, _ in calls], [
|
||||
("user", "add", "--ignore-exists", "pipeline-publisher"),
|
||||
("access", "pipeline-publisher", "pipelines", "write-only"),
|
||||
("token", "add", "pipeline-publisher"),
|
||||
("user", "add", "--ignore-exists", "phone-reader"),
|
||||
("access", "phone-reader", "pipelines", "read-only"),
|
||||
])
|
||||
self.assertEqual([capture for _, capture, _ in calls], [False, False, True, False, False])
|
||||
self.assertEqual([interactive for _, _, interactive in calls], [True, False, False, True, False])
|
||||
self.assertEqual(path.read_text(), token + "\n")
|
||||
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
spec = importlib.util.spec_from_file_location("setup_native", Path(__file__).with_name("setup-native.py"))
|
||||
setup_native = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(setup_native)
|
||||
|
||||
manage_spec = importlib.util.spec_from_file_location("pipelines_manage", Path(__file__).with_name("manage.py"))
|
||||
manage = importlib.util.module_from_spec(manage_spec)
|
||||
manage_spec.loader.exec_module(manage)
|
||||
|
||||
TOKEN = "tk_" + "a" * 29
|
||||
NTFY_VERSION = "2.11.0"
|
||||
NTFY_MODULE = "heckel.io/ntfy/v2"
|
||||
|
||||
|
||||
def resolution():
|
||||
"""The shape resolve_ntfy returns: an exact tag to build the server from."""
|
||||
return {"repository": "binwiederhier/ntfy", "module": NTFY_MODULE,
|
||||
"version": NTFY_VERSION, "tag": "v" + NTFY_VERSION}
|
||||
|
||||
|
||||
class Recorder:
|
||||
def __init__(self, token=TOKEN, volumes=(), image="pipelines-pipelines:latest",
|
||||
fail_create=False, fail_copy=False, fail_volume_ls=False, fail_build=False):
|
||||
self.calls = []
|
||||
self.token = token
|
||||
self.volumes = volumes
|
||||
self.image = image
|
||||
self.fail_create = fail_create
|
||||
self.fail_copy = fail_copy
|
||||
self.fail_volume_ls = fail_volume_ls
|
||||
self.fail_build = fail_build
|
||||
self._module_cache = None
|
||||
self.kwargs = []
|
||||
|
||||
def module_cache(self):
|
||||
"""A stand-in Go module cache holding the pinned ntfy module source."""
|
||||
if self._module_cache is None:
|
||||
self._module_cache = tempfile.mkdtemp(prefix="gomodcache-")
|
||||
source = Path(self._module_cache) / f"{NTFY_MODULE}@{resolution()['tag']}"
|
||||
source.mkdir(parents=True)
|
||||
(source / "go.mod").write_text(f"module {NTFY_MODULE}\n")
|
||||
return self._module_cache
|
||||
|
||||
def __call__(self, command, **kwargs):
|
||||
command = list(command)
|
||||
self.calls.append(command)
|
||||
self.kwargs.append(kwargs)
|
||||
stdout, code = "", 0
|
||||
if command[0] == "docker":
|
||||
if command[1:3] == ["volume", "ls"]:
|
||||
if self.fail_volume_ls:
|
||||
code = 1
|
||||
else:
|
||||
stdout = "\n".join(self.volumes)
|
||||
elif command[1:3] == ["image", "ls"]:
|
||||
stdout = self.image + "\n" if self.image else ""
|
||||
elif command[1] == "create" and self.fail_create:
|
||||
code = 1
|
||||
elif command[1] == "cp":
|
||||
if self.fail_copy:
|
||||
code = 1
|
||||
else:
|
||||
Path(command[3]).write_bytes(b"migrated")
|
||||
elif command[0] == "go" and command[1:3] == ["env", "GOMODCACHE"]:
|
||||
stdout = self.module_cache() + "\n"
|
||||
elif command[0] == "go" and command[1:2] == ["build"]:
|
||||
if self.fail_build:
|
||||
code = 1
|
||||
elif "-o" in command:
|
||||
# Only the ntfy build is inspected by the installer; its output
|
||||
# is created so the atomic install has a real file to move.
|
||||
output = Path(command[command.index("-o") + 1])
|
||||
if output.name == "ntfy":
|
||||
output.write_bytes(b"#!/bin/sh\n")
|
||||
output.chmod(0o755)
|
||||
if command[0].endswith("ntfy") and "token" in command:
|
||||
stdout = f"token {self.token} created for pipeline-publisher\n"
|
||||
return subprocess.CompletedProcess(command, code, stdout=stdout, stderr="")
|
||||
|
||||
def has(self, *parts):
|
||||
return any(all(part in call for part in parts) for call in self.calls)
|
||||
|
||||
|
||||
class SetupNativeTests(unittest.TestCase):
|
||||
def installer(self, directory, recorder):
|
||||
# A disposable XDG config home keeps the client default out of the real
|
||||
# user configuration during tests.
|
||||
return setup_native.Setup(directory, prefix=Path(directory) / "prefix", runner=recorder,
|
||||
resolver=resolution,
|
||||
environ={"XDG_CONFIG_HOME": str(Path(directory) / "config")})
|
||||
|
||||
@staticmethod
|
||||
def which_all(opencode="/usr/bin/opencode", pi=None, cc="/usr/bin/cc"):
|
||||
mapping = {"go": "/usr/bin/go", "python3": "/usr/bin/python3",
|
||||
"git": "/usr/bin/git", "curl": "/usr/bin/curl"}
|
||||
if opencode:
|
||||
mapping["opencode"] = opencode
|
||||
if pi:
|
||||
mapping["pi"] = pi
|
||||
if cc:
|
||||
mapping["cc"] = cc
|
||||
return lambda name: mapping.get(name)
|
||||
|
||||
def test_require_tools_reports_each_missing_tool_without_installing(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder()
|
||||
installer = self.installer(directory, recorder)
|
||||
mapping = {"go": "/usr/bin/go", "python3": "/usr/bin/python3", "git": None, "curl": None}
|
||||
with patch.object(setup_native.shutil, "which", side_effect=lambda name: mapping.get(name)):
|
||||
with self.assertRaises(ValueError) as raised:
|
||||
installer.require_tools()
|
||||
message = str(raised.exception)
|
||||
self.assertIn("git", message)
|
||||
self.assertIn("curl", message)
|
||||
self.assertIn("never installs tools", message)
|
||||
self.assertEqual(recorder.calls, [])
|
||||
|
||||
def test_build_installs_only_server_and_cli_with_trimpath(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder()
|
||||
installer = self.installer(directory, recorder)
|
||||
prefix = Path(directory) / "prefix"
|
||||
installer.prepare_layout(prefix)
|
||||
installer.build(prefix)
|
||||
builds = [call for call in recorder.calls if call[:2] == ["go", "build"]]
|
||||
self.assertEqual(len(builds), 2)
|
||||
for call in builds:
|
||||
self.assertIn("-trimpath", call)
|
||||
self.assertTrue(any(call[-1] == "./cmd/pipelines-server" for call in builds))
|
||||
self.assertTrue(any(call[-1] == "./cmd/pipelines" for call in builds))
|
||||
self.assertFalse(any("pipelines-tui" in part for call in recorder.calls for part in call))
|
||||
|
||||
def test_layout_creates_private_prefix_and_copies_manage(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
installer = self.installer(directory, Recorder())
|
||||
prefix = Path(directory) / "prefix"
|
||||
installer.prepare_layout(prefix)
|
||||
for name in ("bin", "lib", "data", "logs", "secrets"):
|
||||
self.assertTrue((prefix / name).is_dir(), name)
|
||||
self.assertEqual((prefix / "secrets").stat().st_mode & 0o777, 0o700)
|
||||
self.assertTrue((prefix / "lib/manage.py").is_file())
|
||||
|
||||
def test_prefix_honours_pipelines_prefix_environment(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
custom = Path(directory) / "custom"
|
||||
installer = setup_native.Setup(directory, runner=Recorder(), environ={"PIPELINES_PREFIX": str(custom)})
|
||||
self.assertEqual(installer.resolve_prefix(), custom.resolve())
|
||||
|
||||
def test_missing_opencode_fails_actionably(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
installer = self.installer(directory, Recorder())
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all(opencode=None)):
|
||||
with self.assertRaisesRegex(ValueError, "OpenCode was not found"):
|
||||
installer.harness()
|
||||
|
||||
def test_full_run_writes_manifest_tokens_and_config_without_harness_install(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder()
|
||||
installer = self.installer(directory, recorder)
|
||||
prefix = Path(directory) / "prefix"
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all()), \
|
||||
patch.object(setup_native.lan, "detect_lan", return_value="192.168.1.20"), \
|
||||
patch.object(setup_native.configure, "host_timezone", return_value="Europe/London"):
|
||||
installer.run(setup_native.arguments([]))
|
||||
config = manage.load(prefix)
|
||||
self.assertEqual(config["format"], 1)
|
||||
self.assertEqual(config["prefix"], str(prefix.resolve()))
|
||||
self.assertEqual(config["server_port"], 8080)
|
||||
self.assertEqual(config["ntfy"]["mode"], "native")
|
||||
self.assertEqual(config["topic"], "pipelines")
|
||||
self.assertEqual(config["opencode"], "/usr/bin/opencode")
|
||||
self.assertEqual(config["pi"], "")
|
||||
self.assertEqual((prefix / setup_native.MARKER).stat().st_mode & 0o777, 0o600)
|
||||
token_path = prefix / "secrets/publisher-token"
|
||||
self.assertEqual(token_path.read_text(), TOKEN + "\n")
|
||||
self.assertEqual(token_path.stat().st_mode & 0o777, 0o600)
|
||||
text = (prefix / "data/ntfy/server.yml").read_text()
|
||||
for needle in ('listen-http:', 'base-url: "http://192.168.1.20:8090"', 'cache-file:', 'auth-file:',
|
||||
'auth-default-access: "deny-all"', 'enable-login: true', 'attachment-cache-dir:',
|
||||
'attachment-expiry-duration:', 'attachment-total-size-limit:',
|
||||
'attachment-file-size-limit:'):
|
||||
self.assertIn(needle, text)
|
||||
self.assertNotIn("topic", text)
|
||||
self.assertTrue(recorder.has(str(prefix.resolve() / "bin/pipelines"), "skill", "install"))
|
||||
self.assertTrue(recorder.has("access", "pipeline-publisher", "pipelines", "write-only"))
|
||||
self.assertTrue(recorder.has("access", "phone-reader", "pipelines", "read-only"))
|
||||
self.assertFalse(any("opencode" in part for call in recorder.calls for part in call))
|
||||
self.assertFalse(any("pipelines-tui" in part for call in recorder.calls for part in call))
|
||||
self.assertFalse(any(TOKEN in part for call in recorder.calls for part in call))
|
||||
|
||||
def test_reinstall_keeps_existing_publisher_token(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder()
|
||||
installer = self.installer(directory, recorder)
|
||||
prefix = Path(directory) / "prefix"
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all()), \
|
||||
patch.object(setup_native.lan, "detect_lan", return_value="192.168.1.20"), \
|
||||
patch.object(setup_native.configure, "host_timezone", return_value="Europe/London"):
|
||||
installer.run(setup_native.arguments([]))
|
||||
before = (prefix / "secrets/publisher-token").read_text()
|
||||
|
||||
def refuse_resolution():
|
||||
raise AssertionError("re-resolved ntfy despite a verified installed binary")
|
||||
|
||||
installer.resolver = refuse_resolution
|
||||
recorder.calls.clear()
|
||||
installer.run(setup_native.arguments([]))
|
||||
self.assertEqual((prefix / "secrets/publisher-token").read_text(), before)
|
||||
self.assertFalse(any("token" in call for call in recorder.calls), "reinstall re-created the token")
|
||||
|
||||
def test_ntfy_is_built_from_the_pinned_module(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder()
|
||||
installer = self.installer(directory, recorder)
|
||||
prefix = Path(directory) / "prefix"
|
||||
installer.prepare_layout(prefix)
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all()):
|
||||
path = installer.build_ntfy(prefix, resolution())
|
||||
self.assertTrue(path.is_file())
|
||||
self.assertTrue(path.stat().st_mode & 0o100)
|
||||
self.assertTrue(recorder.has("go", "mod", "download", f"{NTFY_MODULE}@v{NTFY_VERSION}"))
|
||||
build = next(call for call in recorder.calls if call[:2] == ["go", "build"] and call[-1] == ".")
|
||||
self.assertIn("-trimpath", build)
|
||||
self.assertIn(setup_native.NTFY_BUILD_TAGS, build)
|
||||
self.assertTrue(any(f"-X main.version=v{NTFY_VERSION}" in part for part in build))
|
||||
self.assertTrue(build[build.index("-o") + 1].endswith("/ntfy"))
|
||||
kwargs = next(kw for call, kw in zip(recorder.calls, recorder.kwargs)
|
||||
if call[:2] == ["go", "build"] and call[-1] == ".")
|
||||
self.assertEqual(kwargs.get("env", {}).get("CGO_ENABLED"), "1")
|
||||
|
||||
def test_ntfy_build_is_reused_for_the_same_version(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder()
|
||||
installer = self.installer(directory, recorder)
|
||||
prefix = Path(directory) / "prefix"
|
||||
installer.prepare_layout(prefix)
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all()):
|
||||
installer.build_ntfy(prefix, resolution())
|
||||
installer.write_manifest(prefix, {"ntfy": {"version": "v" + NTFY_VERSION}})
|
||||
recorder.calls.clear()
|
||||
installer.build_ntfy(prefix, resolution())
|
||||
self.assertFalse(any(call[:2] == ["go", "build"] for call in recorder.calls),
|
||||
"rebuilt ntfy for an already installed version")
|
||||
|
||||
def test_missing_c_compiler_fails_actionably(self):
|
||||
installer = setup_native.Setup("/tmp", runner=Recorder(), resolver=resolution, environ={})
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all(cc=None)):
|
||||
with self.assertRaisesRegex(ValueError, "No C compiler found"):
|
||||
installer.require_compiler()
|
||||
|
||||
def test_ntfy_build_failure_is_reported(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
installer = self.installer(directory, Recorder(fail_build=True))
|
||||
prefix = Path(directory) / "prefix"
|
||||
installer.prepare_layout(prefix)
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all()):
|
||||
with self.assertRaisesRegex(ValueError, "build failed"):
|
||||
installer.build_ntfy(prefix, resolution())
|
||||
|
||||
def test_publisher_token_is_never_overwritten(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
prefix = Path(directory) / "prefix"
|
||||
(prefix / "secrets").mkdir(parents=True)
|
||||
path = prefix / "secrets/publisher-token"
|
||||
setup_native.tokens.save_token(f"token {TOKEN} created", path, 0o600)
|
||||
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
|
||||
with self.assertRaises(ValueError):
|
||||
setup_native.tokens.save_token(f"token {TOKEN} created", path, 0o600)
|
||||
self.assertEqual(path.read_text(), TOKEN + "\n")
|
||||
|
||||
def test_migration_copies_docker_data_only_when_absent(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
prefix = Path(directory) / "prefix"
|
||||
(prefix / "data/ntfy").mkdir(parents=True)
|
||||
recorder = Recorder(volumes=("pipelines_pipelines-data", "pipelines_ntfy-data"))
|
||||
installer = self.installer(directory, recorder)
|
||||
self.assertTrue(installer.migrate_docker(prefix, replace=False))
|
||||
self.assertEqual((prefix / "data/pipelines.db").read_bytes(), b"migrated")
|
||||
self.assertTrue(any(call[:2] == ["docker", "cp"] and call[2].endswith("-wal")
|
||||
for call in installer.runner.calls),
|
||||
"the SQLite write-ahead log sidecar must be copied with the database")
|
||||
self.assertEqual((prefix / "data/ntfy/auth.db").read_bytes(), b"migrated")
|
||||
|
||||
def test_migration_keeps_existing_native_data_without_opt_in(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
prefix = Path(directory) / "prefix"
|
||||
(prefix / "data").mkdir(parents=True)
|
||||
(prefix / "data/pipelines.db").write_bytes(b"native")
|
||||
recorder = Recorder(volumes=("pipelines_pipelines-data",))
|
||||
installer = self.installer(directory, recorder)
|
||||
self.assertTrue(installer.migrate_docker(prefix, replace=False))
|
||||
self.assertEqual((prefix / "data/pipelines.db").read_bytes(), b"native")
|
||||
self.assertFalse(any(call[1:2] == ["cp"] for call in recorder.calls))
|
||||
|
||||
def test_absent_docker_volumes_is_a_noop(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder(volumes=())
|
||||
installer = self.installer(directory, recorder)
|
||||
self.assertFalse(installer.migrate_docker(Path(directory) / "prefix", replace=False))
|
||||
self.assertFalse(any(call[0] == "docker" and call[1] == "create" for call in recorder.calls))
|
||||
|
||||
def test_uninspectable_docker_volumes_abort_instead_of_skipping_migration(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder(fail_volume_ls=True)
|
||||
installer = self.installer(directory, recorder)
|
||||
with self.assertRaisesRegex(ValueError, "Could not inspect Docker volumes"):
|
||||
installer.migrate_docker(Path(directory) / "prefix", replace=False)
|
||||
|
||||
def test_missing_docker_cli_is_a_noop_not_a_failure(self):
|
||||
def no_docker(command, **kwargs):
|
||||
raise FileNotFoundError(command[0])
|
||||
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
installer = setup_native.Setup(directory, prefix=Path(directory) / "prefix",
|
||||
runner=no_docker, resolver=resolution, environ={})
|
||||
self.assertEqual(installer.docker_volumes(), ())
|
||||
|
||||
def test_requested_migration_failure_aborts(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
prefix = Path(directory) / "prefix"
|
||||
(prefix / "data").mkdir(parents=True)
|
||||
recorder = Recorder(volumes=("pipelines_pipelines-data",), fail_copy=True)
|
||||
installer = self.installer(directory, recorder)
|
||||
with self.assertRaisesRegex(ValueError, "install aborted"):
|
||||
installer.migrate_docker(prefix, replace=False)
|
||||
|
||||
def test_full_run_migrates_docker_auth_store_before_native_accounts(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
recorder = Recorder(volumes=("pipelines_ntfy-data",))
|
||||
installer = self.installer(directory, recorder)
|
||||
prefix = Path(directory) / "prefix"
|
||||
with patch.object(setup_native.shutil, "which", side_effect=self.which_all()), \
|
||||
patch.object(setup_native.lan, "detect_lan", return_value="192.168.1.20"), \
|
||||
patch.object(setup_native.configure, "host_timezone", return_value="Europe/London"):
|
||||
installer.run(setup_native.arguments([]))
|
||||
# The migrated store must survive bootstrap instead of being shadowed.
|
||||
self.assertEqual((prefix / "data/ntfy/auth.db").read_bytes(), b"migrated")
|
||||
copy_index = next(i for i, call in enumerate(recorder.calls)
|
||||
if call[0] == "docker" and call[1:2] == ["cp"])
|
||||
add_index = next(i for i, call in enumerate(recorder.calls)
|
||||
if "pipeline-publisher" in call and "user" in call)
|
||||
self.assertLess(copy_index, add_index, "auth store migrated after native accounts were created")
|
||||
|
||||
def test_missing_local_image_aborts_migration(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
prefix = Path(directory) / "prefix"
|
||||
(prefix / "data").mkdir(parents=True)
|
||||
recorder = Recorder(volumes=("pipelines_pipelines-data",), image=None)
|
||||
installer = self.installer(directory, recorder)
|
||||
with self.assertRaisesRegex(ValueError, "no local Pipelines image"):
|
||||
installer.migrate_docker(prefix, replace=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user