diff --git a/README.md b/README.md index 7f2aab58..2fdceb6d 100644 --- a/README.md +++ b/README.md @@ -41,17 +41,33 @@ and the executed route highlighted. ## Install -`./install` installs **Docker server + ntfy**. Docker is the only supported runtime -installation. Setup needs Docker Compose and Python 3. +`./install` installs Pipelines **natively on the host by default** (macOS and +Linux); `./install --docker` installs the **Docker server + ntfy** explicitly. +Native setup detects (but never installs) Go, python3, git, curl and a C compiler +(`cc`/`clang`/`gcc`), plus an existing `opencode`/`pi`; it builds `pipelines-server`, +`pipelines` and ntfy from source into `~/.local/share/pipelines` (honouring +`PIPELINES_PREFIX`) and generates an ntfy configuration mirroring the Docker +settings. ntfy is compiled locally because upstream publishes no macOS server +binary, and the C compiler is required because its SQLite driver is cgo. Docker +setup needs Docker Compose and Python 3 and includes OpenCode/Pi in the persistent +server container. For the host CLI/TUI walkthrough below, also install Go 1.27+; the Docker installer installs/updates those clients when host Go is available. ```sh git clone https://git.a-wan.net/alan/pipelines.git cd pipelines -./install # configure/start Docker services and connect a provider +./install # native host install (Go, python3, git, curl, cc, opencode) +./install --docker # explicit Docker services + container provider login ``` +Native setup reuses your existing host OpenCode login and never installs, +upgrades or downloads OpenCode/Pi. Start it with +`PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start`. +When Docker volumes are present it migrates the existing database and ntfy auth +store (never overwriting newer native data; use `--replace-data` to opt in). See +[installation](docs/install.md) for details. + The installer requires this Git checkout (including `.git`); a standalone source archive cannot self-install. Without host Go you can still use the Docker-contained TUI/CLI. Run Compose from the checkout with `--project-directory` @@ -91,7 +107,7 @@ preselected and marked `(current)`; typing filters, arrow keys or Tab navigate, Enter confirms, and Esc cancels without changing anything). Model selection performs no inference; only the OpenCode path validates the new-pipeline model with a small real generation (provider charges may apply). Fresh -`./install --harness pi` skips OpenCode login and that generation check. Pi has no +`./install --docker --harness pi` skips OpenCode login and that generation check. Pi has no standalone `auth login`: setup opens a fresh `/login` session (regular TUI mode) where you choose an authentication method, reuse existing volume credentials by exiting, or sign in with an API key. Each @@ -107,7 +123,8 @@ services: an image matching the release identity needs no rebuild, while a dirty source always invokes the Docker build and relies on layer caching. The built image includes python3 so the installer/verification tests can also run inside the container. -Configuration and data remain per-user. No native server installer is offered. +Configuration and data remain per-user. The native default installer is described +above and in [installation](docs/install.md). Docker setup starts the server and saves its client endpoint (the container port is **8080**; only the default host mapping is **8081**). That saved endpoint is @@ -124,9 +141,10 @@ them to the public Internet. ### Developer/legacy local server (unsupported) -For developing Pipelines itself or running a previous native setup, an explicit -foreground server remains available. This is **not** a supported end-user -installation and is not part of Docker setup; fresh databases contain no pipelines: +For developing Pipelines itself or running a previous Python-supervised legacy +host setup, an explicit foreground server remains available. This is not the +native installer and is not part of Docker setup; fresh databases contain no +pipelines: ```sh pipelines serve # foreground developer/legacy server diff --git a/docs/docker.md b/docs/docker.md index 39e266b3..4ec0d2fb 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -2,11 +2,12 @@ ## Complete installer (recommended for both services in Docker) -Run `./install` from an interactive terminal (Docker is the default; -`./install.sh` remains a compatibility alias). It handles Compose +Run `./install --docker` from an interactive terminal (Docker is the explicit +`--docker`/`docker` mode; `./install.sh` remains a compatibility alias, and a bare +`./install` now performs the native host installation). It handles Compose configuration, ntfy users/permissions/token, image build, container skill install, first-time container harness provider login (OpenCode by default; `./install ---harness pi` configures Pi and skips OpenCode login), startup, and an +--docker --harness pi` configures Pi and skips OpenCode login), startup, and an authenticated test notification. Choose your own provider in the configured harness; signup/login credentials stay in the Docker volume, not your host account. Re-run to update while @@ -80,8 +81,9 @@ are sent until it is filled. `.env`, `secrets/`, and `data/` are git-ignored and excluded from image builds. The container publishes port **8080**; the default host mapping is **8081**, bound to host `0.0.0.0`, avoiding the existing native server on 8080. Query the actual mapped port with -`docker compose port pipelines 8080`. Android does not need access to the -unauthenticated pipeline API. +`docker compose port pipelines 8080`. The unauthenticated pipeline WebSocket API +shares that port, so expose it only on a trusted LAN/VPN and never to the public +internet. Use HTTP only on a trusted LAN or encrypted VPN. For public access, use HTTPS with a reverse proxy/VPN; configure the external URL in `NTFY_BASE_URL`. The @@ -186,8 +188,8 @@ npm, then builds `FROM` the OpenCode image's immutable `sha256` digest and label the built image with the resolved versions, digest and source identity. It fails closed (keeping the last verified Dockerfile fallback pins) rather than guessing or silently substituting a stale pin when discovery, digest resolution, build or -compatibility verification fails. A full `./install` always invokes the Docker build -for a dirty source and relies on layer caching; a model/login-only reconfiguration -rebuilds only when the verified release image identity (versions/digest) no longer -matches the resolved releases, so a source change alone does not rebuild on -reconfigure. +compatibility verification fails. A full `./install --docker` always invokes the +Docker build for a dirty source and relies on layer caching; a model/login-only +reconfiguration rebuilds only when the verified release image identity +(versions/digest) no longer matches the resolved releases, so a source change +alone does not rebuild on reconfigure. diff --git a/docs/first-version-testing.md b/docs/first-version-testing.md index 43d178bb..d6878bac 100644 --- a/docs/first-version-testing.md +++ b/docs/first-version-testing.md @@ -7,9 +7,10 @@ history. It is not a guarantee that a model completes every task correctly. ## Before installing - Use a Git checkout (including `.git`) so Docker setup can stamp its build identity. -- **Docker is the only supported runtime installation.** The - [recommended installer](install.md) needs Docker Compose and Python 3; it builds - the Go CLI/TUI clients on the host only when Go 1.27+ is available. +- **Native host installation is the default** (`./install`/`--native`); the + [Docker installation](install.md) is explicit with `./install --docker` and needs + Docker Compose and Python 3. Docker setup builds the Go CLI/TUI clients on the host + only when Go 1.27+ is available. - Docker setup starts services, asks for ntfy passwords, opens provider sign-in, and then saves **two separate model defaults** (the interactive harness model and the new-pipeline model) in two fuzzy pickers. Signing in to a provider is @@ -25,9 +26,10 @@ history. It is not a guarantee that a model completes every task correctly. ## Install and identify the build -From the checkout, run `./install`, then restart any existing TUI. Docker is the -only supported runtime installation; no mode flag is needed. The installer -requires a Git checkout; a standalone source archive cannot self-install. +From the checkout, run `./install` for the native host installation, or +`./install --docker` for the Docker server + ntfy installation, then restart any +existing TUI. The installer requires a Git checkout; a standalone source archive +cannot self-install. Follow [installation](install.md) for PATH and provider setup. Do not start a native server alongside your Docker server. diff --git a/docs/harnesses-and-control.md b/docs/harnesses-and-control.md index c1f21cb0..bd8eb0f7 100644 --- a/docs/harnesses-and-control.md +++ b/docs/harnesses-and-control.md @@ -3,7 +3,7 @@ Configure two independent model defaults: ```sh -./install --harness pi # Docker setup without OpenCode login +./install --docker --harness pi # Docker setup without OpenCode login ./install --opencode --default ./install --pi pipelines harness @@ -34,9 +34,11 @@ On existing Docker setups, the reconfiguration commands resolve and verify the current OpenCode V2/Pi releases, then operate inside the container without restarting it. A rebuild is required only when the verified release image identity (versions/digest) differs from the resolved releases; a -model/login-only change does not rebuild. A full `./install` always invokes the -Docker build for a dirty source and relies on layer caching. Installation is -Docker-only; reconfiguration never falls back to host harness execution. +model/login-only change does not rebuild. A full `./install --docker` always +invokes the Docker build for a dirty source and relies on layer caching. Harness +installation stays Docker-only: the default native setup only detects an existing +host OpenCode/Pi and never installs, upgrades or downloads them, and +reconfiguration never falls back to host harness execution. Container credentials and accounts are separate from host credentials and are reused on reinstall/reconfigure. Defaults affect only new drafts/starters, not existing immutable versions. diff --git a/docs/install.md b/docs/install.md index 6069f7e8..53e76164 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,21 +1,103 @@ # Installation -To reconfigure login and separate interactive/new-pipeline models without a -service reinstall: `./install --opencode` or `./install --pi`. +`./install` installs **natively on the host by default** (macOS and Linux); +`./install --docker` runs the existing Docker server + ntfy installation. +To reconfigure login and separate interactive/new-pipeline models inside an +existing Docker installation without a service reinstall: `./install --opencode` +or `./install --pi`. See [configured harnesses](harnesses-and-control.md) for scope and verification limits. -`./install` sets up **Docker server + ntfy**, the only supported runtime installation. -It requires Docker Compose and Python 3 and includes OpenCode/Pi in the persistent -server container. Host Go builds the CLI/TUI clients, not a native server. - ```sh git clone https://git.a-wan.net/alan/pipelines.git cd pipelines -./install -# Optional: choose Pi rather than OpenCode for setup: -./install --harness pi +./install # native: build server, CLI and ntfy on this host +./install --docker # explicit Docker server + ntfy ``` +## Native installation (default) + +`./install`, `./install --native` and `./install native` run +`scripts/setup-native.py`. It only detects the tools it needs and never installs +or upgrades them: **Go, python3, git, curl and a C compiler (`cc`/`clang`/`gcc`)** +must already be on PATH. The C compiler is required because ntfy stores its cache +and auth database in SQLite through the cgo driver `mattn/go-sqlite3`. It also +detects an existing `opencode` (required) and `pi` (optional) on PATH and records +their absolute paths; it never downloads, installs or upgrades OpenCode or Pi +(harness installation stays Docker-only). + +Setup resolves `PIPELINES_PREFIX` or defaults to `~/.local/share/pipelines`, +creates `bin/`, `lib/`, `data/`, `logs/` and `secrets/` (`secrets` and `data` at +0700), copies `scripts/manage.py` to `/lib/manage.py`, builds exactly +`./cmd/pipelines-server` and `./cmd/pipelines` with `go build -trimpath` into +`/bin`, and runs `/bin/pipelines skill install` so the packaged +authoring skill lands in your OpenCode skills directory. No `pipelines-tui` is +built. It writes `/pipelines-install.json` atomically at 0600; that +manifest is what `scripts/manage.py` and the Go CLI load. Provider/model defaults +are never invented: your existing host OpenCode configuration is reused (an +explicit `--model` is recorded if supplied), and setup never starts a picker. + +ntfy is built from source with no Homebrew or package manager. Setup resolves the +exact stable `binwiederhier/ntfy` tag (never the moving `latest` tag and never an +older fallback), then compiles the server from the pinned `heckel.io/ntfy/v2` Go +module at that tag into `/bin/ntfy`. A source build is required because +upstream publishes **no macOS server binary** (its `darwin_all` asset is built +with `-tags noserver` and only provides the client commands), so both macOS and +Linux take the same path; the Go toolchain verifies the module against its public +checksum database. The build uses +`-tags sqlite_omit_load_extension,osusergo,netgo` and creates the same empty web +placeholders the upstream Makefile uses for local builds, so the bundled ntfy web +UI is not populated (notifications and the API are unaffected). A binary for the +same resolved version is reused instead of rebuilt. Setup generates +`/data/ntfy/server.yml` mirroring `compose.yaml` (`listen-http`, +`base-url`, `cache-file`, `auth-file`, `auth-default-access: deny-all`, +`enable-login`, attachment cache directory, expiry and size limits), creates the +`pipeline-publisher` (write-only on `pipelines`) and `phone-reader` (read-only) +accounts with ntfy's own CLI (`--ignore-exists`, so existing passwords are kept; +the account subcommands read the generated config through `NTFY_CONFIG_FILE`), +and writes the publisher token only to `/secrets/publisher-token` at 0600. + +When Docker volumes `pipelines_pipelines-data` or `pipelines_ntfy-data` exist, +setup migrates the pipeline database to `/data/pipelines.db` and the ntfy +auth store to `/data/ntfy/auth.db` using the local Pipelines image +(never an implicit pull). Existing native data is never overwritten unless you +pass `--replace-data`; a requested migration that fails aborts the install +instead of starting empty. Absent volumes are a no-op. If the Docker CLI is +present but its volumes cannot be inspected (for example a stopped daemon), setup +fails clearly instead of silently skipping migration, so an existing Docker +database is never shadowed by an empty native start. + +Start and manage it with the manifest's prefix on PATH: + +```sh +PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines start +PIPELINES_PREFIX=~/.local/share/pipelines ~/.local/share/pipelines/bin/pipelines status +``` + +`start`, `stop`, `restart`, `doctor`, `logs` and `uninstall` reach +`/lib/manage.py`. Ports default to 8080 (server) and 8090 (ntfy) and can +be changed with `--port`/`--ntfy-port`; setup starts both services itself, so +those ports must be free (a running Docker installation holds them — stop it +first; the migration still needs its volumes and image). ntfy runs from +`/bin/ntfy` with the generated config; per-user launchd (macOS) or +`systemd --user` (Linux) registration and the supervised fallback both remain. + +Dated evidence (2026-10-13, macOS arm64): a scratch-prefix run of +`scripts/setup-native.py` resolved ntfy v2.30.0, compiled it from source (`ntfy +--version` = v2.30.0 with the `Server commands` present), built the server and +CLI, installed the skill, migrated 8 pipeline definitions and 28 runs plus both +ntfy accounts from the live Docker volumes, and wrote the manifest. +`pipelines start` then brought up ntfy and the server through the supervisor +(`{"healthy":true}` from both `/v1/health` and `/healthz` on spare ports) and +`pipelines stop` released both ports. launchd/systemd registration and real phone +delivery are **not tested**. + +## Docker installation (explicit) + +`./install --docker` (or `./install docker`) runs the unchanged +`scripts/setup-docker.py`. It requires Docker Compose and Python 3 and includes +OpenCode/Pi in the persistent server container. Host Go builds the CLI/TUI +clients, not a native server. + The installer builds host `pipelines` and `pipelines-tui` clients when Go is available, using `GOBIN` or `GOPATH/bin`; put that directory on PATH. It does not install a host server. Reconfiguration uses `./install --opencode` or `./install --pi` @@ -111,9 +193,9 @@ For fresh configuration you can supply `--ntfy-url https://ntfy.example.com --bind-address 127.0.0.1 --timezone Europe/London --port 8081` directly to `./install`. The defaults are ready for direct phone access on your trusted LAN. Only use plain HTTP on a trusted network. Other existing `.env` settings are not replaced. -Docker is also the default without a terminal; setup fails rather than -silently falling back to native binaries if the LAN address cannot be detected. -Use an interactive terminal for first Docker/provider setup. +These Docker flags apply to `./install --docker`; the explicit Docker mode fails +rather than silently falling back to native binaries if the LAN address cannot be +detected. Use an interactive terminal for first Docker/provider setup. The Docker Pipelines API is unauthenticated: keep its port on a trusted network and do not forward it to the Internet. For a manual loopback-only deployment, set `PIPELINES_BIND_ADDRESS=127.0.0.1` after installation and use Compose directly. @@ -202,8 +284,8 @@ The installer does not hardcode a system destination or run sudo. ## Developer/legacy host server (unsupported) This explicit foreground server is for developing Pipelines itself or running a -previous native setup. It is **not** a supported installation; a fresh database -starts empty. +previous Python-supervised legacy host setup. It is **not** the native installer; +a fresh database starts empty. ```sh pipelines serve # foreground developer/legacy server; Ctrl+C stops it diff --git a/install.sh b/install.sh index afc8e4e4..563b04fd 100755 --- a/install.sh +++ b/install.sh @@ -13,8 +13,8 @@ case "$mode" in exec python3 scripts/setup-docker.py "$@" ;; --native|native) - echo 'Native installation is not supported. Run ./install for Docker setup.' >&2 - exit 2 + shift + exec python3 scripts/setup-native.py "$@" ;; - *) exec python3 scripts/setup-docker.py "$@" ;; + *) exec python3 scripts/setup-native.py "$@" ;; esac diff --git a/internal/author/skill/references/harnesses.md b/internal/author/skill/references/harnesses.md index b319982b..9c0b5414 100644 --- a/internal/author/skill/references/harnesses.md +++ b/internal/author/skill/references/harnesses.md @@ -63,10 +63,13 @@ stay authoritative and a dirty source still rebuilds. If discovery, digest resolution or completeness cannot be established they stop with an error (keeping the last verified Dockerfile fallback pins); they never guess, fall back to a host runtime or install a stale pin. -Without Docker setup, these commands stop and require `./install`. Container/host -credentials and accounts remain independent and are reused on reinstall/reconfigure. -Docker is the only supported runtime installation: run `./install`, with no mode flag. -Fresh setup accepts `./install --harness pi` to skip OpenCode +Without a Docker installation, `./install --opencode`/`--pi` stop and require an +existing Docker setup; the native installation reuses the host OpenCode +configuration. Container and host credentials and accounts remain independent and +are reused on reinstall/reconfigure. `./install` installs natively by default +(`--native` is equivalent); explicit `./install --docker` selects the Docker +server + ntfy installation. +Fresh setup accepts `./install --docker --harness pi` to skip OpenCode login. Existing setup can configure either harness independently. Pi setup does not run a generation check; `provider-check --harness pi --model PROVIDER/MODEL` is a separate confirmed, potentially chargeable check. diff --git a/internal/author/skill/references/operations.md b/internal/author/skill/references/operations.md index fa43d3f8..0ce54ee1 100644 --- a/internal/author/skill/references/operations.md +++ b/internal/author/skill/references/operations.md @@ -102,8 +102,10 @@ raw credential/session databases or publish private inspect output as diagnostic In a source checkout, `python3 scripts/smoke-first-install.py` checks a disposable source-built test binaries, draft/publication/transfer and restart persistence without inference, notifications or deployment. It is not a live-provider or phone test. -That is a developer test fixture, not a native installation option. End-user -installation is Docker-only through `./install`; no `--docker` flag is needed. +That is a developer test fixture, not the native installer. End-user installation +runs natively through `./install` by default (`scripts/setup-native.py`, with +`--native` equivalent); the Docker server + ntfy installation is explicit with +`./install --docker`. In the TUI, Home advertises `/` search and `:` commands. Run Tab switches only Details ↔ Graph. `n/p` select recorded visits, including repeated stages; `d` diff --git a/internal/installation/installation.go b/internal/installation/installation.go index 464978b3..cc5b7053 100644 --- a/internal/installation/installation.go +++ b/internal/installation/installation.go @@ -29,6 +29,12 @@ func Load(executable string) (*Config, error) { if err != nil { return nil, err } + // The native installer and scripts/manage.py record the symlink-resolved + // prefix, so resolve the same way here instead of rejecting a symlinked + // HOME or prefix as "invalid". + if resolved, err := filepath.EvalSymlinks(prefix); err == nil { + prefix = resolved + } data, err := os.ReadFile(filepath.Join(prefix, "pipelines-install.json")) if os.IsNotExist(err) { return nil, nil diff --git a/scripts/configure-compose.py b/scripts/configure-compose.py index 0e1f7d88..86760d1d 100644 --- a/scripts/configure-compose.py +++ b/scripts/configure-compose.py @@ -1,8 +1,10 @@ #!/usr/bin/env python3 -"""Create local Compose settings without overwriting settings or credentials.""" +"""Create local Compose settings and the host client default.""" import argparse +import json import os from pathlib import Path +import tempfile from urllib.parse import urlparse from zoneinfo import ZoneInfo, ZoneInfoNotFoundError @@ -42,6 +44,36 @@ def host_workspaces(directory): return {"PIPELINES_HOST_HOME": "'"+home+"'", "PIPELINES_HOST_REPOSITORY": "'"+repository+"'", "PIPELINES_UID": str(os.getuid()), "PIPELINES_GID": str(os.getgid())} +def configure_client(endpoint, environ=None): + """Save the server address the host CLI/TUI dials by default. + + Atomic, 0600, and it preserves unrelated keys. Shared by both installers so + the saved default always matches the installation that just ran. + """ + environ = os.environ if environ is None else environ + root = Path(environ.get("XDG_CONFIG_HOME") or Path.home() / ".config") + path = root / "pipelines" / "client.json" + if path.is_symlink(): + raise ValueError("Refusing to modify symlinked client.json; use the printed --url explicitly") + existing = json.loads(path.read_text()) if path.exists() else {} + if not isinstance(existing, dict): + raise ValueError("client.json must contain an object; existing configuration was not replaced") + existing["server_url"] = endpoint + path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + temporary = None + try: + with tempfile.NamedTemporaryFile(mode="w", dir=str(path.parent), delete=False) as file: + temporary = Path(file.name) + os.chmod(temporary, 0o600) + json.dump(existing, file, indent=2) + file.write("\n") + os.replace(temporary, path) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + return path + + def configure(directory, ntfy_url, bind_address, timezone=None, port=8081): directory = Path(directory) url = urlparse(ntfy_url) diff --git a/scripts/manage.py b/scripts/manage.py index bad9a9c8..98e756b4 100644 --- a/scripts/manage.py +++ b/scripts/manage.py @@ -17,6 +17,15 @@ DEFAULT_PREFIX = Path.home() / ".local/share/pipelines" MARKER = "pipelines-install.json" +def ntfy_mode(config): + """Native ntfy is selected by the manifest; absent/explicit docker uses Compose.""" + return (config.get("ntfy") or {}).get("mode", "docker") + + +def ntfy_health_url(config): + return f"http://127.0.0.1:{config['ntfy_port']}/v1/health" + + def load(prefix): prefix = Path(prefix).expanduser().resolve() path = prefix / MARKER @@ -60,19 +69,27 @@ def wait_health(url, timeout=40): raise RuntimeError(f"Service did not become healthy: {url}; see pipelines logs") +def process_command(pid): + """Return the OS command line for a live PID, or None when it cannot be read.""" + try: + return subprocess.check_output(["ps", "-p", str(pid), "-o", "command="], text=True) + except (OSError, subprocess.CalledProcessError): + return None + + def managed_pid(config): path = Path(config["prefix"]) / "supervisor.pid" try: pid = int(path.read_text()) - if pid <= 1: - return None - command = subprocess.check_output(["ps", "-p", str(pid), "-o", "command="], text=True) - # Never signal an arbitrary/reused PID. - if str(Path(config["prefix"]) / "lib/manage.py") not in command or "supervise" not in command: - return None - return pid - except (OSError, ValueError, subprocess.CalledProcessError): + except (OSError, ValueError): return None + if pid <= 1: + return None + command = process_command(pid) + # Never signal an arbitrary/reused PID. + if command is None or str(Path(config["prefix"]) / "lib/manage.py") not in command or "supervise" not in command: + return None + return pid def registration(config): @@ -112,7 +129,48 @@ def stop(config): time.sleep(0.2) if managed_pid(config) is not None: raise RuntimeError("Supervisor did not stop; inspect logs before trying again") - compose(config, "stop", "ntfy", quiet=True) + if ntfy_mode(config) == "native": + ntfy_stop(config) + else: + compose(config, "stop", "ntfy", quiet=True) + + +def ntfy_start(config): + info = config["ntfy"] + log = (Path(config["prefix"]) / "logs/ntfy.log").open("a") + child = subprocess.Popen([info["bin"], "serve", "--config", info["config"]], + stdin=subprocess.DEVNULL, stdout=log, stderr=log, start_new_session=True) + log.close() + (Path(config["prefix"]) / "ntfy.pid").write_text(str(child.pid)) + return child + + +def ntfy_stop(config): + path = Path(config["prefix"]) / "ntfy.pid" + try: + pid = int(path.read_text()) + except (OSError, ValueError): + return + if pid <= 1: + return + command = process_command(pid) + # Never signal an arbitrary/reused PID: only the managed ntfy serve process. + # Accept the recorded path or its symlink-resolved form so a prefix reached + # through a symlinked HOME (macOS /var → /private/var) is still matched. + binary = str(config["ntfy"]["bin"]) + candidates = {binary, str(Path(binary).resolve())} + if command is None or "serve" not in command or not any(candidate in command for candidate in candidates): + path.unlink(missing_ok=True) + return + os.kill(pid, signal.SIGTERM) + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + try: + os.kill(pid, 0) + except ProcessLookupError: + break + time.sleep(0.1) + path.unlink(missing_ok=True) def supervise(config): @@ -126,18 +184,23 @@ def supervise(config): pidfile.write_text(str(os.getpid())) stopped = False child = None + ntfy_child = None def stopping(*_): nonlocal stopped stopped = True - if child is not None and child.poll() is None: - child.terminate() + for process in (child, ntfy_child): + if process is not None and process.poll() is None: + process.terminate() signal.signal(signal.SIGTERM, stopping) signal.signal(signal.SIGINT, stopping) try: - compose(config, "up", "-d", "ntfy", quiet=True) - wait_health(f"http://127.0.0.1:{config['ntfy_port']}/v1/health") + if ntfy_mode(config) == "native": + ntfy_child = ntfy_start(config) + else: + compose(config, "up", "-d", "ntfy", quiet=True) + wait_health(ntfy_health_url(config)) if stopped: return env = dict(os.environ) @@ -163,7 +226,17 @@ def supervise(config): except subprocess.TimeoutExpired: child.kill() child.wait() - compose(config, "stop", "ntfy", quiet=True, check=False) + if ntfy_mode(config) == "native": + if ntfy_child is not None and ntfy_child.poll() is None: + ntfy_child.terminate() + try: + ntfy_child.wait(timeout=10) + except subprocess.TimeoutExpired: + ntfy_child.kill() + ntfy_child.wait() + (root / "ntfy.pid").unlink(missing_ok=True) + else: + compose(config, "stop", "ntfy", quiet=True, check=False) pidfile.unlink(missing_ok=True) @@ -178,7 +251,8 @@ def uninstall(config, purge=False, yes=False): if platform.system() != "Darwin": run(["systemctl", "--user", "disable", config["label"] + ".service"], quiet=True, check=False) Path(registration(config)).unlink(missing_ok=True) - compose(config, "down", *(["--volumes"] if purge else []), quiet=True) + if ntfy_mode(config) != "native": + compose(config, "down", *(["--volumes"] if purge else []), quiet=True) link = Path(config["launcher"]) if link.is_symlink() and link.resolve() == (root / "bin/pipelines").resolve(): if str(link)=="/usr/local/bin/pipelines" and not os.access(link.parent,os.W_OK): @@ -231,7 +305,7 @@ def main(): stop(config); start(config) elif args.action in ("status", "doctor"): print("Pipeline server:", "healthy" if healthy(f"http://127.0.0.1:{config['server_port']}/healthz") else "stopped/unhealthy") - print("ntfy:", "healthy" if healthy(f"http://127.0.0.1:{config['ntfy_port']}/v1/health") else "stopped/unhealthy") + print("ntfy:", "healthy" if healthy(ntfy_health_url(config)) else "stopped/unhealthy") print("Phone server:", config["public_url"], "· topic:", config["topic"], "· username:", config["username"]) if args.action == "doctor": run([config["opencode"], "api", "get", "/api/info"], quiet=True) diff --git a/scripts/resolve-releases.py b/scripts/resolve-releases.py index 412bf380..d50561c7 100644 --- a/scripts/resolve-releases.py +++ b/scripts/resolve-releases.py @@ -1,10 +1,10 @@ #!/usr/bin/env python3 -"""Bounded resolution of current harness releases for the Docker image. +"""Bounded resolution of current harness and ntfy releases for installation. Network I/O is isolated behind an injectable ``fetch`` callable so unit tests never touch the network. The resolver: -* discovers supported V2 releases from one bounded GitHub request to +* discovers supported V2 OpenCode releases from one bounded GitHub request to ``git/matching-refs/tags/v2``, which the official REST docs (https://docs.github.com/en/rest/git/refs#list-matching-references) document as a complete array with no pagination parameters. There is no page/per_page @@ -15,9 +15,14 @@ never touch the network. The resolver: ones; it never uses the moving ``releases/latest`` anchor; * resolves the selected tag's immutable ``sha256`` digest from GHCR and fails closed when it is missing or invalid, never falling back to an older version; +* resolves the exact Pi package version and validates its Node engine + constraint fail-closed; +* resolves the exact ntfy release tag that the native installer builds its + server from, because upstream publishes only a client-only binary for macOS + and the server is therefore compiled from the pinned Go module on every host; * validates untrusted metadata, versions, digests and fixture values; -* refuses non-registry hosts and unexpected redirects; -* validates the Pi Node engine constraint fail-closed. +* refuses any host outside the explicit allowlist (the container registry, the + GitHub API and the npm registry) and refuses any redirect that leaves it. The selected versions/digests are data, not evergreen hardcoded answers. """ @@ -38,10 +43,18 @@ NPM = "https://registry.npmjs.org" OPENCODE_REPOSITORY = "anomalyco/opencode" OPENCODE_IMAGE = "ghcr.io/anomalyco/opencode" PI_PACKAGE = "@earendil-works/pi-coding-agent" +NTFY_REPOSITORY = "binwiederhier/ntfy" +# The ntfy Go module the native installer builds a server from. Upstream only +# publishes a client-only binary for macOS, so both platforms compile the server +# from this module at the resolved tag (Go verifies the module via its checksum +# database). +NTFY_MODULE = "heckel.io/ntfy/v2" MAX_REFS = 1000 # Explicit ceiling on the documented complete matching-refs array. MAX_BODY = 1 << 20 # Generous ceiling for tag lists and manifest metadata. SUPPORTED_MAJOR = 2 HTTP_TIMEOUT = 5.0 +# Registry, GitHub API and npm metadata hosts only. The native installer builds +# ntfy from its Go module, so no release-asset download hosts are needed. ALLOWED_HOSTS = frozenset({"ghcr.io", "api.github.com", "registry.npmjs.org"}) GITHUB_HEADERS = {"Accept": "application/vnd.github+json", "User-Agent": "pipelines-setup"} MANIFEST_ACCEPT = ", ".join([ @@ -72,19 +85,26 @@ def parse_version(text): return tuple(int(part) for part in match.groups()) -class _NoRedirect(urllib.request.HTTPRedirectHandler): - """Registry metadata lookups must never be silently redirected elsewhere.""" +class _AllowlistRedirect(urllib.request.HTTPRedirectHandler): + """Follow only redirects that stay on an explicitly allowed HTTPS host.""" def redirect_request(self, req, fp, code, msg, headers, newurl): - raise ReleaseError(f"refusing unexpected redirect from {req.full_url!r} to {newurl!r}") + if not is_allowlisted(newurl): + raise ReleaseError(f"refusing unexpected redirect from {req.full_url!r} to {newurl!r}") + return super().redirect_request(req, fp, code, msg, headers, newurl) -_OPENER = urllib.request.build_opener(_NoRedirect) +_OPENER = urllib.request.build_opener(_AllowlistRedirect) + + +def is_allowlisted(url): + """True only for HTTPS URLs on an explicitly allowed host (shared guard).""" + parsed = urllib.parse.urlsplit(url) + return parsed.scheme == "https" and parsed.hostname in ALLOWED_HOSTS def _default_fetch(url, headers=None): - parsed = urllib.parse.urlsplit(url) - if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS: + if not is_allowlisted(url): raise ReleaseError(f"refusing non-registry host {url!r}") request = urllib.request.Request(url, headers=headers or {}) try: @@ -118,15 +138,16 @@ def _json(fetch, url, headers=None): return data -def _github_v2_tags(fetch): +def _github_matching_refs(fetch, repository, tag_prefix, normalize=lambda tag: tag): """One bounded request for the documented complete matching-refs array. GitHub's "List matching references" endpoint documents a complete array and no pagination parameters, so a single ``MAX_BODY``/``MAX_REFS``-bounded GET is sufficient. Exact stable versions become candidates; version-shaped tags with a prerelease/build suffix are ignored; other malformed tags fail closed. + ``normalize`` maps a raw tag to the candidate key (OpenCode strips the ``v``). """ - url = f"{GITHUB}/repos/{OPENCODE_REPOSITORY}/git/matching-refs/tags/v{SUPPORTED_MAJOR}" + url = f"{GITHUB}/repos/{repository}/git/matching-refs/tags/{tag_prefix}" data, _ = _response_json(fetch, url, GITHUB_HEADERS) if not isinstance(data, list): raise ReleaseError("GitHub matching-refs payload is not a list") @@ -141,7 +162,7 @@ def _github_v2_tags(fetch): ref = entry.get("ref") if not isinstance(ref, str) or not ref.startswith("refs/tags/"): raise ReleaseError(f"GitHub matching-refs entry has an unexpected ref {ref!r}") - tag = registry_tag(ref[len("refs/tags/"):]) + tag = normalize(ref[len("refs/tags/"):]) version = parse_version(tag) if version is None: if VERSION_SUFFIX.fullmatch(tag): @@ -152,6 +173,10 @@ def _github_v2_tags(fetch): return candidates +def _github_v2_tags(fetch): + return _github_matching_refs(fetch, OPENCODE_REPOSITORY, f"v{SUPPORTED_MAJOR}", registry_tag) + + def _ghcr_token(fetch): data = _json(fetch, f"{GHCR}/token?scope=repository:{OPENCODE_REPOSITORY}:pull") if not isinstance(data, dict): @@ -247,6 +272,30 @@ def resolve_pi(fetch=_default_fetch, node_version=None): return {"package": PI_PACKAGE, "version": version, "engine": engine} +def _github_ntfy_tags(fetch): + """One bounded request for the complete ntfy vX.Y.Z matching-refs array.""" + return _github_matching_refs(fetch, NTFY_REPOSITORY, "v") + + +def resolve_ntfy(fetch=_default_fetch): + """Resolve the exact ntfy release the native installer builds from source. + + The selected version comes from the complete official matching-refs tag list + (never the moving ``latest`` release and never an older fallback); a missing + or malformed tag list fails closed. The native installer then compiles the + server from the pinned ``heckel.io/ntfy/v2`` module, so no release assets or + their checksums are downloaded here. + """ + candidates = _github_ntfy_tags(fetch) + if not candidates: + raise ReleaseError("no stable ntfy release found in the official tag metadata") + tag = max(candidates, key=lambda name: candidates[name]) + version = registry_tag(tag) + if parse_version(version) is None: + raise ReleaseError(f"resolved ntfy tag is not an exact version: {tag!r}") + return {"repository": NTFY_REPOSITORY, "module": NTFY_MODULE, "version": version, "tag": tag} + + def resolve(fetch=_default_fetch, node_version=None): return {"opencode": resolve_opencode(fetch), "pi": resolve_pi(fetch, node_version)} diff --git a/scripts/save-ntfy-token.py b/scripts/save-ntfy-token.py index a48e3f2a..42ea320a 100644 --- a/scripts/save-ntfy-token.py +++ b/scripts/save-ntfy-token.py @@ -1,33 +1,66 @@ #!/usr/bin/env python3 -"""Extract only the token from ntfy's CLI response; never print the credential.""" +"""Shared ntfy credential helpers: token extraction and account bootstrap. + +Both installers (native and Docker) create the same publisher/reader accounts on +ntfy, so the extraction, symlink/non-empty guard and writing live here once. +Never print a credential. +""" import os from pathlib import Path import re import sys +TOKEN_PATTERN = re.compile(r"\btk_[A-Za-z0-9]{29}\b") -def save_token(output, directory): - tokens = re.findall(r"\btk_[A-Za-z0-9]{29}\b", output) + +def save_token(output, path, mode=0o444): + """Write the single token found in ``output`` to ``path``. + + ``mode`` is 0444 for Compose's file secret (read by container uid 10001) and + 0600 for the native install, whose server runs as the installing user. + """ + tokens = TOKEN_PATTERN.findall(output or "") if len(tokens) != 1: raise ValueError("ntfy did not return exactly one token; existing credentials were not changed") - directory = Path(directory) - secrets = directory / "secrets" - secrets.mkdir(mode=0o700, exist_ok=True) - secrets.chmod(0o700) - path = secrets / "ntfy-token.txt" + path = Path(path) + path.parent.mkdir(mode=0o700, exist_ok=True) + path.parent.chmod(0o700) if path.is_symlink() or (path.exists() and path.stat().st_size != 0): raise ValueError("Token file already contains credentials; refusing to overwrite") fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) with os.fdopen(fd, "w") as file: file.write(tokens[0] + "\n") - # Compose's file secret is a bind mount. Container uid 10001 needs read access; - # the private host parent directory restricts access by other host users. - path.chmod(0o444) + path.chmod(mode) + return path + + +def bootstrap_accounts(run, token_path, mode=0o444, log=print): + """Create the pipeline-publisher and phone-reader accounts on an ntfy server. + + ``run(args, capture=False, interactive=False)`` executes the ntfy CLI (directly + or through Compose) and returns the completed process. Existing accounts and an + existing token file are preserved on reinstall. + """ + token_path = Path(token_path) + if token_path.is_symlink(): + raise ValueError("Refusing a symlink for the publishing token") + if token_path.exists() and token_path.stat().st_size: + log("Keeping existing ntfy publishing token.") + else: + log("Set a password for ntfy's pipeline-publisher account (not a model-provider account).") + run(["user", "add", "--ignore-exists", "pipeline-publisher"], interactive=True) + run(["access", "pipeline-publisher", "pipelines", "write-only"]) + result = run(["token", "add", "pipeline-publisher"], capture=True) + save_token(result.stdout + result.stderr, token_path, mode) + # --ignore-exists preserves the phone account's password on reinstall. + log("Set the phone-reader password if this is a new ntfy account; existing accounts are preserved.") + run(["user", "add", "--ignore-exists", "phone-reader"], interactive=True) + run(["access", "phone-reader", "pipelines", "read-only"]) if __name__ == "__main__": try: - save_token(sys.stdin.read(), Path(__file__).resolve().parents[1]) + save_token(sys.stdin.read(), Path(__file__).resolve().parents[1] / "secrets/ntfy-token.txt", 0o444) except ValueError as error: sys.exit(str(error)) print("Saved ntfy publishing token; restart pipelines if it was already running.") diff --git a/scripts/setup-docker.py b/scripts/setup-docker.py index e1eabb4c..9ace9099 100644 --- a/scripts/setup-docker.py +++ b/scripts/setup-docker.py @@ -105,21 +105,13 @@ class Setup: print("Use this address on your phone on the same trusted LAN; do not forward this HTTP port to the Internet.") def bootstrap_ntfy(self): - token = self.directory / "secrets/ntfy-token.txt" - if token.is_symlink(): - raise ValueError("Refusing a symlink for the publishing token") - if token.exists() and token.stat().st_size: - print("Keeping existing ntfy publishing token.") - else: - print("Set a password for ntfy's pipeline-publisher account (not a model-provider account).") - self.compose("exec", "ntfy", "ntfy", "user", "add", "--ignore-exists", "pipeline-publisher") - self.compose("exec", "-T", "ntfy", "ntfy", "access", "pipeline-publisher", "pipelines", "write-only") - result = self.compose("exec", "-T", "ntfy", "ntfy", "token", "add", "pipeline-publisher", capture=True) - tokens.save_token(result.stdout + result.stderr, self.directory) - # --ignore-exists preserves the phone account's password on re-install. - print("Set the phone-reader password if this is a new ntfy account; existing accounts are preserved.") - self.compose("exec", "ntfy", "ntfy", "user", "add", "--ignore-exists", "phone-reader") - self.compose("exec", "-T", "ntfy", "ntfy", "access", "phone-reader", "pipelines", "read-only") + def run_ntfy(args, capture=False, interactive=False): + # -T is required for the non-interactive calls; the interactive + # `user add` password prompts need the allocated TTY. + exec_args = ["exec"] + ([] if interactive else ["-T"]) + ["ntfy", "ntfy", *args] + return self.compose(*exec_args, capture=capture) + + tokens.bootstrap_accounts(run_ntfy, self.directory / "secrets/ntfy-token.txt", 0o444) def authenticate(self, relogin=False): def connected(): @@ -554,27 +546,7 @@ printf 'header = "Authorization: Bearer %s"\\n' "$(cat "$PIPELINES_NTFY_TOKEN_FI print("Verified: host client reaches the ntfy-enabled Docker server at " + endpoint) def configure_client(self, endpoint): - root = Path(os.environ.get("XDG_CONFIG_HOME") or Path.home()/".config") - directory = root/"pipelines" - path = directory/"client.json" - if path.is_symlink(): - raise ValueError("Refusing to modify symlinked client.json; use the printed --url explicitly") - existing = json.loads(path.read_text()) if path.exists() else {} - if not isinstance(existing, dict): - raise ValueError("client.json must contain an object; existing configuration was not replaced") - existing["server_url"] = endpoint - directory.mkdir(mode=0o700, parents=True, exist_ok=True) - temporary = None - try: - with tempfile.NamedTemporaryFile(mode="w", dir=directory, delete=False) as file: - temporary = Path(file.name) - os.chmod(temporary, 0o600) - json.dump(existing, file, indent=2) - file.write("\n") - os.replace(temporary, path) - finally: - if temporary is not None: - temporary.unlink(missing_ok=True) + path = configure.configure_client(endpoint) print(f"Saved Docker client default: {endpoint} ({path}). Restart the TUI.") override = os.environ.get("PIPELINES_URL", "") if override and override != endpoint: diff --git a/scripts/setup-native.py b/scripts/setup-native.py new file mode 100644 index 00000000..fb3f31ec --- /dev/null +++ b/scripts/setup-native.py @@ -0,0 +1,465 @@ +#!/usr/bin/env python3 +"""Install Pipelines natively on the host (macOS and Linux). + +The installer only detects the tools it needs (go, python3, git, curl and a C +compiler for ntfy's SQLite driver); it never installs or upgrades them, and it +never installs, upgrades or downloads OpenCode or Pi. It builds the server, CLI +and ntfy from source into a per-user prefix, generates an ntfy configuration that +mirrors compose.yaml, creates the publishing/phone accounts and writes the +manifest scripts/manage.py loads. ntfy is compiled from its pinned Go module on +every host because upstream publishes no macOS server binary. It reuses existing +host OpenCode configuration; model selection stays at runtime and is never +invented here. +""" +import argparse +import importlib.util +import json +import os +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +from zoneinfo import ZoneInfoNotFoundError + + +def module(name, file): + spec = importlib.util.spec_from_file_location(name, Path(__file__).with_name(file)) + result = importlib.util.module_from_spec(spec) + spec.loader.exec_module(result) + return result + + +configure = module("configure_compose", "configure-compose.py") +lan = module("lan_address", "lan-address.py") +release_lookup = module("resolve_releases", "resolve-releases.py") +tokens = module("save_ntfy_token", "save-ntfy-token.py") + +DEFAULT_PREFIX = Path.home() / ".local/share/pipelines" +MARKER = "pipelines-install.json" +LABEL = "net.a-wan.pipelines" +REQUIRED_TOOLS = ("go", "python3", "git", "curl") +# ntfy's SQLite driver (mattn/go-sqlite3) is cgo, so a C compiler is required to +# build the server. Upstream's macOS release asset is client-only, so both +# platforms compile ntfy from the pinned Go module. +C_COMPILERS = ("cc", "clang", "gcc") +NTFY_BUILD_TAGS = "sqlite_omit_load_extension,osusergo,netgo" +DEFAULT_NTFY_PORT = 8090 +DEFAULT_SERVER_PORT = 8080 + + +class Setup: + def __init__(self, directory, prefix=None, runner=subprocess.run, resolver=None, environ=None): + self.directory = Path(directory) + self.prefix_argument = prefix + self.runner = runner + self.resolver = resolver or release_lookup.resolve_ntfy + self.environ = environ if environ is not None else os.environ + + # --- detection ------------------------------------------------------- + + def require_compiler(self): + """ntfy is built locally, so a C toolchain must already be present.""" + for compiler in C_COMPILERS: + path = shutil.which(compiler) + if path: + return str(Path(path).resolve()) + raise ValueError( + "No C compiler found (tried " + ", ".join(C_COMPILERS) + + "). ntfy's SQLite driver is cgo, so the server is compiled locally; " + "install a C toolchain (Xcode Command Line Tools on macOS, gcc or clang on Linux) and rerun.") + + def require_tools(self): + found, missing = {}, [] + for tool in REQUIRED_TOOLS: + path = shutil.which(tool) + if path: + found[tool] = str(Path(path).resolve()) + else: + missing.append(tool) + if missing: + raise ValueError( + "Missing required tool(s): " + ", ".join(missing) + + ". Install them yourself and rerun; this installer never installs tools or uses a package manager.") + return found["python3"] + + def harness(self): + """Detect existing harnesses only; never install, upgrade or download them.""" + opencode = shutil.which("opencode") + if not opencode: + raise ValueError( + "OpenCode was not found on PATH. Install OpenCode yourself (this installer never installs, " + "upgrades or downloads it) and rerun; model selection happens in your existing OpenCode configuration.") + pi = shutil.which("pi") + return {"opencode": str(Path(opencode).resolve()), "pi": str(Path(pi).resolve()) if pi else ""} + + def resolve_prefix(self): + default = Path.home() / ".local/share/pipelines" + raw = self.prefix_argument or self.environ.get("PIPELINES_PREFIX") or default + return Path(raw).expanduser().resolve() + + # --- prefix layout --------------------------------------------------- + + def prepare_layout(self, prefix): + prefix.mkdir(parents=True, exist_ok=True) + for name in ("bin", "lib", "data", "logs", "secrets"): + (prefix / name).mkdir(exist_ok=True) + (prefix / "secrets").chmod(0o700) + (prefix / "data").chmod(0o700) + source = Path(__file__).with_name("manage.py") + target = prefix / "lib/manage.py" + if target.is_symlink(): + raise ValueError("Refusing to replace a symlinked lib/manage.py") + shutil.copyfile(source, target) + os.chmod(target, 0o700) + return target + + # --- build and skill ------------------------------------------------- + + def build(self, prefix): + for command in ("pipelines-server", "pipelines"): + result = self.runner( + ["go", "build", "-trimpath", "-o", str(prefix / "bin" / command), "./cmd/" + command], + cwd=str(self.directory)) + if result.returncode: + raise ValueError(f"go build failed for ./cmd/{command}; the native install is incomplete") + + def install_skill(self, prefix): + result = self.runner([str(prefix / "bin/pipelines"), "skill", "install"], cwd=str(self.directory)) + if result.returncode: + raise ValueError("pipelines skill install failed; the native install is incomplete") + + # --- ntfy server build and configuration ---------------------------- + + def resolve_ntfy(self): + try: + return self.resolver() + except release_lookup.ReleaseError as error: + raise ValueError( + f"Could not resolve the ntfy release ({error}); refusing to build an unverified version") from None + + def build_ntfy(self, prefix, resolution): + """Compile the ntfy server from the pinned Go module on this host. + + Upstream publishes no macOS server binary (its darwin asset is built with + ``-tags noserver``), so every platform builds here. The module is fetched + through the Go toolchain, which verifies it against the public checksum + database, and cgo is required for ntfy's SQLite driver. + """ + path = prefix / "bin/ntfy" + if path.is_symlink(): + raise ValueError("Refusing to replace a symlinked bin/ntfy") + version = resolution.get("tag") or resolution.get("version") + module = resolution.get("module") or release_lookup.NTFY_MODULE + # A previously built binary for the same resolved version is reused. + if path.is_file() and self._existing_ntfy_version(prefix) == version: + return path + self.require_compiler() + pinned = f"{module}@{version}" + result = self.runner(["go", "mod", "download", pinned], cwd=str(self.directory)) + if result.returncode: + raise ValueError(f"go mod download {pinned} failed; the native install is incomplete") + cache = self.runner(["go", "env", "GOMODCACHE"], cwd=str(self.directory), + capture_output=True, text=True) + if cache.returncode or not (cache.stdout or "").strip(): + raise ValueError("Could not locate the Go module cache to build ntfy from") + source = Path(cache.stdout.strip()) / f"{module}@{version}" + if not source.is_dir(): + raise ValueError(f"the Go module cache has no source for {pinned}") + with tempfile.TemporaryDirectory(prefix="pipelines-ntfy-") as temporary: + checkout = Path(temporary) / "src" + shutil.copytree(source, checkout, symlinks=True) + # The module cache is read-only and carries no built web assets; the + # upstream Makefile creates the same placeholders for local builds. + self.runner(["chmod", "-R", "u+w", str(checkout)]) + (checkout / "server/docs").mkdir(parents=True, exist_ok=True) + (checkout / "server/site").mkdir(parents=True, exist_ok=True) + (checkout / "server/docs/index.html").touch() + (checkout / "server/site/app.html").touch() + built = Path(temporary) / "ntfy" + env = dict(self.environ) + env["CGO_ENABLED"] = "1" + build = self.runner( + ["go", "build", "-trimpath", "-tags", NTFY_BUILD_TAGS, + "-ldflags", f"-s -w -X main.version={version}", "-o", str(built), "."], + cwd=str(checkout), env=env) + if build.returncode: + raise ValueError( + "ntfy server build failed; a working C compiler is required for its SQLite driver") + if not built.is_file(): + raise ValueError("ntfy server build produced no binary") + os.chmod(built, 0o755) + os.replace(built, path) + return path + + def _existing_ntfy_version(self, prefix): + manifest = prefix / MARKER + if manifest.is_symlink() or not manifest.is_file(): + return None + try: + data = json.loads(manifest.read_text()) + except (OSError, ValueError): + return None + return (data.get("ntfy") or {}).get("version") if isinstance(data, dict) else None + + def write_ntfy_config(self, prefix, base_url, ntfy_port): + directory = prefix / "data/ntfy" + directory.mkdir(mode=0o700, exist_ok=True) + directory.chmod(0o700) + path = directory / "server.yml" + if path.is_symlink(): + raise ValueError("Refusing to replace a symlinked ntfy configuration") + text = ( + f'base-url: "{base_url}"\n' + f'listen-http: ":{ntfy_port}"\n' + f'cache-file: "{directory}/cache.db"\n' + 'cache-duration: "168h"\n' + f'auth-file: "{directory}/auth.db"\n' + 'auth-default-access: "deny-all"\n' + 'enable-login: true\n' + f'attachment-cache-dir: "{directory}/attachments"\n' + 'attachment-expiry-duration: "168h"\n' + 'attachment-total-size-limit: "1G"\n' + 'attachment-file-size-limit: "1M"\n' + ) + self._write_atomic(path, text.encode(), 0o600) + return path + + # --- ntfy users and publishing token -------------------------------- + + def ntfy(self, prefix, config_path, args, capture=False): + # The account subcommands locate the server configuration through + # NTFY_CONFIG_FILE; --config is a `serve`-only flag. + env = dict(self.environ) + env["NTFY_CONFIG_FILE"] = str(config_path) + result = self.runner([str(prefix / "bin/ntfy"), *args], capture_output=capture, text=True, env=env) + if result.returncode: + step = " ".join(args[:2]) + raise ValueError(f"ntfy {step} failed; the native install is incomplete") + return result + + def bootstrap_ntfy(self, prefix, config_path): + def run_ntfy(args, capture=False, interactive=False): + return self.ntfy(prefix, config_path, list(args), capture=capture) + + tokens.bootstrap_accounts(run_ntfy, prefix / "secrets/publisher-token", 0o600) + + # --- Docker migration ------------------------------------------------ + + def docker_volumes(self): + try: + result = self.runner(["docker", "volume", "ls", "--format", "{{.Name}}"], + capture_output=True, text=True) + except FileNotFoundError: + # No Docker CLI at all: there is no Docker installation to migrate. + return () + except OSError as error: + raise ValueError( + f"Could not inspect Docker volumes for migration ({error}); refusing to start empty") from None + if result.returncode: + # A present-but-unusable Docker daemon must not silently skip migration: + # Docker data may still exist, so fail instead of starting empty. + raise ValueError( + "Could not inspect Docker volumes for migration; refusing a possibly empty start") + names = set(result.stdout.split()) + return tuple(name for name in ("pipelines_pipelines-data", "pipelines_ntfy-data") if name in names) + + def _migration_image(self): + result = self.runner(["docker", "image", "ls", "--format", "{{.Repository}}:{{.Tag}}"], + capture_output=True, text=True) + if result.returncode: + return None + for line in result.stdout.splitlines(): + name = line.strip() + repository = name.rsplit(":", 1)[0] + if repository == "pipelines" or repository.endswith("/pipelines") or repository.endswith("-pipelines"): + return name + return None + + def _docker_copy_out(self, image, volume, inner, destination, replace): + if destination.exists() and not replace: + print(f"Keeping existing native {destination.name}; not overwriting it during migration.") + return False + name = f"pipelines-migrate-{os.getpid()}" + created = self.runner(["docker", "create", "--name", name, "--volume", f"{volume}:/mnt:ro", image], + capture_output=True, text=True) + if created.returncode: + raise ValueError(f"Could not open Docker volume {volume} for migration; install aborted") + try: + result = self.runner(["docker", "cp", f"{name}:/mnt/{inner}", str(destination)], + capture_output=True, text=True) + if result.returncode: + raise ValueError(f"Could not copy {inner} from Docker volume {volume}; install aborted") + # The live server keeps recent commits in the SQLite write-ahead log, + # so copy the -wal sidecar too or the newest definitions and runs are + # lost. SQLite recovers it on next open and rebuilds -shm itself. + for suffix in ("-wal",): + sidecar = Path(str(destination) + suffix) + sidecar.unlink(missing_ok=True) + copy = self.runner(["docker", "cp", f"{name}:/mnt/{inner}{suffix}", str(sidecar)], + capture_output=True, text=True) + if copy.returncode or not sidecar.exists() or sidecar.stat().st_size == 0: + sidecar.unlink(missing_ok=True) + finally: + self.runner(["docker", "rm", "-f", name], capture_output=True, text=True, check=False) + return True + + def migrate_docker(self, prefix, replace=False): + volumes = self.docker_volumes() + if not volumes: + return False + image = self._migration_image() + if not image: + raise ValueError( + "Docker volumes exist but no local Pipelines image is available to read them; " + "refusing to pull an image. Build the Docker image or rerun without migration.") + (prefix / "data").mkdir(exist_ok=True) + if "pipelines_pipelines-data" in volumes: + self._docker_copy_out(image, "pipelines_pipelines-data", "pipelines.db", + prefix / "data/pipelines.db", replace) + if "pipelines_ntfy-data" in volumes: + directory = prefix / "data/ntfy" + directory.mkdir(mode=0o700, exist_ok=True) + self._docker_copy_out(image, "pipelines_ntfy-data", "auth.db", + directory / "auth.db", replace) + print("Existing Docker data was migrated where it did not already exist.") + return True + + # --- manifest -------------------------------------------------------- + + def write_manifest(self, prefix, values): + path = prefix / MARKER + if path.is_symlink(): + raise ValueError("Refusing to replace a symlinked installation manifest") + self._write_atomic(path, (json.dumps(values, indent=2, sort_keys=True) + "\n").encode(), 0o600) + return path + + def _write_atomic(self, path, data, mode): + handle, temporary = tempfile.mkstemp(dir=str(path.parent), prefix=".pipelines-") + try: + os.chmod(temporary, mode) + with os.fdopen(handle, "wb") as file: + file.write(data) + if path.is_symlink(): + raise ValueError(f"Refusing to replace a symlinked file {path}") + os.replace(temporary, path) + finally: + if os.path.exists(temporary): + os.unlink(temporary) + + # --- run ------------------------------------------------------------- + + def run(self, args): + python = self.require_tools() + if args.prefix: + self.prefix_argument = args.prefix + if not 1024 <= args.port <= 65535: + raise ValueError("Invalid server port; choose a value between 1024 and 65535") + if not 1024 <= args.ntfy_port <= 65535: + raise ValueError("Invalid ntfy port; choose a value between 1024 and 65535") + if args.ntfy_port == args.port: + raise ValueError("The server and ntfy ports must differ") + prefix = self.resolve_prefix() + self.prepare_layout(prefix) + harness = self.harness() + # Reuse an already built ntfy binary and skip the release lookup, so a + # reinstall with the same version makes no network requests at all. + installed = self._existing_ntfy_version(prefix) + if (prefix / "bin/ntfy").is_file() and installed: + resolution = {"module": release_lookup.NTFY_MODULE, "version": installed, "tag": installed} + else: + resolution = self.resolve_ntfy() + self.build(prefix) + self.install_skill(prefix) + base_url = args.ntfy_url or f"http://{lan.detect_lan()}:{args.ntfy_port}" + config_path = self.write_ntfy_config(prefix, base_url, args.ntfy_port) + self.build_ntfy(prefix, resolution) + # Migrate existing Docker data before creating native ntfy accounts so a + # migrated auth store (the phone's credentials) is preserved rather than + # shadowed by a freshly bootstrapped one. + self.migrate_docker(prefix, args.replace_data) + self.bootstrap_ntfy(prefix, config_path) + timezone = args.timezone + if not timezone: + try: + timezone = configure.host_timezone() + except (ValueError, ZoneInfoNotFoundError): + timezone = "UTC" + opencode_env = {} + for key in ("XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME", "XDG_CACHE_HOME", + "OPENCODE_SERVER", "OPENCODE_DB"): + if self.environ.get(key): + opencode_env[key] = self.environ[key] + if args.model: + opencode_env["PIPELINES_DEFAULT_MODEL"] = args.model + path_value = str(prefix / "bin") + os.pathsep + self.environ.get("PATH", "") + manifest = { + "format": 1, + "prefix": str(prefix), + "python": python, + "server_port": args.port, + "ntfy_port": args.ntfy_port, + "opencode": harness["opencode"], + "pi": harness["pi"], + "opencode_env": opencode_env, + "path": path_value, + "topic": "pipelines", + "public_url": base_url, + "username": "phone-reader", + "workspace": str(Path.home()), + "launcher": str(prefix / "bin/pipelines"), + "registration": None, + "label": LABEL, + "docker": "docker", + "project": "pipelines", + "timezone": timezone, + "ntfy": {"mode": "native", "bin": str(prefix / "bin/ntfy"), + "config": str(config_path), "base_url": base_url, + "version": resolution.get("tag") or resolution.get("version")}, + } + self.write_manifest(prefix, manifest) + # Save the address the host CLI/TUI dials by default, so a plain + # `pipelines` (or its TUI) reaches this installation instead of a stale + # Docker address. + endpoint = f"ws://127.0.0.1:{args.port}/connect" + client_path = configure.configure_client(endpoint, self.environ) + print(f"Saved native client default: {endpoint} ({client_path}). Restart the TUI.") + override = self.environ.get("PIPELINES_URL", "") + if override and override != endpoint: + print("WARNING: PIPELINES_URL overrides the saved native address; unset it or pass --url explicitly.") + print(f"Native Pipelines installed at {prefix}." + f" ntfy {resolution.get('tag') or resolution.get('version')} was built from source.") + # Parity with the Docker installer: leave the server and ntfy running. + started = self.runner([python, str(prefix / "lib/manage.py"), "start", "--prefix", str(prefix)]) + if started.returncode: + raise ValueError( + "The native services were installed but could not be started; run " + f"'{prefix}/bin/pipelines start' to see the error. Check that ports " + f"{args.port} and {args.ntfy_port} are free (a running Docker installation uses them).") + print(f"ntfy is configured for {base_url} and both services are running.") + print("Create the phone account password if it is a new account, then subscribe to the pipelines topic.") + + +def arguments(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--prefix", type=Path, help="Installation prefix (default: ~/.local/share/pipelines)") + parser.add_argument("--ntfy-url", help="Override the auto-detected LAN ntfy address") + parser.add_argument("--timezone", help="Override the detected timezone (fallback: UTC)") + parser.add_argument("--port", type=int, default=DEFAULT_SERVER_PORT, help="Pipelines server port") + parser.add_argument("--ntfy-port", type=int, default=DEFAULT_NTFY_PORT, help="ntfy listen port") + parser.add_argument("--model", help="Record an explicit default model; otherwise host OpenCode configuration is reused") + parser.add_argument("--replace-data", action="store_true", + help="Allow Docker migration to replace an existing native database/auth store") + return parser.parse_args(argv) + + +if __name__ == "__main__": + try: + Setup(Path(__file__).resolve().parents[1]).run(arguments()) + except KeyboardInterrupt: + print("\nNative setup canceled; existing configuration and data were retained.", file=sys.stderr) + sys.exit(130) + except (ValueError, OSError, json.JSONDecodeError, subprocess.SubprocessError, + ZoneInfoNotFoundError) as error: + sys.exit(f"Native setup incomplete: {error}") diff --git a/scripts/test_configure_compose.py b/scripts/test_configure_compose.py index e48b2af7..5050947a 100644 --- a/scripts/test_configure_compose.py +++ b/scripts/test_configure_compose.py @@ -1,4 +1,5 @@ import importlib.util +import json import os from pathlib import Path import tempfile @@ -52,6 +53,35 @@ class ConfigureTests(unittest.TestCase): with tempfile.TemporaryDirectory() as directory, self.assertRaises(ValueError): configure.configure(directory, "https://example.com", "0.0.0.0\nEVIL=yes", "Europe/London") + def test_client_default_is_saved_privately_and_preserves_other_keys(self): + with tempfile.TemporaryDirectory() as directory: + environ = {"XDG_CONFIG_HOME": directory} + path = configure.configure_client("ws://127.0.0.1:8080/connect", environ) + self.assertEqual(path, Path(directory) / "pipelines/client.json") + self.assertEqual(path.stat().st_mode & 0o777, 0o600) + self.assertEqual(json.loads(path.read_text())["server_url"], "ws://127.0.0.1:8080/connect") + # Unrelated keys survive a later reinstall with a different address. + path.write_text(json.dumps({"server_url": "ws://old/connect", "theme": "dark"})) + configure.configure_client("ws://127.0.0.1:9090/connect", environ) + saved = json.loads(path.read_text()) + self.assertEqual(saved["server_url"], "ws://127.0.0.1:9090/connect") + self.assertEqual(saved["theme"], "dark") + + def test_client_default_refuses_symlink_and_non_object(self): + with tempfile.TemporaryDirectory() as directory: + environ = {"XDG_CONFIG_HOME": directory} + target = Path(directory) / "real.json" + target.write_text("{}") + directory_path = Path(directory) / "pipelines" + directory_path.mkdir() + (directory_path / "client.json").symlink_to(target) + with self.assertRaisesRegex(ValueError, "symlinked client.json"): + configure.configure_client("ws://127.0.0.1:8080/connect", environ) + (directory_path / "client.json").unlink() + (directory_path / "client.json").write_text("[]") + with self.assertRaisesRegex(ValueError, "must contain an object"): + configure.configure_client("ws://127.0.0.1:8080/connect", environ) + if __name__ == "__main__": unittest.main() diff --git a/scripts/test_installation.py b/scripts/test_installation.py index 1748acfa..38d4231d 100644 --- a/scripts/test_installation.py +++ b/scripts/test_installation.py @@ -7,13 +7,19 @@ import manage class InstallationTests(unittest.TestCase): - def test_docker_installer_is_available_through_main_entrypoint(self): + def test_native_installer_is_the_default_entrypoint(self): result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),"--help"],capture_output=True,text=True) self.assertEqual(result.returncode,0) + self.assertIn("--prefix",result.stdout) + self.assertIn("--ntfy-url",result.stdout) + + def test_docker_installer_is_available_explicitly(self): + result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),"--docker","--help"],capture_output=True,text=True) + self.assertEqual(result.returncode,0) self.assertIn("--ntfy-url",result.stdout) self.assertIn("--relogin",result.stdout) - def test_default_installer_delegates_to_docker_without_terminal(self): + def dispatch(self,*args): with tempfile.TemporaryDirectory() as directory: root=Path(directory) capture=root/"args" @@ -21,27 +27,21 @@ class InstallationTests(unittest.TestCase): executable.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > "'+str(capture)+'"\n') executable.chmod(0o700) env=dict(os.environ,PATH=str(root)+os.pathsep+os.environ["PATH"]) - subprocess.run([str(Path(__file__).resolve().parents[1]/"install")],env=env,stdin=subprocess.DEVNULL,check=True) - self.assertEqual(capture.read_text().splitlines(),["scripts/setup-docker.py"]) + subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),*args],env=env,stdin=subprocess.DEVNULL,check=True) + return capture.read_text().splitlines() - def test_default_installer_forwards_docker_options(self): - result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install.sh"),"--help"],capture_output=True,text=True) - self.assertEqual(result.returncode,0) - self.assertIn("--ntfy-url",result.stdout) + def test_default_installer_delegates_to_native_without_terminal(self): + self.assertEqual(self.dispatch(),["scripts/setup-native.py"]) - def test_native_installer_is_rejected_without_running_go(self): - with tempfile.TemporaryDirectory() as directory: - root=Path(directory) - capture=root/"args" - executable=root/"go" - executable.write_text('#!/bin/sh\nprintf "%s\\n" "$GOBIN" "$@" >> "'+str(capture)+'"\n') - executable.chmod(0o700) - env=dict(os.environ,PATH=str(root)+os.pathsep+os.environ["PATH"],GOBIN=str(root/"bin")) - result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install"),"--native"],env=env,capture_output=True,text=True) - self.assertEqual(result.returncode,2) - self.assertIn("Run ./install",result.stderr) - self.assertFalse(capture.exists()) - self.assertFalse((root/"bin").exists()) + def test_explicit_modes_dispatch_to_the_right_installer(self): + self.assertEqual(self.dispatch("--native"),["scripts/setup-native.py"]) + self.assertEqual(self.dispatch("native"),["scripts/setup-native.py"]) + self.assertEqual(self.dispatch("--docker"),["scripts/setup-docker.py"]) + self.assertEqual(self.dispatch("docker"),["scripts/setup-docker.py"]) + + def test_docker_mode_forwards_docker_options(self): + self.assertEqual(self.dispatch("--docker","--ntfy-url","http://192.168.1.20:8090"), + ["scripts/setup-docker.py","--ntfy-url","http://192.168.1.20:8090"]) def test_standard_installer_rejects_legacy_provisioning_flags(self): result=subprocess.run([str(Path(__file__).resolve().parents[1]/"install.sh"),"--system-command"],capture_output=True,text=True) diff --git a/scripts/test_manage_native.py b/scripts/test_manage_native.py new file mode 100644 index 00000000..367459db --- /dev/null +++ b/scripts/test_manage_native.py @@ -0,0 +1,97 @@ +import importlib.util +import os +from pathlib import Path +import signal +import tempfile +import unittest +from unittest.mock import patch + +spec = importlib.util.spec_from_file_location("manage_native", Path(__file__).with_name("manage.py")) +manage = importlib.util.module_from_spec(spec) +spec.loader.exec_module(manage) + + +class FakeProcess: + def __init__(self, pid=4321): + self.pid = pid + + def poll(self): + return None + + +class ManageNativeTests(unittest.TestCase): + def config(self, root, mode="native"): + return { + "format": 1, "prefix": str(root), "python": "python3", "server_port": 8080, + "ntfy_port": 8090, "opencode": "opencode", "opencode_env": {}, "path": "", + "topic": "pipelines", "public_url": "http://192.168.1.20:8090", "username": "phone-reader", + "workspace": str(root), "launcher": str(root / "bin/pipelines"), "registration": None, + "label": "net.a-wan.pipelines", "docker": "docker", "project": "pipelines", "timezone": "UTC", + "ntfy": {"mode": mode, "bin": str(root / "bin/ntfy"), + "config": str(root / "data/ntfy/server.yml"), "base_url": "http://192.168.1.20:8090"}, + } + + def test_mode_selection_defaults_to_docker(self): + self.assertEqual(manage.ntfy_mode({}), "docker") + self.assertEqual(manage.ntfy_mode({"ntfy": {}}), "docker") + self.assertEqual(manage.ntfy_mode({"ntfy": {"mode": "native"}}), "native") + self.assertEqual(manage.ntfy_health_url({"ntfy_port": 8090}), "http://127.0.0.1:8090/v1/health") + + def test_ntfy_start_launches_binary_with_generated_config(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "bin").mkdir() + (root / "logs").mkdir() + config = self.config(root) + with patch.object(manage.subprocess, "Popen", return_value=FakeProcess(pid=4321)) as popen: + child = manage.ntfy_start(config) + self.assertEqual(popen.call_args[0][0], [config["ntfy"]["bin"], "serve", "--config", config["ntfy"]["config"]]) + self.assertEqual(child.pid, 4321) + self.assertEqual((root / "ntfy.pid").read_text(), "4321") + + def test_ntfy_stop_terminates_managed_process_and_removes_pid(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "bin").mkdir() + (root / "bin/ntfy").write_text("") + (root / "ntfy.pid").write_text(str(os.getpid())) + config = self.config(root) + killed = [] + + def fake_kill(pid, sig): + if sig == 0: + raise ProcessLookupError + killed.append((pid, sig)) + + with patch.object(manage.subprocess, "check_output", + return_value=str(root / "bin/ntfy") + " serve --config x"), \ + patch.object(manage.os, "kill", side_effect=fake_kill), \ + patch.object(manage, "compose", side_effect=AssertionError("compose used for native ntfy")): + manage.ntfy_stop(config) + self.assertIn((os.getpid(), signal.SIGTERM), killed) + self.assertFalse((root / "ntfy.pid").exists()) + + def test_native_stop_never_calls_compose(self): + with tempfile.TemporaryDirectory() as directory: + config = self.config(Path(directory)) + with patch.object(manage, "registration", return_value=None), \ + patch.object(manage, "managed_pid", return_value=None), \ + patch.object(manage, "ntfy_stop") as stop, \ + patch.object(manage, "compose", side_effect=AssertionError("compose used for native ntfy")): + manage.stop(config) + stop.assert_called_once_with(config) + + def test_uninstall_native_skips_compose_down(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "bin").mkdir() + config = self.config(root) + with patch.object(manage, "stop"), \ + patch.object(manage, "registration", return_value=None), \ + patch.object(manage, "compose", side_effect=AssertionError("compose used for native ntfy")): + manage.uninstall(config, purge=False, yes=True) + self.assertTrue((root / manage.MARKER).is_file()) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_resolve_releases.py b/scripts/test_resolve_releases.py index cc8f224d..0670be5d 100644 --- a/scripts/test_resolve_releases.py +++ b/scripts/test_resolve_releases.py @@ -147,6 +147,14 @@ class ReleaseResolutionTests(unittest.TestCase): with self.assertRaisesRegex(resolve_releases.ReleaseError, "non-registry host"): resolve_releases._default_fetch("https://evil.example.com/tags/list") + def test_shared_allowlist_guard_requires_https_allowlisted_host(self): + self.assertTrue(resolve_releases.is_allowlisted("https://api.github.com/x")) + self.assertTrue(resolve_releases.is_allowlisted("https://ghcr.io/x")) + self.assertTrue(resolve_releases.is_allowlisted("https://registry.npmjs.org/x")) + self.assertFalse(resolve_releases.is_allowlisted("http://api.github.com/x")) + self.assertFalse(resolve_releases.is_allowlisted("https://evil.example.com/x")) + self.assertFalse(resolve_releases.is_allowlisted("https://api.github.com.evil.example/x")) + def test_malformed_pi_metadata_is_rejected(self): for pi in ({"version": "not-semver"}, {"version": "1.1.0", "engines": "node"}, 42): with self.assertRaises(resolve_releases.ReleaseError): @@ -178,5 +186,51 @@ class ReleaseResolutionTests(unittest.TestCase): resolve_releases.load_fixture(path) +class FakeNtfy: + """Deterministic ntfy tag fixture served through the same fetch seam.""" + + VERSION = "2.11.0" + + def __init__(self, tags=None, tags_response=None): + self.tags = tags if tags is not None else ["v" + self.VERSION] + self.tags_response = tags_response + self.calls = [] + + def __call__(self, url, headers=None): + self.calls.append(url) + if "/git/matching-refs/" in url: + if self.tags_response is not None: + return 200, {}, json.dumps(self.tags_response).encode() + return 200, {}, json.dumps([{"ref": "refs/tags/" + tag} for tag in self.tags]).encode() + return 404, {}, b"" + + +class NtfyResolutionTests(unittest.TestCase): + def test_resolves_exact_version_from_official_tags_only(self): + registry = FakeNtfy(tags=["v2.10.0", "v2.11.0"]) + result = resolve_releases.resolve_ntfy(registry) + self.assertEqual(result["version"], "2.11.0") + self.assertEqual(result["tag"], "v2.11.0") + self.assertEqual(result["module"], resolve_releases.NTFY_MODULE) + self.assertEqual(result["repository"], resolve_releases.NTFY_REPOSITORY) + # No release assets or checksums are fetched: the native installer builds + # the server from the pinned module instead. + self.assertNotIn("assets", result) + self.assertEqual(len(registry.calls), 1) + self.assertTrue(registry.calls[0].startswith( + "https://api.github.com/repos/binwiederhier/ntfy/git/matching-refs/")) + self.assertFalse(any("releases/latest" in url for url in registry.calls)) + + def test_malformed_tag_metadata_is_rejected(self): + for tags in ([42], [{"ref": "refs/heads/main"}], [{"ref": "refs/tags/not-a-version"}], + [{"noref": "refs/tags/v2.11.0"}]): + with self.assertRaises(resolve_releases.ReleaseError): + resolve_releases.resolve_ntfy(FakeNtfy(tags_response=tags)) + + def test_no_stable_release_fails_closed(self): + with self.assertRaisesRegex(resolve_releases.ReleaseError, "no stable ntfy release"): + resolve_releases.resolve_ntfy(FakeNtfy(tags=[])) + + if __name__ == "__main__": unittest.main() diff --git a/scripts/test_save_ntfy_token.py b/scripts/test_save_ntfy_token.py index 0bd3a69e..ad0d2aac 100644 --- a/scripts/test_save_ntfy_token.py +++ b/scripts/test_save_ntfy_token.py @@ -12,15 +12,22 @@ class TokenTests(unittest.TestCase): def test_extract_prose_and_protect_source_directory(self): token = "tk_" + "a" * 29 with tempfile.TemporaryDirectory() as directory: - save_token.save_token(f"token {token} created for user pipeline-publisher, never expires\n", directory) path = Path(directory, "secrets/ntfy-token.txt") + save_token.save_token(f"token {token} created for user pipeline-publisher, never expires\n", path, 0o444) self.assertEqual(path.read_text(), token + "\n") self.assertEqual(path.stat().st_mode & 0o777, 0o444) self.assertEqual(path.parent.stat().st_mode & 0o777, 0o700) with self.assertRaises(ValueError): - save_token.save_token(f"token {token} created", directory) + save_token.save_token(f"token {token} created", path, 0o444) self.assertEqual(path.read_text(), token + "\n") + def test_native_mode_is_private_to_the_installing_user(self): + token = "tk_" + "b" * 29 + with tempfile.TemporaryDirectory() as directory: + path = Path(directory, "secrets/publisher-token") + save_token.save_token(f"token {token} created", path, 0o600) + self.assertEqual(path.stat().st_mode & 0o777, 0o600) + def test_failures_do_not_replace_empty_placeholder(self): with tempfile.TemporaryDirectory() as directory: path = Path(directory, "secrets/ntfy-token.txt") @@ -28,9 +35,38 @@ class TokenTests(unittest.TestCase): path.write_text("") for output in ["", "connection failed", "tk_" + "a" * 29 + " tk_" + "b" * 29]: with self.assertRaises(ValueError): - save_token.save_token(output, directory) + save_token.save_token(output, path, 0o444) self.assertEqual(path.read_text(), "") + def test_accounts_are_bootstrapped_once_with_the_shared_sequence(self): + token = "tk_" + "c" * 29 + + class FakeCompleted: + returncode = 0 + stdout = f"token {token} created for pipeline-publisher\n" + stderr = "" + + with tempfile.TemporaryDirectory() as directory: + path = Path(directory, "secrets/publisher-token") + calls = [] + + def run(args, capture=False, interactive=False): + calls.append((tuple(args), capture, interactive)) + return FakeCompleted() + + save_token.bootstrap_accounts(run, path, 0o600, log=lambda _message: None) + self.assertEqual([args for args, _, _ in calls], [ + ("user", "add", "--ignore-exists", "pipeline-publisher"), + ("access", "pipeline-publisher", "pipelines", "write-only"), + ("token", "add", "pipeline-publisher"), + ("user", "add", "--ignore-exists", "phone-reader"), + ("access", "phone-reader", "pipelines", "read-only"), + ]) + self.assertEqual([capture for _, capture, _ in calls], [False, False, True, False, False]) + self.assertEqual([interactive for _, _, interactive in calls], [True, False, False, True, False]) + self.assertEqual(path.read_text(), token + "\n") + self.assertEqual(path.stat().st_mode & 0o777, 0o600) + if __name__ == "__main__": unittest.main() diff --git a/scripts/test_setup_native.py b/scripts/test_setup_native.py new file mode 100644 index 00000000..92cc13dc --- /dev/null +++ b/scripts/test_setup_native.py @@ -0,0 +1,362 @@ +import importlib.util +import os +from pathlib import Path +import subprocess +import tempfile +import unittest +from unittest.mock import patch + +spec = importlib.util.spec_from_file_location("setup_native", Path(__file__).with_name("setup-native.py")) +setup_native = importlib.util.module_from_spec(spec) +spec.loader.exec_module(setup_native) + +manage_spec = importlib.util.spec_from_file_location("pipelines_manage", Path(__file__).with_name("manage.py")) +manage = importlib.util.module_from_spec(manage_spec) +manage_spec.loader.exec_module(manage) + +TOKEN = "tk_" + "a" * 29 +NTFY_VERSION = "2.11.0" +NTFY_MODULE = "heckel.io/ntfy/v2" + + +def resolution(): + """The shape resolve_ntfy returns: an exact tag to build the server from.""" + return {"repository": "binwiederhier/ntfy", "module": NTFY_MODULE, + "version": NTFY_VERSION, "tag": "v" + NTFY_VERSION} + + +class Recorder: + def __init__(self, token=TOKEN, volumes=(), image="pipelines-pipelines:latest", + fail_create=False, fail_copy=False, fail_volume_ls=False, fail_build=False): + self.calls = [] + self.token = token + self.volumes = volumes + self.image = image + self.fail_create = fail_create + self.fail_copy = fail_copy + self.fail_volume_ls = fail_volume_ls + self.fail_build = fail_build + self._module_cache = None + self.kwargs = [] + + def module_cache(self): + """A stand-in Go module cache holding the pinned ntfy module source.""" + if self._module_cache is None: + self._module_cache = tempfile.mkdtemp(prefix="gomodcache-") + source = Path(self._module_cache) / f"{NTFY_MODULE}@{resolution()['tag']}" + source.mkdir(parents=True) + (source / "go.mod").write_text(f"module {NTFY_MODULE}\n") + return self._module_cache + + def __call__(self, command, **kwargs): + command = list(command) + self.calls.append(command) + self.kwargs.append(kwargs) + stdout, code = "", 0 + if command[0] == "docker": + if command[1:3] == ["volume", "ls"]: + if self.fail_volume_ls: + code = 1 + else: + stdout = "\n".join(self.volumes) + elif command[1:3] == ["image", "ls"]: + stdout = self.image + "\n" if self.image else "" + elif command[1] == "create" and self.fail_create: + code = 1 + elif command[1] == "cp": + if self.fail_copy: + code = 1 + else: + Path(command[3]).write_bytes(b"migrated") + elif command[0] == "go" and command[1:3] == ["env", "GOMODCACHE"]: + stdout = self.module_cache() + "\n" + elif command[0] == "go" and command[1:2] == ["build"]: + if self.fail_build: + code = 1 + elif "-o" in command: + # Only the ntfy build is inspected by the installer; its output + # is created so the atomic install has a real file to move. + output = Path(command[command.index("-o") + 1]) + if output.name == "ntfy": + output.write_bytes(b"#!/bin/sh\n") + output.chmod(0o755) + if command[0].endswith("ntfy") and "token" in command: + stdout = f"token {self.token} created for pipeline-publisher\n" + return subprocess.CompletedProcess(command, code, stdout=stdout, stderr="") + + def has(self, *parts): + return any(all(part in call for part in parts) for call in self.calls) + + +class SetupNativeTests(unittest.TestCase): + def installer(self, directory, recorder): + # A disposable XDG config home keeps the client default out of the real + # user configuration during tests. + return setup_native.Setup(directory, prefix=Path(directory) / "prefix", runner=recorder, + resolver=resolution, + environ={"XDG_CONFIG_HOME": str(Path(directory) / "config")}) + + @staticmethod + def which_all(opencode="/usr/bin/opencode", pi=None, cc="/usr/bin/cc"): + mapping = {"go": "/usr/bin/go", "python3": "/usr/bin/python3", + "git": "/usr/bin/git", "curl": "/usr/bin/curl"} + if opencode: + mapping["opencode"] = opencode + if pi: + mapping["pi"] = pi + if cc: + mapping["cc"] = cc + return lambda name: mapping.get(name) + + def test_require_tools_reports_each_missing_tool_without_installing(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder() + installer = self.installer(directory, recorder) + mapping = {"go": "/usr/bin/go", "python3": "/usr/bin/python3", "git": None, "curl": None} + with patch.object(setup_native.shutil, "which", side_effect=lambda name: mapping.get(name)): + with self.assertRaises(ValueError) as raised: + installer.require_tools() + message = str(raised.exception) + self.assertIn("git", message) + self.assertIn("curl", message) + self.assertIn("never installs tools", message) + self.assertEqual(recorder.calls, []) + + def test_build_installs_only_server_and_cli_with_trimpath(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder() + installer = self.installer(directory, recorder) + prefix = Path(directory) / "prefix" + installer.prepare_layout(prefix) + installer.build(prefix) + builds = [call for call in recorder.calls if call[:2] == ["go", "build"]] + self.assertEqual(len(builds), 2) + for call in builds: + self.assertIn("-trimpath", call) + self.assertTrue(any(call[-1] == "./cmd/pipelines-server" for call in builds)) + self.assertTrue(any(call[-1] == "./cmd/pipelines" for call in builds)) + self.assertFalse(any("pipelines-tui" in part for call in recorder.calls for part in call)) + + def test_layout_creates_private_prefix_and_copies_manage(self): + with tempfile.TemporaryDirectory() as directory: + installer = self.installer(directory, Recorder()) + prefix = Path(directory) / "prefix" + installer.prepare_layout(prefix) + for name in ("bin", "lib", "data", "logs", "secrets"): + self.assertTrue((prefix / name).is_dir(), name) + self.assertEqual((prefix / "secrets").stat().st_mode & 0o777, 0o700) + self.assertTrue((prefix / "lib/manage.py").is_file()) + + def test_prefix_honours_pipelines_prefix_environment(self): + with tempfile.TemporaryDirectory() as directory: + custom = Path(directory) / "custom" + installer = setup_native.Setup(directory, runner=Recorder(), environ={"PIPELINES_PREFIX": str(custom)}) + self.assertEqual(installer.resolve_prefix(), custom.resolve()) + + def test_missing_opencode_fails_actionably(self): + with tempfile.TemporaryDirectory() as directory: + installer = self.installer(directory, Recorder()) + with patch.object(setup_native.shutil, "which", side_effect=self.which_all(opencode=None)): + with self.assertRaisesRegex(ValueError, "OpenCode was not found"): + installer.harness() + + def test_full_run_writes_manifest_tokens_and_config_without_harness_install(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder() + installer = self.installer(directory, recorder) + prefix = Path(directory) / "prefix" + with patch.object(setup_native.shutil, "which", side_effect=self.which_all()), \ + patch.object(setup_native.lan, "detect_lan", return_value="192.168.1.20"), \ + patch.object(setup_native.configure, "host_timezone", return_value="Europe/London"): + installer.run(setup_native.arguments([])) + config = manage.load(prefix) + self.assertEqual(config["format"], 1) + self.assertEqual(config["prefix"], str(prefix.resolve())) + self.assertEqual(config["server_port"], 8080) + self.assertEqual(config["ntfy"]["mode"], "native") + self.assertEqual(config["topic"], "pipelines") + self.assertEqual(config["opencode"], "/usr/bin/opencode") + self.assertEqual(config["pi"], "") + self.assertEqual((prefix / setup_native.MARKER).stat().st_mode & 0o777, 0o600) + token_path = prefix / "secrets/publisher-token" + self.assertEqual(token_path.read_text(), TOKEN + "\n") + self.assertEqual(token_path.stat().st_mode & 0o777, 0o600) + text = (prefix / "data/ntfy/server.yml").read_text() + for needle in ('listen-http:', 'base-url: "http://192.168.1.20:8090"', 'cache-file:', 'auth-file:', + 'auth-default-access: "deny-all"', 'enable-login: true', 'attachment-cache-dir:', + 'attachment-expiry-duration:', 'attachment-total-size-limit:', + 'attachment-file-size-limit:'): + self.assertIn(needle, text) + self.assertNotIn("topic", text) + self.assertTrue(recorder.has(str(prefix.resolve() / "bin/pipelines"), "skill", "install")) + self.assertTrue(recorder.has("access", "pipeline-publisher", "pipelines", "write-only")) + self.assertTrue(recorder.has("access", "phone-reader", "pipelines", "read-only")) + self.assertFalse(any("opencode" in part for call in recorder.calls for part in call)) + self.assertFalse(any("pipelines-tui" in part for call in recorder.calls for part in call)) + self.assertFalse(any(TOKEN in part for call in recorder.calls for part in call)) + + def test_reinstall_keeps_existing_publisher_token(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder() + installer = self.installer(directory, recorder) + prefix = Path(directory) / "prefix" + with patch.object(setup_native.shutil, "which", side_effect=self.which_all()), \ + patch.object(setup_native.lan, "detect_lan", return_value="192.168.1.20"), \ + patch.object(setup_native.configure, "host_timezone", return_value="Europe/London"): + installer.run(setup_native.arguments([])) + before = (prefix / "secrets/publisher-token").read_text() + + def refuse_resolution(): + raise AssertionError("re-resolved ntfy despite a verified installed binary") + + installer.resolver = refuse_resolution + recorder.calls.clear() + installer.run(setup_native.arguments([])) + self.assertEqual((prefix / "secrets/publisher-token").read_text(), before) + self.assertFalse(any("token" in call for call in recorder.calls), "reinstall re-created the token") + + def test_ntfy_is_built_from_the_pinned_module(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder() + installer = self.installer(directory, recorder) + prefix = Path(directory) / "prefix" + installer.prepare_layout(prefix) + with patch.object(setup_native.shutil, "which", side_effect=self.which_all()): + path = installer.build_ntfy(prefix, resolution()) + self.assertTrue(path.is_file()) + self.assertTrue(path.stat().st_mode & 0o100) + self.assertTrue(recorder.has("go", "mod", "download", f"{NTFY_MODULE}@v{NTFY_VERSION}")) + build = next(call for call in recorder.calls if call[:2] == ["go", "build"] and call[-1] == ".") + self.assertIn("-trimpath", build) + self.assertIn(setup_native.NTFY_BUILD_TAGS, build) + self.assertTrue(any(f"-X main.version=v{NTFY_VERSION}" in part for part in build)) + self.assertTrue(build[build.index("-o") + 1].endswith("/ntfy")) + kwargs = next(kw for call, kw in zip(recorder.calls, recorder.kwargs) + if call[:2] == ["go", "build"] and call[-1] == ".") + self.assertEqual(kwargs.get("env", {}).get("CGO_ENABLED"), "1") + + def test_ntfy_build_is_reused_for_the_same_version(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder() + installer = self.installer(directory, recorder) + prefix = Path(directory) / "prefix" + installer.prepare_layout(prefix) + with patch.object(setup_native.shutil, "which", side_effect=self.which_all()): + installer.build_ntfy(prefix, resolution()) + installer.write_manifest(prefix, {"ntfy": {"version": "v" + NTFY_VERSION}}) + recorder.calls.clear() + installer.build_ntfy(prefix, resolution()) + self.assertFalse(any(call[:2] == ["go", "build"] for call in recorder.calls), + "rebuilt ntfy for an already installed version") + + def test_missing_c_compiler_fails_actionably(self): + installer = setup_native.Setup("/tmp", runner=Recorder(), resolver=resolution, environ={}) + with patch.object(setup_native.shutil, "which", side_effect=self.which_all(cc=None)): + with self.assertRaisesRegex(ValueError, "No C compiler found"): + installer.require_compiler() + + def test_ntfy_build_failure_is_reported(self): + with tempfile.TemporaryDirectory() as directory: + installer = self.installer(directory, Recorder(fail_build=True)) + prefix = Path(directory) / "prefix" + installer.prepare_layout(prefix) + with patch.object(setup_native.shutil, "which", side_effect=self.which_all()): + with self.assertRaisesRegex(ValueError, "build failed"): + installer.build_ntfy(prefix, resolution()) + + def test_publisher_token_is_never_overwritten(self): + with tempfile.TemporaryDirectory() as directory: + prefix = Path(directory) / "prefix" + (prefix / "secrets").mkdir(parents=True) + path = prefix / "secrets/publisher-token" + setup_native.tokens.save_token(f"token {TOKEN} created", path, 0o600) + self.assertEqual(path.stat().st_mode & 0o777, 0o600) + with self.assertRaises(ValueError): + setup_native.tokens.save_token(f"token {TOKEN} created", path, 0o600) + self.assertEqual(path.read_text(), TOKEN + "\n") + + def test_migration_copies_docker_data_only_when_absent(self): + with tempfile.TemporaryDirectory() as directory: + prefix = Path(directory) / "prefix" + (prefix / "data/ntfy").mkdir(parents=True) + recorder = Recorder(volumes=("pipelines_pipelines-data", "pipelines_ntfy-data")) + installer = self.installer(directory, recorder) + self.assertTrue(installer.migrate_docker(prefix, replace=False)) + self.assertEqual((prefix / "data/pipelines.db").read_bytes(), b"migrated") + self.assertTrue(any(call[:2] == ["docker", "cp"] and call[2].endswith("-wal") + for call in installer.runner.calls), + "the SQLite write-ahead log sidecar must be copied with the database") + self.assertEqual((prefix / "data/ntfy/auth.db").read_bytes(), b"migrated") + + def test_migration_keeps_existing_native_data_without_opt_in(self): + with tempfile.TemporaryDirectory() as directory: + prefix = Path(directory) / "prefix" + (prefix / "data").mkdir(parents=True) + (prefix / "data/pipelines.db").write_bytes(b"native") + recorder = Recorder(volumes=("pipelines_pipelines-data",)) + installer = self.installer(directory, recorder) + self.assertTrue(installer.migrate_docker(prefix, replace=False)) + self.assertEqual((prefix / "data/pipelines.db").read_bytes(), b"native") + self.assertFalse(any(call[1:2] == ["cp"] for call in recorder.calls)) + + def test_absent_docker_volumes_is_a_noop(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder(volumes=()) + installer = self.installer(directory, recorder) + self.assertFalse(installer.migrate_docker(Path(directory) / "prefix", replace=False)) + self.assertFalse(any(call[0] == "docker" and call[1] == "create" for call in recorder.calls)) + + def test_uninspectable_docker_volumes_abort_instead_of_skipping_migration(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder(fail_volume_ls=True) + installer = self.installer(directory, recorder) + with self.assertRaisesRegex(ValueError, "Could not inspect Docker volumes"): + installer.migrate_docker(Path(directory) / "prefix", replace=False) + + def test_missing_docker_cli_is_a_noop_not_a_failure(self): + def no_docker(command, **kwargs): + raise FileNotFoundError(command[0]) + + with tempfile.TemporaryDirectory() as directory: + installer = setup_native.Setup(directory, prefix=Path(directory) / "prefix", + runner=no_docker, resolver=resolution, environ={}) + self.assertEqual(installer.docker_volumes(), ()) + + def test_requested_migration_failure_aborts(self): + with tempfile.TemporaryDirectory() as directory: + prefix = Path(directory) / "prefix" + (prefix / "data").mkdir(parents=True) + recorder = Recorder(volumes=("pipelines_pipelines-data",), fail_copy=True) + installer = self.installer(directory, recorder) + with self.assertRaisesRegex(ValueError, "install aborted"): + installer.migrate_docker(prefix, replace=False) + + def test_full_run_migrates_docker_auth_store_before_native_accounts(self): + with tempfile.TemporaryDirectory() as directory: + recorder = Recorder(volumes=("pipelines_ntfy-data",)) + installer = self.installer(directory, recorder) + prefix = Path(directory) / "prefix" + with patch.object(setup_native.shutil, "which", side_effect=self.which_all()), \ + patch.object(setup_native.lan, "detect_lan", return_value="192.168.1.20"), \ + patch.object(setup_native.configure, "host_timezone", return_value="Europe/London"): + installer.run(setup_native.arguments([])) + # The migrated store must survive bootstrap instead of being shadowed. + self.assertEqual((prefix / "data/ntfy/auth.db").read_bytes(), b"migrated") + copy_index = next(i for i, call in enumerate(recorder.calls) + if call[0] == "docker" and call[1:2] == ["cp"]) + add_index = next(i for i, call in enumerate(recorder.calls) + if "pipeline-publisher" in call and "user" in call) + self.assertLess(copy_index, add_index, "auth store migrated after native accounts were created") + + def test_missing_local_image_aborts_migration(self): + with tempfile.TemporaryDirectory() as directory: + prefix = Path(directory) / "prefix" + (prefix / "data").mkdir(parents=True) + recorder = Recorder(volumes=("pipelines_pipelines-data",), image=None) + installer = self.installer(directory, recorder) + with self.assertRaisesRegex(ValueError, "no local Pipelines image"): + installer.migrate_docker(prefix, replace=False) + + +if __name__ == "__main__": + unittest.main()