diff --git a/README.md b/README.md index 5509f016..6f63f34d 100644 --- a/README.md +++ b/README.md @@ -1,120 +1,143 @@ # Pipelines -Small, reusable AI workflows for coding, research, and everyday project work. -Run them in your browser or terminal using your existing configured harnesses. -Inspect each stage's output, file changes, artifacts, and recorded session. +Small, reusable AI workflows for coding, research and everyday project work, +plus chats with the coding agents you already use. -**Alpha:** single-user, supervised work. Models can make mistakes and editing -workflows can change files and run commands. Start in a disposable project; -review results before trusting them. Provider charges may apply. +Pipelines is one per-user server with three clients: -## Install on macOS or Linux +- **Web app** (Flutter, served behind a sign-in) and the same app as an + **Android APK**. +- **CLI and TUI** (`pipelines`) for scripting, inspection and authoring. -Use a Git checkout and an already configured OpenCode installation. You also -need Go 1.27+, Python 3.9+, Git, curl, a C compiler, and Node.js 18+. -The installer checks prerequisites; it does not replace your harness or provider -credentials. Pi can also be used when configured. +A **pipeline** is a versioned Markdown definition of stages (model, human +decision, notification). A **run** executes it in a project directory on the +server machine. A **chat** is a conversation with an agent harness (OpenCode, +Claude Code or Codex) that the server keeps going while you are away. + +**Alpha:** single-user, supervised work. Models make mistakes and editing +stages change files and run commands. Start in a disposable project, review +results, and expect provider charges. + +![Workflow](docs/images/workflow.png) + +## Install ```sh git clone https://git.a-wan.net/alan/pipelines.git cd pipelines -./install +./install # server + CLI + web app, login autostart +./install --apk # optional: Android app, offered to phones for update ``` -Native installation is the default: Pipelines runs on your host and reuses your -existing harness setup. Follow the installer's connection details to open the -web app (default `http://localhost:8082/`). Setup creates or reuses its private -web sign-in credentials and enables startup at login. Run as your normal user -in a working macOS launchd or Linux systemd user session. -If no usable model default is configured, setup asks you to select one inside -`./install`; no separate configuration command or test generation is required. +Needs Go (version in `go.mod`), Node.js 18+, Python 3.9+, git, curl, a C +compiler, a **running Docker** (the web app is built in Docker; the install +refuses without it) and an existing OpenCode on `PATH`. The installer never +installs these for you. See [Installation](docs/install.md). -For optional terminal commands, select the installation and add its binaries -to PATH (use your custom prefix if you changed it): +Then open the URL the installer prints (default `http://localhost:8082/`) and +sign in with the password you chose. + +## What you can do + +**Runs** +- Runs list with filters, multi-select and delete, and sorting by newest, + oldest, **largest on disk**, duration, pipeline or status. +- Run detail tabs: Summary, Timeline, and Artifacts and Changes when the run + has any. +- Artifacts viewer for files a stage produced, including the recorded session + export (`session.html`). +- Changes: the file diffs each stage made in the workspace. +- Human decision stages: a run waits for you; decide in the app or with + `pipelines decide`. +- Retry from the failed stage, cancel, rename, per-run model overrides. + +**Pipelines** +- Starter templates, draft-first authoring with a harness, versioned + publishing, import/export as ZIP. +- [Schedules](docs/schedules.md): recurring runs of an existing pipeline. +- Search across pipelines, versions, runs, steps, fields and artifacts. + +**Chat** +- Chat with **OpenCode**, **Claude Code** or **Codex** in any folder on the + server, picked with the folder picker. +- Send messages while the agent is working: Claude Code and Codex receive them + mid-turn; otherwise the server queues them durably and delivers them in order + when the turn ends, even with no client attached. +- Approvals, questions, uploads, model choice and interrupt in the app. +- Adopt an existing harness session with `pipelines chat-adopt`. + +**Notifications** +- The Android app keeps its own connection and shows notifications in the + background (no ntfy app needed). +- Settings: any run ended, chat needs you, chat reply finished, and + **Also send to ntfy** for devices that still use ntfy. + +Pi runs pipeline stages (`harness=pi`) but is not offered as a chat harness. +See [Harnesses](docs/harnesses.md). + +## Architecture + +``` + browser / Android app ──HTTPS (tailscale serve) or HTTP──┐ + ▼ + web proxy (node, 0.0.0.0:8082) ── sign-in, web app, /updates/ + │ same-origin /connect, /artifacts/ + ▼ + pipelines CLI/TUI ──ws://127.0.0.1:8080/connect──► pipelines-server (SQLite) + │ + ┌────────────────────┬───────────────┼───────────────┐ + ▼ ▼ ▼ ▼ + opencode serve pi (RPC) claude (stdio) Codex App Server + stages + chat stages chat chat (socket) + │ + ▼ + ntfy (8090, optional fallback) +``` + +- **One server process** (`pipelines-server`) owns the database, runs, chats + and notifications. Its API has no authentication and binds to + `127.0.0.1:8080` by default. +- **One web app.** The authenticated Node proxy (`web/serve.mjs`) serves the + installed web build and proxies to the server. Every address (localhost, + LAN, tailnet) reaches this same proxy. +- **Tailnet HTTPS.** When `PIPELINES_WEB_PUBLIC_URL` is an `https://*.ts.net` + address, `./install` runs `tailscale serve` to point it at the proxy. +- **ntfy** is built from source and runs alongside, for optional fallback + delivery and stage notification hooks. +- A login service (launchd on macOS, `systemd --user` on Linux) supervises the + server, web proxy and ntfy together. + +## Terminal quick start ```sh export PATH="$HOME/.local/share/pipelines/bin:$PATH" -pipelines service enable -pipelines service status +cd /path/to/your/project +pipelines # TUI +pipelines run "pipeline name" "Describe a small task" +pipelines runs +pipelines inspect 1 pipelines doctor ``` -See [installation](docs/install.md) for autostart controls, custom ports, data -locations, and troubleshooting. No sudo is needed. Keep the raw -Pipelines and harness APIs private. Use the authenticated web proxy for browser -access; do not expose the control APIs to the Internet. +Runs use the directory you start them from, on the server machine. A browser +on another computer does not upload its files. -### Prefer Docker? +## Documentation -```sh -./install --docker -``` - -Docker remains optional, not the default. It uses its own persistent harness -credentials and needs Docker Compose and Python 3. Do not start a second native -server alongside it. [Docker setup and migration](docs/install.md). - -## Try a starter - -In the web app, open **Settings → Templates**, select a starter, and create a -pipeline. In the terminal, run `pipelines` and use `:templates`. - -Choose a small code change, a read-only review, research, or a document utility. -Starters use your task and selected project—not this repository's paths, stack, -or accounts. They inherit configured model defaults. Templates are available -immediately but are never published or run without your choice. - -1. Select the project directory **on the server machine**. -2. Give the pipeline a specific task and run it. -3. Inspect the report, stage visits, and any changed files. - -For terminal use, launch from your project directory: - -```sh -cd /path/to/your/project -pipelines -# After creating a pipeline: -pipelines run "your pipeline name" "Describe a small task" -pipelines runs -pipelines inspect 1 -``` - -A browser on another machine does not upload that machine's project files. -Recurring work uses separate [saved run schedules](docs/schedules.md), not workflow -definitions. Reuse one pipeline with different prompts and timers; filter both -Pipelines and Runs by scheduling state. -For editing work, use version control and keep a backup. Cancellation and retries -are **not rollback**; a new execution can repeat side effects. - -## Sessions and notifications - -Pipelines records and exports stage sessions itself. A model stage cannot hand -its result to the next stage until its session is settled and its export is -verified. If export fails, the computed output is retained while export-only -recovery waits; this is not a request to rerun the agent or make a human decision. -See [session capture](docs/notes/session-export-handoff-gate.md). - -Notifications have separate channels: - -- **Web alerts:** require the page to be open and browser permission where - supported. Closed-app browser push is not implemented. -- **ntfy:** optional phone notifications, independent of the Pipelines app. - Removing the app does not unsubscribe an ntfy client. - -See [notification setup](docs/ntfy.md). The installer checks publishing, not -actual delivery to your phone. - -## More - -- [Alpha walkthrough and known limits](docs/first-version-testing.md) -- [Installation and autostart](docs/install.md) -- [Configured harnesses and models](docs/harnesses-and-control.md) -- [CLI commands](docs/cli.md) +- [Installation and lifecycle](docs/install.md) +- [Harnesses: OpenCode, Claude Code, Codex, Pi](docs/harnesses.md) +- [Notifications](docs/notifications.md) · [ntfy reference](docs/ntfy.md) +- [CLI and authoring](docs/cli.md) +- [Schedules](docs/schedules.md) - [Pipeline format](internal/author/skill/references/format.md) -- [App (web and Android)](app/README.md) · [Web proxy](web/README.md) +- [Reasoning effort and stage limits](docs/harnesses-and-control.md) +- [Chat backend details](docs/chat-backends.md) - [Artifacts](docs/stage-artifacts.md) · [File diffs](docs/stage-file-diffs.md) - [Transfer pipeline definitions](docs/pipeline-transfer.md) +- [Docker installation](docs/docker.md) +- [Web proxy](web/README.md) +- [Alpha walkthrough and known limits](docs/first-version-testing.md) When reporting a problem, include your OS, install mode, reproduction steps, -`pipelines version --json`, and `pipelines ping --json`. Redact credentials and -private task/file content; do not post your database or native session storage. +`pipelines version --json` and `pipelines ping --json`. Redact credentials and +private content; do not post your database or harness session storage. diff --git a/docs/first-version-testing.md b/docs/first-version-testing.md index 8cd2b3a4..d8d9f31c 100644 --- a/docs/first-version-testing.md +++ b/docs/first-version-testing.md @@ -58,9 +58,10 @@ a new execution can repeat side effects. Never delete data/volumes to fix a run. ## Notifications Browser alerts require the page to be open; closed-app web push is not supported. -ntfy is a separate optional channel and can notify your phone without the +The Android app delivers notifications in the background over its own +connection. ntfy is an optional fallback channel that works without the Pipelines Android app. Installation verifies publishing, not reception. Check -your subscription and device reception before relying on it. See [ntfy](ntfy.md). +device reception before relying on it. See [Notifications](notifications.md). ## Known limits diff --git a/docs/harnesses.md b/docs/harnesses.md new file mode 100644 index 00000000..0f80abc2 --- /dev/null +++ b/docs/harnesses.md @@ -0,0 +1,220 @@ +# Harnesses + +A harness is the agent program Pipelines drives. Pipelines never installs, +upgrades or signs in a harness for you; it uses the copy and login you already +have on the server machine. + +| Harness | Pipeline stages | Chat | Mid-turn messages | Authoring/debug sessions | +| --- | --- | --- | --- | --- | +| OpenCode | yes (default) | yes (default) | queued until the turn ends | yes | +| Pi | yes (`harness=pi`) | no | n/a | yes | +| Claude Code | no | yes | yes, delivered into the running turn | no | +| Codex | no | yes | yes, via `turn/steer` | no | + +Stage configs accept only `harness=opencode` or `harness=pi`. + +## Choosing models + +**Pipeline stages.** Each model stage pins its own model in its `configs` +block: + +````markdown +```configs +provider=PROVIDER +model=MODEL +harness=opencode +reasoning_effort=high +``` +```` + +`harness` defaults to `opencode`. `reasoning_effort` is optional; see +[harnesses and control](harnesses-and-control.md). Use the exact IDs your +harness lists (`opencode models`, `pi --list-models`). + +**Per run**, without changing the definition: + +```sh +pipelines run "my pipeline" "task" --model PROVIDER/MODEL +pipelines run "my pipeline" "task" --stage-model review=PROVIDER/MODEL +``` + +Precedence: `--stage-model` > `--model` > the model pinned in the stage. +Unknown stage keys and malformed models are refused before the run starts. + +**Defaults for new pipelines and interactive sessions** come from harness +profiles in `~/.config/pipelines/harnesses.json` (override with +`PIPELINES_HARNESS_CONFIG`). Each profile has an `executable`, a `model` +(interactive authoring/debug sessions) and a `pipeline_model` (stages in new +drafts and starters). Published versions are never rewritten. + +```sh +pipelines harness list # profiles, default, launch mode +pipelines harness setup opencode --host # fuzzy-pick both models +pipelines harness setup pi --host --login # sign in to Pi, then pick models +pipelines harness setup opencode --host --model P/M --pipeline-model P/M +pipelines harness default pi # default harness for new/edit/debug +``` + +`--host` writes a native profile and clears any Docker link. `./install` +writes the OpenCode host profile itself (from `--model`, the saved profile, or +OpenCode's own default) and keeps an existing valid Pi profile; it never adds +a Pi profile on its own. `./install --opencode` and `./install --pi` (which run +`scripts/setup-harness.py`) apply only to a Docker installation. + +**Chat** models are chosen in the chat's model picker; each harness supplies +its own catalog. + +## Verifying + +```sh +pipelines doctor # health + OpenCode API discovery, no inference +pipelines provider-check --model P/M # one tiny tool-free generation (may cost) +pipelines provider-check --harness pi --model P/M +python3 scripts/check-pi-rpc.py # Pi RPC and policy, no inference +``` + +`pipelines restart` and `pipelines doctor` print the server's startup report. +Its **Chat** row lists the chat harnesses the server found (for example +`claude, codex, opencode`), and the **Harnesses** rows list the profiles. Then +start a chat with each harness in the app. + +## OpenCode + +**Used for:** default stage harness, default chat harness, and `pipelines +new`, `edit` and `debug` sessions. + +**Found by:** `PIPELINES_OPENCODE`, then `PATH`, then `/opt/homebrew/bin`, +`/usr/local/bin`, `/usr/bin`, `~/.local/bin`. `./install` requires `opencode` +on `PATH` and records its absolute path. + +**Launched as:** the server starts its own loopback `opencode serve` for chat +and for stage sessions, using your normal OpenCode configuration and login. +A server started with `-opencode-endpoint URL` (or +`PIPELINES_OPENCODE_ENDPOINT`) attaches chat to that service instead; the +native installer does not set this. + +**Install and sign in:** install OpenCode yourself, then sign in to your +providers with OpenCode (`opencode auth login`). Pipelines reads the default +model with `opencode api get /api/model/default`. + +**Mid-turn messages:** not supported by the harness. A message sent while a +turn runs is stored in the server's durable queue (up to 8 per chat) and sent +in order when the turn ends, also after a server restart. + +## Pi + +**Used for:** pipeline stages with `harness=pi`, and `new`/`edit`/`debug` +with `--harness pi`. Not a chat harness. + +**Found by:** the `executable` in the `pi` profile. A run with a Pi stage is +refused unless that profile exists on the server. + +**Launched as:** `pi --mode rpc --provider P --model M --session-dir +--no-approve --no-extensions --no-skills --no-context-files +--no-prompt-templates --no-themes --extension policy.js`. The policy extension +enforces the stage's tool and edit permissions. `reasoning_effort` maps to Pi's +thinking level and is verified before the task is sent. + +**Install and sign in:** + +```sh +# install Pi yourself so `pi` is on PATH, then: +pipelines harness setup pi --host --login +``` + +`--login` opens a bare Pi session; run `/login` there, finish the provider +sign-in, and exit with Ctrl+C or Ctrl+D to return to model selection. + +## Claude Code + +**Used for:** chat only. + +**Found by:** `PIPELINES_CLAUDE`, then `PATH`, then `~/.local/bin/claude`, +`/opt/homebrew/bin/claude`, `/usr/local/bin/claude`. The server checks again +every 10 seconds, so installing it later needs no restart. + +**Launched as:** one `claude -p` process per active chat with stream-json input +and output and `--permission-prompt-tool=stdio` (at most 8 live at once). Idle +chats are resumed with `--resume `. Approvals and `AskUserQuestion` appear +as cards in the app. + +**Sign in:** use your existing Claude Code login. A background service can see +a stale keychain entry; when the default login check fails, the server retries +with the login file in `~/.claude`. For a login that does not depend on either, +create a long-lived token and store it privately: + +```sh +claude setup-token +# save the printed token: +install -m 600 /dev/null ~/.config/pipelines/claude-token +$EDITOR ~/.config/pipelines/claude-token +``` + +`PIPELINES_CLAUDE_TOKEN_FILE` names another file. The file must be a regular, +owner-only file. The startup report shows `Claude service token present` when +it is read. An `ANTHROPIC_API_KEY`, `CLAUDE_CODE_OAUTH_TOKEN` or +`CLAUDE_CONFIG_DIR` in the server's environment takes precedence. + +**Models:** the catalog comes from the CLI; switching applies live and is kept +for the next turn. + +**Mid-turn messages:** yes. The message is written to the running CLI's stdin +and read at its next step. Model changes apply from the next turn. + +## Codex + +**Used for:** chat only. + +**Found by:** the Codex App Server control socket at +`$CODEX_HOME/app-server-control/app-server-control.sock` (default +`~/.codex/...`). The Codex app provides it while it runs. The server checks for +it every 10 seconds and attaches when it appears. Pipelines never starts a +Codex daemon or changes Codex settings. + +**Sign in:** your existing Codex login. Threads created from Pipelines are +ordinary Codex threads and also show in the Codex app. + +**Behavior:** new threads use the `workspace-write` sandbox and Codex's own +approval policy; command, file-change and permission requests appear as +approval cards. + +**Mid-turn messages:** yes, with `turn/steer` pinned to the running turn. If +the turn already ended, the message is queued for the next turn instead. + +## Adopting existing sessions + +```sh +pipelines chat-adopt [--harness opencode|claude|codex] [--title TEXT] +``` + +Without `--harness`, each available harness is asked. Stop using the session +elsewhere once you continue it in Pipelines. + +## Limits + +- Stage session exports: the native session read is capped at 8 MiB and the + rendered `session.html` at 8 MiB. A session over the read cap cannot be + exported by retrying; the stage reports it. +- Claude Code chat history: a transcript over 64 MiB is reported as exceeding + the history limit instead of being truncated. +- Codex tool detail in history is truncated at 64 KiB and marked as such. +- Losing the Codex App Server connection drops pending approvals; they are + reported as an error, not shown as still actionable. +- Queued chat messages: at most 8 per chat while a turn runs. + +## Troubleshooting + +- **Harness works in a terminal but not from the service.** The login service + does not read your shell profile. It uses the `PATH` recorded when you ran + `./install` (prefixed with `/bin`). Rerun `./install` from a shell + where the harness is on `PATH`, or set an absolute `executable` with + `pipelines harness setup --host --executable /path/to/bin`. +- **Claude Code not listed under Chat.** Check `claude` resolves in one of the + locations above and `claude auth status` reports logged in. +- **Codex not listed under Chat.** Open the Codex app and confirm the socket + exists under `~/.codex/app-server-control/`. The installer does not record + a custom `CODEX_HOME` for the service, so the default `~/.codex` is used. +- **"Pi is not configured on this execution server".** Run + `pipelines harness setup pi --host` on the server machine. +- **"invalid provider/model identifier".** Use the exact IDs from the harness's + model list; placeholders such as `default/default` are refused. diff --git a/docs/install.md b/docs/install.md index 4d423bc8..7a850db6 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,564 +1,255 @@ # Installation -`./install` installs **natively on the host by default** (macOS and Linux); -`./install --docker` runs the existing Docker server + ntfy installation. -To reconfigure login and separate interactive/new-pipeline models inside an -existing Docker installation without a service reinstall: `./install --opencode` -or `./install --pi`. -See [configured harnesses](harnesses-and-control.md) for scope and verification limits. +`./install` installs Pipelines natively for your user on macOS or Linux. It +builds the server, the CLI and the web app together, installs them under +`~/.local/share/pipelines`, and starts them under a login service. No sudo. ```sh git clone https://git.a-wan.net/alan/pipelines.git cd pipelines -./install # native: server, authenticated web, CLI, ntfy + login autostart -./install --docker # explicit Docker server + ntfy -./install --web # rebuild the web app (Docker) after app changes -./install --apk # optional: build the Android app (Docker, slower) -pipelines update # later: pull, reinstall the same way, restart +./install ``` -## Updating +## Prerequisites -`pipelines update` brings an installation to the newest source. It waits while -runs are executing, fast-forwards the checkout it was installed from -(`git pull --ff-only`), re-runs `./install` with the options recorded at install -time, restarts the server, web proxy and ntfy, and prints the startup report. -Data, the web password, ntfy accounts and the Android signing key are kept; the -web app and APK are rebuilt only when their sources changed. Phones pick up a -new APK (after `./install --apk`) from Settings → App updates. +The installer only detects tools. It never installs, upgrades or downloads +them, and it stops naming whatever is missing. -`./install --fresh` starts with empty data even when volumes from an earlier -Docker installation exist (by default they are migrated). +| Tool | Why | +| --- | --- | +| Go (version in `go.mod`, currently 1.27) | builds `pipelines-server`, `pipelines` and ntfy | +| Node.js 18+ | runs the authenticated web proxy (`web/serve.mjs`) | +| Python 3.9+ | installer and the lifecycle manager (`manage.py`) | +| git, curl | source checkout, updates, release lookup | +| C compiler (`cc`, `clang` or `gcc`) | ntfy's SQLite driver is cgo (Xcode Command Line Tools on macOS) | +| **Docker, running** | builds the Flutter web app and the APK | +| OpenCode on `PATH` | required; default stage and chat harness | +| Pi, Claude Code, Codex | optional; see [Harnesses](harnesses.md) | +| tailscale | optional; tailnet HTTPS access | -## Native installation (default) +The install is **all or nothing**: the web app and the server are built before +either is installed. If Docker is not running, `./install` stops with +"Docker is not running, so the web app cannot be built" and leaves any existing +installation running unchanged. -`./install`, `./install --native` and `./install native` run -`scripts/setup-native.py`. It only detects the tools it needs and never installs -or upgrades them: **Go (version required by `go.mod`), Python 3.9+, Node.js 18+, -git, curl and a C compiler (`cc`/`clang`/`gcc`)** -must already be on PATH, and **Docker** must be running to build the app. The C compiler is required because ntfy stores its cache -and auth database in SQLite through the cgo driver `mattn/go-sqlite3`. It also -detects an existing `opencode` (required) and `pi` (optional) on PATH and records -their absolute paths; it never downloads, installs or upgrades OpenCode or Pi -(harness installation stays Docker-only). +Run as your normal logged-in user. macOS needs `launchctl` in a GUI login; +Linux needs `systemctl --user`. Other platforms are refused. -Run as your normal logged-in user, not root. macOS requires `launchctl` and an -active GUI login; Linux requires `systemctl` and a working `systemd --user` -session. Unsupported platforms and missing tools fail instead of silently -installing an unsupervised server. Go downloads dependencies and the pinned ntfy -source during a fresh build; the installer does not install system prerequisites. +## What `./install` does -**`./install` is the only setup command.** It builds the Flutter web app (`app/`) -in a Docker container pinned to `app/.flutter-version`, installs it at -`/lib/app-web`, configures the authenticated Node proxy (`web/serve.mjs`, -copied to `/lib/web`) and starts all three processes under one login -service. Builds are cached in `~/.cache/pipelines/app-builds` by a digest of the -app sources: an unchanged app is reused, a changed one rebuilt. Without Docker -the install still completes and the web page says to run `./install --web`. +1. Checks tools, port settings, the login service manager and the OpenCode + executable. +2. Builds the Flutter web app in Docker (pinned to `app/.flutter-version`). + Builds are cached in `~/.cache/pipelines/app-builds` by a digest of the app + sources, so an unchanged app is reused. +3. Builds `pipelines-server` and `pipelines` with `go build -trimpath`. +4. Installs into the prefix: binaries in `bin/`, web app in `lib/app-web`, web + proxy in `lib/web`, `lib/manage.py`, and an empty `lib/app-updates` for APKs. +5. Runs `pipelines skill install` (the authoring skill for OpenCode and Claude + Code skill roots). +6. Writes host harness profiles to `~/.config/pipelines/harnesses.json`. If no + usable OpenCode default model is found, it opens the model picker in the + terminal (or fails without a terminal; pass `--model`). +7. Builds ntfy from its pinned Go module, writes its config and creates the + `pipeline-publisher` and `phone-reader` accounts. +8. Migrates data from an earlier Docker installation, if its volumes exist. +9. Registers the login service, writes the manifest + `/pipelines-install.json`, saves `ws://127.0.0.1:/connect` as + the CLI default in `~/.config/pipelines/client.json`, and starts everything. +10. Retires the legacy standalone web agent `net.awan.pipelines.web` on macOS + (stopped, plist renamed to `.disabled`), so only one web app is served. +11. If `PIPELINES_WEB_PUBLIC_URL` is an `https://…ts.net` address, runs + `tailscale serve --bg --https= http://127.0.0.1:`. +12. Prints the browser URLs. -The Android app is optional because its build needs the Android SDK and takes -longer (on Apple Silicon the x86-64 build image runs under emulation). -`./install --apk` builds it in Docker for this installation's web address and -offers it at `/updates/` on that address, behind the sign-in, so phones install -it from Settings → App updates → Check for updates. Every APK is signed with one -key kept in `~/.config/pipelines/android/` (seeded from `~/.android/debug.keystore` -when present) so each new build installs over the last. +It also warns when a different `pipelines` earlier on `PATH` shadows the new +one; it does not touch that binary. -`pipelines start`, `restart` and `status` print the server's own startup report -(version, API, database, OpenCode, chat backends and harnesses) and the web, -web-app and phone-app status. The default browser URL is -`http://localhost:8082/` on this computer, or `http://HOST:8082/` on a trusted -LAN/VPN (`--web-port` / `--web-host` override it). Setup prints the actual URL. -Re-running `./install` preserves the saved API, web and ntfy ports and bind -addresses. Explicit flags override those settings, for example -`./install --web-port 8096`. An already-running supervisor retains its current -settings until restarted; restart only when no work is active. -Native installation and web/APK builds finish by printing the browser URLs, -including the saved web port, LAN address and configured public origin. -The unauthenticated Go API defaults to **127.0.0.1:8080**; expose only the web -proxy. Use TLS termination or an encrypted VPN for remote access; plain HTTP -does not encrypt the sign-in password. +### Addresses -Existing web credentials in `PIPELINES_WEB_AUTH_FILE`, or -`~/.config/pipelines/web-auth.json`, are reused unchanged. On a new computer -setup asks for the sign-in password (8+ characters, typed twice; set -`PIPELINES_WEB_PASSWORD` to answer without a terminal) and writes a private 0600 -file there with it and a generated cookie-signing `secret`. An invalid or -non-private existing file stops setup instead of resetting credentials. The -service never inherits the test-only `PIPELINES_WEB_AUTH=disabled` bypass. +| Service | Default | Flag | +| --- | --- | --- | +| Web proxy (sign-in, web app, `/updates/`) | `0.0.0.0:8082` | `--web-port`, `--web-host` | +| Server API (no auth) | `127.0.0.1:8080` | `--port`, `--bind-address` | +| ntfy | `:8090` | `--ntfy-port`, `--ntfy-url` | -A **fresh** native installation records `allow_authoring: true` in its manifest -and explicitly starts Go with `-allow-authoring`, so the single-user web workflow -can create and publish pipelines. `./install --no-authoring` instead selects an -execution-only role. Reinstalls preserve the recorded role; an older manifest -without this field stays execution-only. To deliberately enable authoring for -such an installation, use `./install --allow-authoring`. The installer prints the -selected role. This is a server permission, not a browser authentication bypass; -the authenticated proxy and loopback API defaults still apply. Docker's role and -startup flags are unchanged. -Role changes apply on the supervisor's next start. A reinstall does not forcibly -restart an already-running supervisor; if reconfiguring an active installation, -use its `pipelines restart` only when no work is active. +Expose only the web proxy. A reinstall keeps the saved ports and bind +addresses unless you pass the flag again, so an installation set up with +`--web-port 8083` stays on 8083. -Setup resolves `PIPELINES_PREFIX` or defaults to `~/.local/share/pipelines`, -creates `bin/`, `lib/`, `data/`, `logs/` and `secrets/` (`secrets` and `data` at -0700), copies `scripts/manage.py` to `/lib/manage.py`, builds exactly -`./cmd/pipelines-server` and `./cmd/pipelines` with `go build -trimpath` into -`/bin`, and runs `/bin/pipelines skill install` so the packaged -authoring skill lands in your OpenCode skills directory. No `pipelines-tui` is -built. It writes `/pipelines-install.json` atomically at 0600; that -manifest is what `scripts/manage.py` and the Go CLI load. Provider/model defaults -are never invented: valid saved Pipelines defaults are preserved, and an explicit -`--model PROVIDER/MODEL` fills missing OpenCode profile fields and is recorded as -the server's default for new starters. Otherwise setup reads OpenCode's resolved -host default using `opencode api get /api/model/default` from your home directory, -not the installer checkout. OpenCode handles its own JSONC and credentials; the -installer does not parse configuration files or generate a model response. +`PIPELINES_WEB_PUBLIC_URL` is the address phones and notification links use. +Set it once; reinstalls keep it: -After the build, setup checks whether a **different** `pipelines` resolves earlier -on PATH than `/bin/pipelines` (the common cause of a stale CLI talking to a -newer server). When it does, setup prints a `WARNING` naming that absolute path and -both versions (`version --json` on each binary, read-only) and tells you to put -`/bin` first on PATH. It never modifies, moves, deletes or upgrades that -other binary, and it installs no unrelated tool; the manifest keeps -`/bin` first on the recorded PATH. +```sh +PIPELINES_WEB_PUBLIC_URL=https://mymac.tailnet-name.ts.net ./install +``` -The native install also writes **host** harness profiles for the harnesses it -detects into XDG config home's `pipelines/harnesses.json` (no -`docker_directory`), preserving a valid host default (or choosing OpenCode instead -of a stale Docker default) and taking models from the saved profile, `--model`, -or OpenCode's actual configured default. The selected stage default is also -recorded as `PIPELINES_DEFAULT_MODEL` in the private manifest's server environment, -so web starters receive a real provider/model rather than `default/default`. -When migrating a Docker install it clears the recorded Docker link instead of -leaving the config Docker-linked. It never installs, upgrades or downloads a -harness. If no usable default exists, **`./install` itself opens the existing model -picker in a terminal**, choosing separate interactive and new-pipeline defaults -without logging in or performing inference. No separate configuration command is -needed. Noninteractive setup without a usable model fails before claiming ready -or starting services; pass `./install --model PROVIDER/MODEL` with an actual model -you have configured, or rerun `./install` in a terminal. Existing valid defaults, -credentials and unrelated profile fields are retained. -Discovery and selection do not verify an actual model generation or incur an -inference charge; provider execution remains a runtime check. +Without it the installer records `http://:`. Use HTTPS or a +VPN for remote access; plain HTTP does not protect the password. -ntfy is built from source with no Homebrew or package manager. Setup resolves the -exact stable `binwiederhier/ntfy` tag (never the moving `latest` tag and never an -older fallback), then compiles the server from the pinned `heckel.io/ntfy/v2` Go -module at that tag into `/bin/ntfy`. A source build is required because -upstream publishes **no macOS server binary** (its `darwin_all` asset is built -with `-tags noserver` and only provides the client commands), so both macOS and -Linux take the same path; the Go toolchain verifies the module against its public -checksum database. The build uses -`-tags sqlite_omit_load_extension,osusergo,netgo` and creates the same empty web -placeholders the upstream Makefile uses for local builds, so the bundled ntfy web -UI is not populated (notifications and the API are unaffected). A binary for the -same resolved version is reused instead of rebuilt. Setup generates -`/data/ntfy/server.yml` mirroring `compose.yaml` (`listen-http`, -`base-url`, `cache-file`, `auth-file`, `auth-default-access: deny-all`, -`enable-login`, attachment cache directory, expiry and size limits), creates the -`pipeline-publisher` (write-only on `pipelines`) and `phone-reader` (read-only) -accounts with ntfy's own CLI (`--ignore-exists`, so existing passwords are kept; -the account subcommands read the generated config through `NTFY_CONFIG_FILE`), -and writes the publisher token only to `/secrets/publisher-token` at 0600. +## Flags -When Docker volumes `pipelines_pipelines-data` or `pipelines_ntfy-data` exist, -setup migrates the pipeline database to `/data/pipelines.db` and the ntfy -auth store to `/data/ntfy/auth.db` using the local Pipelines image -(never an implicit pull). Existing native data is never overwritten unless you -pass `--replace-data`; a requested migration that fails aborts the install -instead of starting empty. Absent volumes are a no-op. If the Docker CLI is -present but its volumes cannot be inspected (for example a stopped daemon), setup -fails clearly instead of silently skipping migration, so an existing Docker -database is never shadowed by an empty native start. +From `arguments()` in `scripts/setup-native.py`: -Setup already starts it and enables login autostart, and prints a copyable -`/bin/pipelines service enable` command. That installed executable safely -selects its own native manifest; an unrelated CLI never guesses a default prefix. -Optional terminal use needs only the installed binary on PATH: +| Flag | Effect | +| --- | --- | +| `--prefix PATH` | install location (default `~/.local/share/pipelines`, or `PIPELINES_PREFIX`) | +| `--port N` | server API port (default 8080) | +| `--bind-address IP` | server API bind (default `127.0.0.1`) | +| `--web-port N` | web proxy port (default 8082) | +| `--web-host IP` | web proxy bind (default `0.0.0.0`) | +| `--ntfy-port N` | ntfy port (default 8090) | +| `--ntfy-url URL` | ntfy address given to phones (default `http://:`) | +| `--timezone TZ` | override the detected timezone (fallback UTC) | +| `--model PROVIDER/MODEL` | default model for new starters and missing OpenCode profile fields | +| `--allow-authoring` | allow creating and publishing pipelines (fresh installs default to this) | +| `--no-authoring` | execution-only server | +| `--fresh` | start empty even if earlier Docker volumes exist | +| `--replace-data` | let Docker migration overwrite existing native data | + +Reinstalls keep the recorded authoring role unless you pass one of the two +role flags. + +## Other install modes + +```sh +./install --web # rebuild and install only the web app (Docker), then reload the page +./install --apk # build the Android APK (Docker, slower) and offer it to phones +./install --docker # server and ntfy in Docker Compose instead of native +``` + +`./install --apk` builds a release APK for this installation's web address, +copies it to `/lib/app-updates/pipelines.apk` with a `version.json`, +and serves it at `/updates/` behind the sign-in. On the phone: +**Settings → App updates → Check for updates**, then **Download and install**. +The first install can use `/updates/pipelines.apk` directly. Every APK is +signed with the key in `~/.config/pipelines/android/` (seeded from +`~/.android/debug.keystore` if present), so each build installs over the last. +Options: `--server-url ws(s)://host:port/connect`, `--notes TEXT`, `--prefix`. + +`./install --docker` is the optional Compose installation with its own +container harness credentials; see [Docker](docker.md). Do not run it +alongside a native install on the same ports. `./install --opencode` and +`./install --pi` reconfigure a **Docker** installation's harness and refuse on +a native install; native installs use `pipelines harness setup` (see +[Harnesses](harnesses.md)). + +## Lifecycle commands + +Put the installed CLI on `PATH`: ```sh export PATH="$HOME/.local/share/pipelines/bin:$PATH" -pipelines service enable # idempotently enable/start this native install -pipelines service enable --print # read-only existing definition preview -pipelines service status # login registration state; no credentials -pipelines service disable # disable login autostart AND stop its supervisor -pipelines doctor -# pipelines stop / start / restart / logs ``` -`start`, `stop`, `restart`, `doctor`, `logs` and `uninstall` reach -`/lib/manage.py`. Ports default to 8080 (server) and 8090 (ntfy) and can -be changed with `--port`/`--ntfy-port`; setup starts both services itself, so -those ports and the web port (8082 by default) must be free (a running Docker installation holds them — stop it -first; the migration still needs its volumes and image). ntfy runs from -`/bin/ntfy` with the generated config; per-user launchd (macOS) or -`systemd --user` (Linux) starts ` /lib/manage.py supervise --prefix -`, which owns server, ntfy and web lifecycles. macOS writes -`~/Library/LaunchAgents/net.a-wan.pipelines.plist` and uses `launchctl enable`, -`bootstrap`, and `kickstart` in `gui/UID`. Linux writes -`$XDG_CONFIG_HOME/systemd/user/net.a-wan.pipelines.service` (default `~/.config`) -and runs `systemctl --user daemon-reload` then `enable --now -net.a-wan.pipelines.service`. This is login autostart, not boot-without-login; -Linux lingering is an explicit administrator decision. `pipelines service` uses -this existing supervisor and database, never a second developer server. An -explicit `PIPELINES_PREFIX` can also select it from another CLI. Native service -commands reject standalone `--db`/`--addr` options; use installer flags to change -the manifest. Definitions belonging to a developer service or another native -prefix are never overwritten by these commands. Logs are in -`/logs/{server,ntfy,web,supervisor}.log` (Linux supervisor output also goes -to `journalctl --user -u net.a-wan.pipelines.service`). +The installed `pipelines` finds its own manifest. `PIPELINES_PREFIX` selects +another installation. -Dated evidence (2026-10-13, macOS arm64): a scratch-prefix run of -`scripts/setup-native.py` resolved ntfy v2.30.0, compiled it from source (`ntfy ---version` = v2.30.0 with the `Server commands` present), built the server and -CLI, installed the skill, migrated 8 pipeline definitions and 28 runs plus both -ntfy accounts from the live Docker volumes, and wrote the manifest. -`pipelines start` then brought up ntfy and the server through the supervisor -(`{"healthy":true}` from both `/v1/health` and `/healthz` on spare ports) and -`pipelines stop` released both ports. launchd/systemd registration and real phone -delivery are **not tested**. +| Command | Effect | +| --- | --- | +| `pipelines start` | start server, web proxy and ntfy; print the startup report | +| `pipelines stop` | stop all three (active runs are interrupted; edits are not rolled back) | +| `pipelines restart` | stop then start, keeping data | +| `pipelines doctor` | startup report, health of server and ntfy, OpenCode API check (no inference) | +| `pipelines status` | server, host and ntfy publishing health, from the running server | +| `pipelines logs` | follow `logs/server.log` | +| `pipelines update` | pull and reinstall the same way, then restart (see below) | +| `pipelines uninstall [--purge] [--yes]` | remove the service and binaries; `--purge` also deletes data | +| `pipelines service enable [--print]` | enable and start login autostart (`--print` previews the definition) | +| `pipelines service disable` | disable autostart and stop the supervisor | +| `pipelines service status` | autostart and supervisor state | -The newer integrated web/autostart path is covered by scratch-prefix installer -mocks, rendered launchd/systemd definitions and mocked lifecycle/auth-preservation -checks. It has **not** been live-installed or tested through a real login on -either OS; Linux coverage here is mocked, not native Linux verification. +The login service is `~/Library/LaunchAgents/net.a-wan.pipelines.plist` on +macOS and `~/.config/systemd/user/net.a-wan.pipelines.service` on Linux. It +runs `manage.py supervise`, which owns all three processes. This is login +autostart, not boot-without-login. -## Docker installation (explicit) +## Where things live -`./install --docker` (or `./install docker`) runs the unchanged -`scripts/setup-docker.py`. It requires Docker Compose and Python 3 and includes -OpenCode/Pi in the persistent server container. Host Go builds the CLI/TUI -clients, not a native server. +| Path | Contents | +| --- | --- | +| `/pipelines-install.json` | manifest (ports, paths, install args, environment) | +| `/bin/` | `pipelines`, `pipelines-server`, `ntfy` | +| `/data/pipelines.db` | SQLite database (definitions, runs, chats, settings) | +| `/data/artifacts/` | run artifacts and session exports | +| `/data/ntfy/` | ntfy config, cache and accounts | +| `/secrets/publisher-token` | ntfy publishing token | +| `/logs/` | `server.log`, `web.log`, `ntfy.log`, `supervisor.log` | +| `~/.config/pipelines/web-auth.json` | web password and cookie secret (`PIPELINES_WEB_AUTH_FILE`) | +| `~/.config/pipelines/harnesses.json` | harness profiles (`PIPELINES_HARNESS_CONFIG`) | +| `~/.config/pipelines/client.json` | default server URL for the CLI/TUI | +| `~/.config/pipelines/claude-token` | optional Claude Code service token | +| `~/.config/pipelines/android/` | APK signing key | +| `~/.cache/pipelines/app-builds/` | cached web and APK builds | -The installer builds host `pipelines` and `pipelines-tui` clients when Go is -available, using `GOBIN` or `GOPATH/bin`; put that directory on PATH. It does not -install a host server. Reconfiguration uses `./install --opencode` or `./install --pi` -inside the existing Docker installation and never falls back to host execution. It -resolves and verifies the same current releases and refuses to configure against an -image that does not match them (rebuild with `./install --docker` first); it does not restart -or redeploy services. +`` is `~/.local/share/pipelines` unless changed. On Linux the +supervisor's output also reaches `journalctl --user -u net.a-wan.pipelines.service`. -Native Go builds embed VCS identity; Docker setup passes the checkout revision and -dirty state as compile-time build arguments (without copying `.git` into the image). -For a build-only refresh use `python3 scripts/build_metadata.py`; it resolves the -current OpenCode V2/Pi releases and runs `docker compose build` with the identity -arguments plus the resolved version/digest arguments, without restarting services. -A plain `docker compose build` cannot resolve the OpenCode image digest, so it must -be given `OPENCODE_VERSION`, `OPENCODE_DIGEST` and `PI_VERSION` through Docker -`--build-arg` flags (and `PIPELINES_VERSION`, `PIPELINES_COMMIT` and -`PIPELINES_DIRTY` outside a Git checkout); `./install --docker` supplies all of them. After deployment, -`pipelines --url ws://HOST:PORT/connect ping --json` identifies the running server. -Docker setup checks that the started server's pong matches the source identity -passed to the build and fails setup if metadata is missing or mismatched. +## Web sign-in password -Each setup resolves the current releases before touching settings or services: the -newest supported **OpenCode V2** release from the single documented complete -`GET .../git/matching-refs/tags/v2` response (a bounded complete array with no -pagination parameters; never the moving `latest` tag or the registry's insertion -order) and the current Pi release from npm. The image is built `FROM` the OpenCode -image's immutable `sha256` digest, resolved from GHCR, and labeled with the -resolved OpenCode/Pi versions, digest and source identity; the moving `latest` tag -and version-only references are never used. Setup verifies the built image reports -exactly those executable versions and a Node-compatible Pi engine, and stops with -a clear error (keeping the last verified Dockerfile fallback pins) when discovery, -resolution, build or compatibility fails rather than guessing or silently -installing stale or unchecked copies. A full install skips a rebuild only when the -release receipt **and** the actual built image's release and source identity labels -already match; a dirty source always invokes the Docker build and relies on layer -caching. A model/login-only reconfiguration compares only the verified release -image identity, so a changed source alone does not rebuild on reconfigure. -Setup writes that release receipt to the machine-local, generated root file -`.pipelines-releases.json` (with atomic `.pipelines-releases-*` temporary files) -and both are git-ignored, so a fresh clean checkout stays clean after setup and -reinstall still reuses the verified release/image identity. The actual built -image's labels stay authoritative, clean-checkout identities/reinstall reuse are -unchanged, and a dirty source still rebuilds. +On a new computer `./install` asks for the password (8+ characters, typed +twice). Without a terminal, set `PIPELINES_WEB_PASSWORD`. It is stored in +`~/.config/pipelines/web-auth.json` (mode 0600) with a generated cookie secret, +and reused unchanged on every reinstall. An invalid or non-private file stops +the installer rather than being reset. -Docker mode detects the computer's LAN IPv4 address and timezone automatically, -configures `http://LAN_IP:8090`, and binds both Docker services to `0.0.0.0` without asking -for networking settings (timezone falls back to UTC). On reinstall it updates only -the bind settings to `0.0.0.0`, refreshes the detected host LAN IP for Status, -and retains existing URLs/timezone/ports/credentials. It bootstraps -the publishing/phone accounts and publishing token, builds the server, installs -the skill inside the container, and opens **container OpenCode provider login** -on first setup. Choose any supported provider; signup is handled by that provider. -Provider credentials remain in the `opencode-home` volume. Reinstalls skip login -after completed setup; `./install --relogin` opens it again without -deleting accounts. Setup then chooses **separate saved defaults** in order: first the -interactive OpenCode harness model, then the new-pipeline model, in two -fuzzy-searchable pickers: type to filter, arrow keys or Tab navigate, Enter confirms, -the saved value is preselected and marked `(current)`, and Esc cancels with nothing -changed. `--model PROVIDER/MODEL` and `--harness-model -PROVIDER/MODEL` set them explicitly; reinstall retests the saved selection. Container -credentials and accounts stay separate from host credentials and are reused on -reinstall/reconfigure. Picker -selection performs no inference; the OpenCode path then requires a real, small -**tool-free model response** for the new-pipeline model before marking setup complete -(provider charges may apply). Login, discovery and validation prime -OpenCode's integration catalog in their own container/service. Discovery retries empty model -reads in the same container, avoiding a cold-start empty-catalog response. -An empty catalog is not proof of an invalid key; saved credentials are retained. -New drafts/templates inherit the new-pipeline selection; -published definitions and retry versions stay unchanged. Failed checks print -bounded, credential-redacted evidence and retain their OpenCode session. -The installer then starts/recreates the Compose services, -checks ntfy health through Pipelines, and sends an authenticated test notification. -It does not register an OS service, delete volumes, or seed pipeline definitions. -Phone reception is not verified by installation. +The same password is given to the ntfy `phone-reader` account when that account +is first created; existing ntfy accounts keep their passwords. -Docker setup mounts the host user's home and the installation checkout at their -original absolute paths, and runs Pipelines/OpenCode using that host user's UID -and GID. This makes a client invocation directory such as -`/home/tank/repos/pipelines` refer to the actual host checkout rather than an -unavailable container path or the unrelated image checkout at `/workspace`. -Stage permissions/edit scopes still apply; this is a filesystem mount, not a -workspace allowlist or automatic file transfer from another client computer. -For repositories outside those mounts, add a Compose bind at the same absolute -path. Reinstallation stops the old Pipelines service before changing ownership -of its managed data/auth/workspace volumes. It never recursively chowns the host -home/repository, deletes credentials, or removes volumes. Existing data/auth -volumes use `nocopy` so Docker cannot reset a freshly prepared empty volume's -ownership to the image UID. Setup verifies that the server can enter the host -checkout before reporting completion. +To change the web password, edit `password` in `web-auth.json` (keep it 0600) +and run `pipelines restart`. -For fresh Docker configuration you can supply `--ntfy-url https://ntfy.example.com ---bind-address 127.0.0.1 --timezone Europe/London --port 8081` to `./install --docker`. -The defaults are ready for direct phone access on your trusted LAN. Only use -plain HTTP on a trusted network. Other existing `.env` settings are not replaced. -These Docker flags apply to `./install --docker`; the explicit Docker mode fails -rather than silently falling back to native binaries if the LAN address cannot be -detected. Use an interactive terminal for first Docker/provider setup. -The Docker Pipelines API is unauthenticated: keep its port on a trusted network -and do not forward it to the Internet. For a manual loopback-only deployment, -set `PIPELINES_BIND_ADDRESS=127.0.0.1` after installation and use Compose directly. -Ctrl+C cancels cleanly without deleting configuration, credentials or volumes. -If Go is available, Docker setup also updates standard host CLI/TUI binaries and -the skill. Restart an already-running TUI to load the changes. Without Go, the -installer explicitly warns that the host client was not updated; Docker still runs -the server, and you can build the client later with host Go. See [Docker setup](docker.md). - -After ntfy verification, setup also checks the published Pipelines endpoint from -the host (when Go or a host CLI is available) and saves it in -`$XDG_CONFIG_HOME/pipelines/client.json`, or `~/.config/pipelines/client.json`. -Only `server_url` is updated; unrelated settings are preserved. CLI/TUI precedence -is `--url`, then `PIPELINES_URL`, then the saved default, then localhost:8080. -Docker's default host mapping is port 8081 (the container listens on 8080), so an -old native server on 8080 must not accidentally become the client target. Setup warns if an environment override still selects -another address; it does not stop or delete that other server. - -If Status says ntfy is **not configured**, check which Pipelines server the client -is reaching before reinstalling. On the Docker server computer: +## Upgrading ```sh -docker compose port pipelines 8080 -pipelines --url ws://127.0.0.1:8081/connect status --json -pipelines --url ws://127.0.0.1:8081/connect tui +pipelines update ``` -Use the actual mapped port instead of 8081 if customized. On a different computer, -use the Docker host's LAN address instead of 127.0.0.1. The saved default applies -only on the computer/user that ran the installer. Phone reception remains untested -by these checks. +`update` refuses while runs are executing (finish or cancel them first). It +then runs `git pull --ff-only` in the checkout it was installed from (skipped +when the branch has no upstream), re-runs `./install` with the recorded +arguments, restarts, and prints the startup report. Data, passwords, ntfy +accounts and the signing key are kept. If the installer fails, the previous +version keeps running. -## Failure evidence and confirmations - -TUI run/retry/cancel requests cannot be sent before their confirmation modal. -Draft creation/editing and opening a debugging session also require approval. -CLI mutations prompt with a default of No; `--yes` is explicit noninteractive -approval. The standalone TUI always keeps its modals, including with `-yes`. - -`pipelines debug RUN_ID --no-open` retrieves the retained run plus read-only -OpenCode session evidence from the **server**, not the client's unrelated service. -Debug includes the exact pinned pipeline definition and full run snapshot directly. -Large context packets are retained in a private client cache file which OpenCode -is instructed to read, rather than discarding evidence or exceeding argument limits. -New failed operations retain operation, provider/model, executable, directory, -timing, exit/HTTP status and bounded stdout/stderr under `opencode_failures` in -the step transcript. Credentials are redacted, but project/provider messages may -still contain private context: review before sharing. Older failures cannot gain -discarded subprocess output retroactively; Debug attempts retained session reads. - -Native OpenCode schema support is checked with a retained, deny-all, no-inference -POST/readback probe. OpenCode 1.18.35 accepts explicit formats but rejects their -stored messages on readback; affected backends omit `format` and require the same -strict local `{outcome, output}` validation. Compatible backends retain schema -mode with no automatic tool-replaying schema retries. The compatibility decision -and probe session are recorded, without rewriting published definitions/history. -For old format-bearing sessions whose HTTP readback fails, Debug uses read-only -`opencode export` with an exact session-ID check to recover original messages. -`StructuredOutputError` is surfaced directly; DSML/tool-call markup and flattened -results are rejected and retained, never silently interpreted as successful tool -execution or wrapped into a guessed result. Native HTTP evidence distinguishes -adapter/HTTP failures from actual subprocess exit codes. - -To validate a model on the execution computer, run: - -```sh -pipelines provider-check --model PROVIDER/MODEL -# Docker credentials are separate from host credentials (plain CLI: no TTY): -docker compose exec -T pipelines pipelines provider-check --model PROVIDER/MODEL -``` - -For a developer or legacy host build, a private repository uses your normal Git -credentials and `GOPRIVATE` to compile the client/host binary (this is not a -supported server installation): - -```sh -GOPRIVATE=git.a-wan.net go install git.a-wan.net/alan/pipelines/cmd/...@latest -pipelines skill install -``` - -For a system-wide command, build as your normal user and install the three binaries -into an administrator-selected bin directory using your normal system permissions. -Services and OpenCode accounts still run as the logged-in user, never as root. -The installer does not hardcode a system destination or run sudo. - -## Developer/legacy host server (unsupported) - -This explicit foreground server is for developing Pipelines itself or running a -previous Python-supervised legacy host setup. It is **not** the native installer; -a fresh database starts empty. - -```sh -pipelines serve # foreground developer/legacy server; Ctrl+C stops it -# In another terminal: -pipelines -``` - -The native installer binds the Go server to loopback by default and exposes the -authenticated web proxy instead. An explicit `--bind-address` changes the manifest. -The Pipelines API -has no authentication, so keep that port on a trusted LAN/VPN. Change into your -repository and run a pipeline there; the client sends its current directory to the -server. The server's own working directory does not restrict which repositories -you can use. Model tool permissions and approved edit paths remain separate, -per-run/stage controls. - -The SQLite database uses the standard user data location, honoring `XDG_DATA_HOME` -when set. On macOS it uses the user Application Support directory. `-db FILE` -selects another database, including an existing one; nothing is imported or moved -automatically. `pipelines serve --help` prints the actual defaults on your machine. -Drafts use the OS user cache; `PIPELINES_DRAFT_DIR` overrides that location. - -## Developer/legacy autostart (unsupported) - -For a standalone developer binary (not the CLI inside a native install), stop any -existing foreground or legacy server first, then explicitly configure the OS user service: - -```sh -pipelines service enable -pipelines service status -``` - -This starts the server now and at subsequent logins. Linux uses -`systemctl --user`; macOS uses a LaunchAgent in your user `Library/LaunchAgents`. -There is no sudo, Docker dependency, or second supervisor. `--db` -and `--addr` select the database/listen address, just like foreground startup. -Re-run enable to update the definition and restart with the new settings. - -```sh -pipelines service enable --print # preview only -pipelines service disable # stop and remove -``` - -Setup records the current PATH, HOME, XDG directory overrides, selected OpenCode -executable, and supported ntfy URL/token-file settings in a private service file. -It does not copy provider credentials or the rest of the shell environment. -Use `PIPELINES_NTFY_TOKEN_FILE`, not an inline token. Re-run enable after changing -these settings or moving binaries. Relative database/token-file paths are resolved -at configuration time. Services do not read your shell startup files. - -On Linux this is **login autostart**, not automatically boot-without-login. -If needed, explicitly enable lingering using your system's normal administration -policy (`loginctl enable-linger "$USER"`). Pipelines does not change that policy. -macOS LaunchAgents likewise require a user login. Windows autostart is unsupported. -Use `journalctl --user -u pipelines.service` for Linux logs. On macOS inspect -`pipelines service status`; run `pipelines serve` in the foreground for diagnostics. - -## OpenCode runtime (developer/legacy host) - -For a developer or legacy host server, use your existing `opencode` on PATH, or set -`PIPELINES_OPENCODE` to an explicit executable. No OpenCode version is required just -to install binaries. At runtime: - -- The standard/native API release uses an owned, authenticated loopback - `opencode serve` process, scoped to the requested repository and using the - existing user's configuration/login. The API, not a version number, is checked. -- A CLI advertising `api` uses the V2 command interface and its normal discovery. -- Both adapters select the pipeline's configured model explicitly and preserve - stage tool permissions. Native `shell` permissions map to OpenCode's `bash`. -- Audited native sessions record their backend and a private service reference - so recovery addresses the same service, rather than a new unrelated instance. -- `new`/`edit` open the normal interface directly with `@pipelines` and a draft - context-file reference. The skill handles questions and draft validation inside - OpenCode; no preliminary session/API calls or instruction preload are performed. - -The global skill is installed in `$XDG_CONFIG_HOME/opencode/skills/pipelines`, -or the normal `~/.config/opencode/skills/pipelines` default. Only unchanged files -previously installed by Pipelines are updated. A user-managed/modified skill is -never overwritten; preserve or rename it before running `pipelines skill install`. -Installation does not require OpenCode itself. Uninstall retains the skill and drafts. -The packaged format, notification, operations, transfer and verification references -are self-contained; agents read only those relevant to the task. Authoring -does not require Pipelines source in the current repository. Without a task prompt, -OpenCode is instructed to ask what to create/change before investigating files or -running CLI commands. See [authoring session review](authoring-session-review.md) -for the reasoning and verification limits. -New features and confirmed limitations are maintained in the packaged skill alongside -the topic docs. Skill tests check that every embedded reference is installed and -its links resolve. Refresh the global managed skill from current source with -`go run ./cmd/pipelines skill install`; this does not deploy server/runtime fixes. - -The bundled examples retain `opencode-go/deepseek-v4.1-flash`. Availability and -authentication for that model are runtime requirements, not installation checks. -No login or provider credentials are copied or exported by Pipelines. - -## Optional notifications - -Installation configures this automatically via `./install`. -ntfy is independent of Android: it is an optional notification channel for any -compatible subscriber, including a separate ntfy app. A web-only user does not -need an Android client, and observing that the browser is the only client is not -a reason to disable an existing ntfy configuration. Browser-closed web push is -**not implemented**; installing/configuring ntfy does not add that capability. -For a developer/legacy host server, use an existing ntfy server or the Docker setup in [docs/ntfy.md](ntfy.md). -Configure `PIPELINES_NTFY_URL`, `PIPELINES_NTFY_PUBLIC_URL`, and a publishing token -via `PIPELINES_NTFY_TOKEN_FILE` in the server's environment. Keep secrets out of -pipeline definitions. Use HTTPS or an encrypted VPN for phone access; do not -expose an unauthenticated pipeline server or HTTP credentials to untrusted networks. +Running `./install` again from the checkout does the same without the pull. +Run `./install --apk` again to offer a new APK to phones. ## Uninstall -For the native installer, select the manifest and run `pipelines uninstall`: - ```sh -PIPELINES_PREFIX="$HOME/.local/share/pipelines" "$HOME/.local/share/pipelines/bin/pipelines" uninstall +pipelines uninstall # keeps data and credentials in +pipelines uninstall --purge # also deletes , including the database ``` -This stops its supervisor and removes its login definition and managed binaries; -data and credentials are retained. The following applies only to standalone -developer binaries: disable `pipelines service disable` if configured, then: +Neither touches OpenCode, Claude Code, Codex, Pi or files in +`~/.config/pipelines`. + +## Developer server + +For working on Pipelines itself, run a server in the foreground on spare +ports. This is not the supported installation and does not serve the web app: ```sh -pipelines uninstall -# or pipelines uninstall --yes +go run ./cmd/pipelines-server -db /tmp/dev.db -addr 127.0.0.1:8081 +go run ./cmd/pipelines --url ws://127.0.0.1:8081/connect tui ``` -Only Go binaries whose embedded build metadata matches this repository are -removed from the current command's bin directory. History, drafts, OpenCode, -and optional ntfy data remain untouched. Use the permissions appropriate to your -chosen bin directory. Custom database paths cannot be inferred, so uninstallation -does not purge data. +`go run ./cmd/pipelines-server -help` lists the server flags (`-addr`, `-db`, +`-allow-authoring`, `-opencode-endpoint`). Without `-db` it uses the standard +per-user data directory (`XDG_DATA_HOME`, or Application Support on macOS). -## Existing legacy managed installations +## Troubleshooting -The previous Python-supervised setup is retained only for explicit legacy use. -Set `PIPELINES_PREFIX` to its existing directory for its lifecycle commands; -legacy manifests are never inferred from an executable location. The old -`--system-command`, provisioning, and autostart installer flags have been removed. - -To migrate, stop the old server first, install normal Go binaries, and start with -`pipelines serve -db EXISTING_DATABASE`. Keep the old -data and credentials until you have verified the new setup. There is no automatic -purge, database move, login migration, or shared OpenCode service modification. +- **"Docker is not running"**: start Docker Desktop (or the daemon) and rerun. +- **Ports busy**: a Docker installation or an old server may hold 8080, 8082 or + 8090. Stop it, or choose other ports with the flags above. +- **"No usable configured harness model"**: rerun `./install` in a terminal to + pick models, or pass `--model PROVIDER/MODEL`. +- **CLI talks to the wrong server**: `PIPELINES_URL` overrides the saved + default; unset it. Check `pipelines connection`. +- **Web app missing**: the server refuses to start without it. Run + `./install --web`. +- **A harness works in the terminal but not in the service**: the service uses + the `PATH` recorded at install time. Rerun `./install` from a shell where the + harness is on `PATH`. See [Harnesses](harnesses.md#troubleshooting). diff --git a/docs/notifications.md b/docs/notifications.md new file mode 100644 index 00000000..479b804a --- /dev/null +++ b/docs/notifications.md @@ -0,0 +1,90 @@ +# Notifications + +The server records every notification once and delivers it two ways: + +1. **In the Pipelines app** (Android, and the web page while it is open), over + the app's own connection. No ntfy app is needed. +2. **ntfy** (optional fallback), for devices that still use the ntfy app. + +The same settings decide what is sent on both paths. + +## What notifies + +| Event | When | +| --- | --- | +| Run needs your decision | a run reaches a human decision stage (runs not set to Silent) | +| Run failed / succeeded | runs started with **Notify me** (`--notify on`), or every run when **Notify when any run ends** is on | +| Stage hook | a stage's own `notify` block fires ([ntfy reference](ntfy.md)) | +| Chat needs you | a chat asks a question or requests approval | +| Chat replied | an assistant turn ends (off by default) | + +Each event notifies once. Opening or dismissing it on one device clears it on +the others. + +Chat events are seen while some client has the chat open or the server still +holds queued messages for it. A turn that finishes with no client attached and +nothing queued does not produce a "replied" notification. + +## Settings + +In the app, **Settings**: + +**Notifications** asks for the platform's notification permission. + +**Notifications in Pipelines** (Android only): + +- **Receive notifications in Pipelines**: starts a background service that + keeps its own connection to the server and posts notifications. Android shows + a quiet "Pipelines is connected" notification while it runs. It restarts + after a reboot or an app update. +- **Status**: the service's connection state. +- **Allow notifications**: shown when Android is blocking Pipelines + notifications. +- **Allow background delivery**: asks Android to exempt Pipelines from battery + optimisation so delivery stays instant while the phone is idle. Grant it. + +**Phone notifications** (stored on the server, shared by every device): + +| Setting | Default | +| --- | --- | +| Notify when any run ends (otherwise only runs started with **Notify me**) | off | +| Chat needs your answer or approval | on | +| Chat reply finished | off | +| Also send to ntfy (for devices that still use the ntfy app) | on | + +Turning **Also send to ntfy** off stops publishing to ntfy; the app still +receives everything. + +Per run, choose **Notify me**, the default (auto), or **Silent** when starting +it. From the CLI: `pipelines run … --notify on|auto|off`. + +## The web app + +The web page shows notifications only while it is open and the browser allows +them. Closed-browser web push is not implemented. + +## ntfy (optional) + +`./install` builds and runs ntfy on port 8090 with a private `pipelines` topic. +The installer prints the subscription details at the end, and +`pipelines doctor` repeats them (`Phone server … · topic: pipelines · +username: phone-reader`). + +To subscribe from the ntfy app: + +1. Add a server: the ntfy address (default `http://:8090`, or the + `--ntfy-url` you installed with). Off your home network, use a VPN such as + Tailscale. +2. Sign in as `phone-reader`. A freshly created account uses the web sign-in + password; an account kept from an earlier install keeps its old password. +3. Subscribe to the topic `pipelines`. + +A tap opens `pipelines://run/` or `pipelines://chat/` in the Android +app; the **Open in browser** action opens the same page on +`PIPELINES_WEB_PUBLIC_URL`. + +Notifications already shown by the ntfy app cannot be retracted by the server; +swipe them away there. + +See [ntfy reference](ntfy.md) for stage `notify` hooks, explicit ntfy stages, +the app notification protocol and network troubleshooting. diff --git a/docs/ntfy.md b/docs/ntfy.md index 61a4ce31..e33790c5 100644 --- a/docs/ntfy.md +++ b/docs/ntfy.md @@ -1,5 +1,9 @@ # Self-hosted ntfy delivery +For the user-facing overview (in-app Android delivery, settings, subscribing) +see [Notifications](notifications.md). This page is the reference for stage +hooks, the app notification protocol and ntfy itself. + Every stage can have an optional `notify` property block. Notifications go to your own ntfy instance and can attach Markdown and output files. It does not ask an LLM to handle credentials or run curl. The full Markdown remains in the run @@ -179,7 +183,7 @@ explicit `type=ntfy` stages are not affected. | `run_finished` — Notify when any run ends | off | every **auto**-mode run that fails or succeeds notifies; otherwise only runs started with **Notify me** (`on`) announce their end | | `chat_needs_you` — Chat needs your answer or approval | on | chat question/form and approval requests notify | | `chat_reply_finished` — Chat reply finished | off | each finished assistant turn in a chat notifies | -| `ntfy_enabled` — Publish to ntfy | on | when off, nothing is published to ntfy; the app notification stream (below) still delivers every notification | +| `ntfy_enabled` — Also send to ntfy | on | when off, nothing is published to ntfy; the app notification stream (below) still delivers every notification | Change them in the app under **Settings → Phone notifications** (checkboxes; a refused change reverts and shows the server's reason). Over the websocket @@ -342,7 +346,7 @@ phone-reachable address if attachment URLs currently point to a LAN-only address that setting is not a substitute for fixing a TCP timeout. Install the ntfy app, add your own server, authenticate with the reader account, -and subscribe to `anime`. Ensure notifications and required background operation +and subscribe to `pipelines`. Ensure notifications and required background operation are allowed. Outside your home network, both topic subscriptions and attachment downloads need access to your server through VPN or public HTTPS.