a48e4a5b6a
The template catalog is now Example 1-4: one stage, two stages (handoff), feedback loop (runner-owned limits, retries, timeout) and human decision (type=human approve/revise/abort with notify hooks). All read-only or leave changes uncommitted; none commits, pushes or deploys. A brand-new database (schema created by this Open) gets the examples once at server start, bound to the configured default model and harness. A server_settings marker records the decision, so removed examples never come back and existing installs are marked without any pipeline being added or rewritten.
391 lines
22 KiB
Markdown
391 lines
22 KiB
Markdown
# Self-hosted ntfy delivery
|
|
|
|
For the user-facing overview (in-app Android delivery, settings, subscribing)
|
|
see [Notifications](notifications.md). This page is the reference for stage
|
|
hooks, the app notification protocol and ntfy itself.
|
|
|
|
Every stage can have an optional `notify` property block. Notifications go to
|
|
your own ntfy instance and can attach Markdown and output files. It does not ask
|
|
an LLM to handle credentials or run curl. The full Markdown remains in the run
|
|
audit as well as the ntfy attachment. A separate HTML report viewer is not included.
|
|
|
|
## Per-stage notifications (recommended)
|
|
|
|
### One terminal-path alert without adding stages
|
|
|
|
For a sequential workflow such as `research → report`, where intermediate
|
|
blocked/error routes go straight to `#failure`, add `on=failure` notification
|
|
hooks to intermediate stages and `on=success,failure` to the final stage.
|
|
Keep the graph and output contracts unchanged; omit `artifacts` for event-only
|
|
alerts. This avoids intermediate success alerts while covering early stage
|
|
errors and the final success/error result. Putting a hook only on the last stage
|
|
misses earlier failures; putting success hooks on every stage sends progress alerts.
|
|
|
|
These are stage hooks, not a universal run-finalizer: they run before the final
|
|
status is saved, and do not guarantee notifications for cancellation, crashes,
|
|
persistence failures or runtime step/visit limits. Recovery routes and loops can
|
|
notify before termination or more than once. An explicit native publisher stage
|
|
is not necessary for ordinary completion alerts and cannot reliably consume the
|
|
required `title`/`summary`/`markdown` after an upstream execution error.
|
|
|
|
The installed authoring skill includes this recipe and native-delivery details in
|
|
`references/notifications.md`; authoring does not require a checkout of this project.
|
|
|
|
````markdown
|
|
# stage compose
|
|
|
|
```notify
|
|
on=success,failure
|
|
artifacts=markdown,files
|
|
```
|
|
|
|
...stage output, prompt, tools, edges, and configs...
|
|
````
|
|
|
|
- `on` is `success`, `failure`, or `success,failure`. A failure is an execution
|
|
error, the `error` outcome, or a route to `#failure`; any other completed stage
|
|
is a success. This reports the stage's result, not the final run's result.
|
|
- All notification hooks use one system-wide server-to-phone connection. Stages
|
|
do not choose topics or addresses. Configure `PIPELINES_NTFY_URL` (publishing
|
|
topic URL), `PIPELINES_NTFY_PUBLIC_URL` (phone-facing server base URL), and
|
|
`PIPELINES_NTFY_TOKEN` or `PIPELINES_NTFY_TOKEN_FILE` once on the server.
|
|
The topic is `pipelines` in the provided setup. No block means no per-stage
|
|
notifications; a run that needs a person is still announced automatically (see
|
|
below). Legacy `destination` lines remain parseable for immutable-history compatibility
|
|
but no longer select a connection for notification hooks. Explicit native
|
|
`type=ntfy` stages retain their separate destination configuration.
|
|
- `artifacts=markdown` sends the string in `output.markdown` as `digest.md`.
|
|
- `artifacts=files` sends paths listed in `output.artifacts`, e.g.
|
|
`["reports/result.txt", "screenshots/chart.png"]`. Model stages with this policy
|
|
are instructed to list files they actually created/modified. No workspace scan
|
|
or file discovery occurs; an invalid model response may leave no artifact list.
|
|
- Both artifact kinds can be selected. Each file gets a separate notification
|
|
(ntfy supports one attachment per notification). If none were produced, an
|
|
event-only notification is still sent, including on failure.
|
|
- Files must be workspace-relative regular files, at most 1 MiB each and 10 files
|
|
per stage. Traversal, absolute paths, symlinks, hidden paths, known database/key
|
|
paths, `secrets/`, and `node_modules/` are rejected. These guards are not a
|
|
content scanner/DLP system: only list files you intend to share.
|
|
- Notifications have a 30-second total budget, with no automatic retries. Their
|
|
receipts/failures appear in the selected run stage's **Metadata**, separately
|
|
from output. They do not change routing or convert a successful stage into a
|
|
failed run. A crash can leave a `sending` receipt with uncertain delivery;
|
|
notifications are not replayed during recovery. Tool access follows the stage's
|
|
declared tools; notification settings do not grant additional tools.
|
|
|
|
## Automatic run-attention notifications
|
|
|
|
Independently of any `notify` block, the server publishes one run-level
|
|
notification to the `system` destination when a run needs a person or ends and
|
|
was marked to notify. The run's notify mode decides:
|
|
|
|
| Run notify mode | Enters `waiting` | Fails / succeeds |
|
|
| --- | --- | --- |
|
|
| **on** ("Notify me" in the app, `--notify on` in the CLI) | `Run needs your decision` | `Run failed` / `Run succeeded` |
|
|
| **auto** (default) | `Run needs your decision` | silent, unless **Notify when any run ends** is on |
|
|
| **off** ("Silent") | silent | silent |
|
|
|
|
Queued, running, canceled and interrupted runs are always silent. The **Notify
|
|
when any run ends** setting (`run_finished`, off by default) makes every auto
|
|
run's end notify too; it is read in the same transaction that records the run's
|
|
final state, so a change applies to runs finishing after it. A stage `notify`
|
|
block is the pipeline's own opt-in and still publishes for an auto run; when it
|
|
covers the same event the run-level alert is not sent again.
|
|
The payload names the run and its pipeline. Run notifications and stage hooks
|
|
use the configured web run link described below.
|
|
|
|
### Open notifications in the web app
|
|
|
|
Set `PIPELINES_WEB_PUBLIC_URL` on the **Pipelines server** to the authenticated web
|
|
proxy's phone-reachable origin, for example `https://pipelines.example.com` or
|
|
`http://192.168.1.20:8082` on a trusted LAN. Notifications then carry
|
|
`https://pipelines.example.com/runs/<id>`, opening that exact run in the browser
|
|
(or installed web app). The existing sign-in gate remains; after login the web
|
|
proxy returns to the requested run.
|
|
|
|
Use the address you actually open on the phone, not localhost, the API port, or
|
|
the ntfy address. No credentials, query, fragment or subpath are allowed. This
|
|
does not expose a port or add authentication to the underlying control API.
|
|
An invalid configured origin omits the click action and logs a configuration
|
|
warning rather than leaking credentials. Without this setting, existing
|
|
Android-only installations retain `pipelines://run/<id>` for compatibility.
|
|
|
|
Native installs save a default from the web proxy's advertised LAN address/port.
|
|
Override it with `PIPELINES_WEB_PUBLIC_URL=https://pipelines.example.com ./install`;
|
|
reinstalls preserve that saved choice unless overridden. Custom services set the
|
|
environment variable in their service definition; `pipelines service enable`
|
|
preserves it. Docker Compose accepts the same variable, pointing to a separately
|
|
reachable authenticated web proxy. Apply service changes only after active work
|
|
is finished. Already-delivered notifications keep their original click target;
|
|
verify a new notification after configuration is installed.
|
|
|
|
A genuine event notifies once: a run that stays `waiting` is not republished on a
|
|
read or restart, and a stage `notify` policy that already covers the same event
|
|
(`on=waiting`, or a failure hook) suppresses the run-level default, so one event
|
|
means one notification. A failed delivery is logged, never fails the run, and is
|
|
not retried. The Android app shows the same information in one attention surface
|
|
ordered by importance — a run waiting for a decision, then a failed run, then a
|
|
run that finished since the operator last looked — derived from the run events the
|
|
server already pushes, so it needs no refresh and starts nothing.
|
|
|
|
## Tapping a notification
|
|
|
|
Run notifications (stage hooks and run-level waiting/failed/succeeded alerts)
|
|
set the ntfy `Click` to the app deep link `pipelines://run/<id>`, so a tap on the
|
|
phone opens the native Pipelines app at that run. When `PIPELINES_WEB_PUBLIC_URL`
|
|
is set to a valid HTTP(S) origin (no credentials, path, query or fragment), each
|
|
notification also gets one ntfy `view` action, **Open in browser**, pointing at
|
|
`<web>/runs/<id>`. Without it, or with an invalid value, only the app link is
|
|
sent. Queued run notifications derive both links from the run id at delivery,
|
|
so rows queued by older servers (which stored a web URL) also open the app.
|
|
|
|
## Chats that need you
|
|
|
|
When a chat asks the operator something — an unresolved question/form or a
|
|
permission (approval) request — the server publishes one notification to the
|
|
`system` destination: title `<chat title> needs your answer` (or `... needs your
|
|
approval` for permissions; `Chat` when the session has no title), the question or
|
|
permission text as the body, `Click` `pipelines://chat/<session id>`, and the
|
|
**Open in browser** action to `<web>/chat/<session id>` when
|
|
`PIPELINES_WEB_PUBLIC_URL` is set. These are controlled by the **Chat needs your
|
|
answer or approval** setting (`chat_needs_you`, on by default).
|
|
|
|
Finished replies notify only when **Chat reply finished**
|
|
(`chat_reply_finished`) is turned on (off by default): when an assistant turn
|
|
ends (the authoritative, non-intermediate step end), the server publishes
|
|
`<chat title> replied` (`Chat replied` without a title) with the first ~200
|
|
characters of the turn's final assistant text as the body, or `The agent
|
|
finished its reply.` when no text was retained, plus the same click and browser
|
|
action. Each turn notifies once: a repeated or replayed end with no turn
|
|
observed in between, or a turn id already announced, publishes nothing, and
|
|
reading history never publishes.
|
|
|
|
- Each form/request id notifies at most once per server process: duplicate
|
|
events, reconnects and replays of a still-pending question are deduplicated in
|
|
memory, and a form first seen already resolved or cancelled never notifies.
|
|
The record is not persisted, so a request still pending across a server
|
|
restart can notify once more.
|
|
- The server only observes a chat while at least one client is subscribed to
|
|
it, so questions raised while no client watches the session are not seen.
|
|
- Publishing is asynchronous within the same 30-second budget, never blocks or
|
|
fails chat handling, and failures are only logged. Nothing is sent when
|
|
`PIPELINES_NTFY_URL` is not configured.
|
|
|
|
## Notification settings
|
|
|
|
Global, server-side switches choose which of these events are published.
|
|
They are stored in the server database (`server_settings`), survive restarts and
|
|
apply to every device subscribed to the topic. Per-stage `notify` hooks and
|
|
explicit `type=ntfy` stages are not affected.
|
|
|
|
| Setting | Default | Effect |
|
|
| --- | --- | --- |
|
|
| `run_finished` — Notify when any run ends | off | every **auto**-mode run that fails or succeeds notifies; otherwise only runs started with **Notify me** (`on`) announce their end |
|
|
| `chat_needs_you` — Chat needs your answer or approval | on | chat question/form and approval requests notify |
|
|
| `chat_reply_finished` — Chat reply finished | off | each finished assistant turn in a chat notifies |
|
|
| `ntfy_enabled` — Also send to ntfy | on | when off, nothing is published to ntfy; the app notification stream (below) still delivers every notification |
|
|
|
|
Change them in the app under **Settings → Phone notifications** (checkboxes; a
|
|
refused change reverts and shows the server's reason). Over the websocket
|
|
protocol, `{"type":"notification_settings"}` reads them and
|
|
`{"type":"set_notification_settings","notification_settings":{"chat_reply_finished":true}}`
|
|
changes any subset; both reply with type `notification_settings` carrying every
|
|
key. Unknown keys and non-boolean values are refused with an `error`
|
|
reply and nothing is written.
|
|
|
|
`run_finished` changed meaning: it once meant "auto runs announce success" and
|
|
defaulted on. It is now stored under `notify.any_run_ended`, so a value saved
|
|
under the old meaning (`notify.run_finished`) is ignored and every server starts
|
|
from the new default (off).
|
|
|
|
## Clearing notifications across devices
|
|
|
|
The app's in-app attention state is shared through the server: dismissing an
|
|
entry or using **Clear all** in the Needs-you sheet, and opening a chat (which
|
|
clears its **New** badge), clears it on every connected browser and phone at
|
|
once, and on a disconnected device when it next connects. The protocol is
|
|
`{"type":"attention_acks"}` (read the whole set) and
|
|
`{"type":"ack_attention","attention_acks":{"dismissed":["42:Failed"],"chat_seen":{"ses_x":1730000000000}}}`
|
|
(add to it). Both reply `attention_acks`, and every other connection receives an
|
|
`attention_acks_changed` frame with the change. Acknowledgements for deleted runs
|
|
or sessions, or older than 30 days, are pruned.
|
|
|
|
This does **not** reach notifications the ntfy app has already shown on a phone:
|
|
once ntfy delivered a message, the server cannot retract it, so those must still
|
|
be swiped away in the ntfy app (or the Android notification shade).
|
|
|
|
## App notification stream (no ntfy app needed)
|
|
|
|
Every notification the server publishes to ntfy — run waiting/failed/succeeded
|
|
alerts from the run outbox, stage `notify` hook events, and chat
|
|
question/approval/reply alerts — is first recorded in the `app_notifications`
|
|
table (kept 7 days) and streamed over `/connect` to clients that subscribe, so
|
|
the Pipelines app can post them itself. The same settings decide what is
|
|
recorded, so the stream carries exactly the set ntfy gets, with the same title,
|
|
message and `click` deep link. A stage hook with several attachments is one
|
|
stream notification per step event; explicit `type=ntfy` stages are not
|
|
streamed.
|
|
|
|
```text
|
|
→ {"type":"subscribe_notifications","id":"1","after":"run:7:3:waiting"}
|
|
← {"type":"notifications_subscribed","id":"1","notifications":[N...],"ids":["<cleared id>",...]}
|
|
← {"type":"notification","notification":N} (push)
|
|
← {"type":"notifications_cleared","ids":["run:7:3:waiting"]} (push)
|
|
→ {"type":"clear_notifications","id":"2","ids":["run:7:3:waiting"]}
|
|
← {"type":"notifications_cleared","id":"2","ids":[<ids it cleared>]}
|
|
|
|
N = {"id":"run:7:3:waiting","kind":"run_waiting","title":"Run needs your decision",
|
|
"message":"Run #7 · release · waiting for a human decision at stage approve",
|
|
"click":"pipelines://run/7","priority":3,"tags":[],
|
|
"created_at":"2026-11-01T10:00:00Z","group":"run:7"}
|
|
```
|
|
|
|
- `id` is stable: `run:<run>:<step>:<event>` (event `waiting`, `failed`,
|
|
`succeeded`, or `success` for a mid-run stage hook) or
|
|
`chat:<session>:<form|permission|reply>:<id>`. `kind` is one of `run_waiting`,
|
|
`run_failed`, `run_succeeded`, `stage_failed`, `stage_succeeded`,
|
|
`chat_question`, `chat_approval`, `chat_reply`. `group` is `run:<id>` or
|
|
`chat:<id>`. `priority` is 3 and `tags` empty, matching what ntfy receives.
|
|
- The subscribe reply holds the active (uncleared) notifications newer than the
|
|
`after` cursor (the last id the client holds), oldest first, at most 100;
|
|
an empty, unknown or pruned cursor replays them all. Its `ids` lists
|
|
notifications cleared within the 7 days (newest first, at most 500) so a
|
|
reconnecting device can dismiss ones it still shows. Pushes follow in order;
|
|
subscribing again replaces the subscription.
|
|
- `clear_notifications` (the app's dismiss or tap) has the same effect as the
|
|
matching `ack_attention`: `run_waiting` dismisses `<run>:Decision`,
|
|
`run_failed`/`stage_failed` `<run>:Failed`, and a chat notification marks the
|
|
chat seen (clearing its other notifications too). Other subscribers receive
|
|
`notifications_cleared`; connections receive `attention_acks_changed`.
|
|
- Notifications are also cleared, with a push, by `ack_attention` from any
|
|
client, by a run leaving `waiting` (its waiting notifications), by deleting a
|
|
run, and by an answered or cancelled chat question or approval.
|
|
|
|
## Explicit delivery stage (optional)
|
|
|
|
The preceding stage must output three nonempty UTF-8 strings: `title`, `summary`,
|
|
and `markdown`. Title/summary must be single-line (256/2048 byte limits); Markdown
|
|
is limited to 1 MiB. Native delivery stages do not accept model prompts/tools or
|
|
inline credentials. They require `destination` and a `success` edge:
|
|
|
|
````markdown
|
|
# stage notify
|
|
|
|
```configs
|
|
type=ntfy
|
|
destination=system
|
|
```
|
|
|
|
```edges
|
|
success -> #success
|
|
error -> #failure
|
|
```
|
|
````
|
|
|
|
Configure the **pipeline server process** outside the pipeline file:
|
|
|
|
```sh
|
|
export PIPELINES_NTFY_URL='https://ntfy.your-domain.example/pipelines'
|
|
# Set PIPELINES_NTFY_TOKEN_FILE to the existing publishing token file.
|
|
go run ./cmd/pipelines-server
|
|
```
|
|
|
|
The URL includes one topic, without credentials, query parameters, or a fragment.
|
|
`destination=system` uses the same connection as hooks. Other destinations use
|
|
the named pattern: `destination=reports` reads
|
|
`PIPELINES_NTFY_REPORTS_URL` and `PIPELINES_NTFY_REPORTS_TOKEN` (or `_TOKEN_FILE`
|
|
pointing to a secret file). Set only one token source. Destination names
|
|
use lowercase letters, digits, and underscores. A token is optional only when
|
|
the configured server permits anonymous publishing; private deployments need it.
|
|
Never put tokens in a prompt, committed file, or chat. Environment credentials
|
|
are inherited by subprocesses: do not grant untrusted stages shell/code-execution
|
|
access. Prefer a dedicated publishing user with write-only access to this topic.
|
|
|
|
## Local server
|
|
|
|
The root `compose.yaml` runs **both the pipeline server and ntfy** with persistent
|
|
volumes, internal service networking, and a private topic. See the
|
|
[Docker + Android setup](docker.md) for bootstrap, credentials, and timezone
|
|
detection. The previous ntfy-only Compose file has been replaced. Isolated real
|
|
Docker health and publishing checks have passed; phone reception and the user's
|
|
remote Tailscale connectivity remain **not tested** locally.
|
|
|
|
Use a TLS reverse proxy or a private VPN (e.g. Tailscale). A phone cannot reach
|
|
your computer's `127.0.0.1`; that address is only for local publishing/proxying.
|
|
To bind the container to a particular LAN/VPN interface, explicitly set
|
|
`NTFY_BIND_ADDRESS` to that interface IP. Do not expose an unauthenticated server,
|
|
or send access tokens over public plain HTTP. Set `NTFY_BASE_URL` to the same
|
|
address your phone uses so attachment URLs resolve correctly. If the pipeline
|
|
server is another container, localhost is that container, not the ntfy host.
|
|
|
|
## Phone
|
|
|
|
### Tailscale address times out while LAN works
|
|
|
|
Use `http://<server-Tailscale-IP>:8090`, not port 80; enable Tailscale on both the
|
|
server and phone. The default Compose mapping binds ntfy to `0.0.0.0:8090`.
|
|
Binding to all interfaces does not bypass tailnet grants/ACLs, host firewalls,
|
|
or Docker forwarding rules. A timeout alone cannot identify which is blocking.
|
|
Do not disable firewalls or publicly forward this port as a workaround.
|
|
|
|
On the server, these are read-only diagnostics (the remote checks are not tested
|
|
from this development machine):
|
|
|
|
```sh
|
|
tailscale status
|
|
docker compose port ntfy 80
|
|
curl --connect-timeout 5 http://127.0.0.1:8090/v1/health
|
|
curl --connect-timeout 5 "http://$(tailscale ip -4):8090/v1/health"
|
|
```
|
|
|
|
If the host health checks succeed but the phone times out, check that the phone
|
|
is in the intended tailnet and that tailnet access rules/firewalls allow this
|
|
phone to reach server TCP 8090. Once reachable, log in as `phone-reader` for the
|
|
Tailscale server entry, separately from the LAN entry. Set `NTFY_BASE_URL` to a
|
|
phone-reachable address if attachment URLs currently point to a LAN-only address;
|
|
that setting is not a substitute for fixing a TCP timeout.
|
|
|
|
Install the ntfy app, add your own server, authenticate with the reader account,
|
|
and subscribe to `pipelines`. Ensure notifications and required background operation
|
|
are allowed. Outside your home network, both topic subscriptions and attachment
|
|
downloads need access to your server through VPN or public HTTPS.
|
|
|
|
For **iOS instant notifications**, ntfy's current documentation requires
|
|
`NTFY_UPSTREAM_BASE_URL=https://ntfy.sh` (or an APNS-connected upstream with a
|
|
custom app). The upstream sends wake-up requests; message content is fetched
|
|
from your instance. This is not completely independent of third-party push
|
|
infrastructure. Without upstream forwarding, delivery can be substantially
|
|
delayed. Phone delivery/background behavior has **not been tested** here.
|
|
|
|
## Execution and failure behavior
|
|
|
|
- Publishing is bounded to 20 seconds and does not follow redirects.
|
|
- No automatic publish retry: timeouts or malformed acknowledgments may mean the
|
|
notification was delivered but its acknowledgment was lost. Inspect ntfy
|
|
before retrying an ambiguous failure.
|
|
- For the optional native `type=ntfy` stage, a confirmed response adds `delivery` metadata (destination name, message ID,
|
|
attachment URL, filename) to the existing output. Failures take the error route
|
|
and are recorded in normal run/step audits, without remote response bodies or
|
|
credentials in diagnostics.
|
|
- Repeating the same run request key does not publish again. A new run or a graph
|
|
loop visiting delivery again can send another notification; content-level
|
|
"only new items" deduplication is not yet implemented. For the initial digest,
|
|
keep notification last with no feedback edge.
|
|
- Markdown reports remain available in pipeline run history even after ntfy expires its
|
|
attachment. Configure ntfy retention for your needs; its attachment directory
|
|
must be dedicated to ntfy, not shared with other files.
|
|
|
|
File-attachment bytes are stored by ntfy, not permanently archived in SQLite;
|
|
run metadata stores the delivered filename/URL, and workspace files may later
|
|
change. Download attachments before your ntfy retention expires.
|
|
|
|
Bundled examples need no schedule setup. **Example 4: human decision** shows
|
|
`notify` hooks (`plan` on failure, `decide` on `waiting`, `act` on success and
|
|
failure); they only deliver once the system phone connection is configured. Add a
|
|
`notify` block to the stages of your own workflow the same way. Adding a schedule activates it on publication,
|
|
so review it explicitly rather than copying a scheduled niche example.
|
|
|
|
References: [publishing](https://docs.ntfy.sh/publish/),
|
|
[server configuration](https://docs.ntfy.sh/config/),
|
|
[iOS self-hosting](https://docs.ntfy.sh/config/#ios-instant-notifications).
|