d1846441cc
Record every notification published to ntfy (run outbox, stage notify hooks, chat question/approval/reply) in a persisted app_notifications outbox and stream it to subscribe_notifications clients with an after cursor; clear_notifications and ack_attention clear across devices, as do runs leaving waiting, deleted runs and answered chat questions. Add the notify.ntfy_enabled setting (default on) to stop only the ntfy publish.
386 lines
21 KiB
Markdown
386 lines
21 KiB
Markdown
# Self-hosted ntfy delivery
|
|
|
|
Every stage can have an optional `notify` property block. Notifications go to
|
|
your own ntfy instance and can attach Markdown and output files. It does not ask
|
|
an LLM to handle credentials or run curl. The full Markdown remains in the run
|
|
audit as well as the ntfy attachment. A separate HTML report viewer is not included.
|
|
|
|
## Per-stage notifications (recommended)
|
|
|
|
### One terminal-path alert without adding stages
|
|
|
|
For a sequential workflow such as `research → report`, where intermediate
|
|
blocked/error routes go straight to `#failure`, add `on=failure` notification
|
|
hooks to intermediate stages and `on=success,failure` to the final stage.
|
|
Keep the graph and output contracts unchanged; omit `artifacts` for event-only
|
|
alerts. This avoids intermediate success alerts while covering early stage
|
|
errors and the final success/error result. Putting a hook only on the last stage
|
|
misses earlier failures; putting success hooks on every stage sends progress alerts.
|
|
|
|
These are stage hooks, not a universal run-finalizer: they run before the final
|
|
status is saved, and do not guarantee notifications for cancellation, crashes,
|
|
persistence failures or runtime step/visit limits. Recovery routes and loops can
|
|
notify before termination or more than once. An explicit native publisher stage
|
|
is not necessary for ordinary completion alerts and cannot reliably consume the
|
|
required `title`/`summary`/`markdown` after an upstream execution error.
|
|
|
|
The installed authoring skill includes this recipe and native-delivery details in
|
|
`references/notifications.md`; authoring does not require a checkout of this project.
|
|
|
|
````markdown
|
|
# stage compose
|
|
|
|
```notify
|
|
on=success,failure
|
|
artifacts=markdown,files
|
|
```
|
|
|
|
...stage output, prompt, tools, edges, and configs...
|
|
````
|
|
|
|
- `on` is `success`, `failure`, or `success,failure`. A failure is an execution
|
|
error, the `error` outcome, or a route to `#failure`; any other completed stage
|
|
is a success. This reports the stage's result, not the final run's result.
|
|
- All notification hooks use one system-wide server-to-phone connection. Stages
|
|
do not choose topics or addresses. Configure `PIPELINES_NTFY_URL` (publishing
|
|
topic URL), `PIPELINES_NTFY_PUBLIC_URL` (phone-facing server base URL), and
|
|
`PIPELINES_NTFY_TOKEN` or `PIPELINES_NTFY_TOKEN_FILE` once on the server.
|
|
The topic is `pipelines` in the provided setup. No block means no per-stage
|
|
notifications; a run that needs a person is still announced automatically (see
|
|
below). Legacy `destination` lines remain parseable for immutable-history compatibility
|
|
but no longer select a connection for notification hooks. Explicit native
|
|
`type=ntfy` stages retain their separate destination configuration.
|
|
- `artifacts=markdown` sends the string in `output.markdown` as `digest.md`.
|
|
- `artifacts=files` sends paths listed in `output.artifacts`, e.g.
|
|
`["reports/result.txt", "screenshots/chart.png"]`. Model stages with this policy
|
|
are instructed to list files they actually created/modified. No workspace scan
|
|
or file discovery occurs; an invalid model response may leave no artifact list.
|
|
- Both artifact kinds can be selected. Each file gets a separate notification
|
|
(ntfy supports one attachment per notification). If none were produced, an
|
|
event-only notification is still sent, including on failure.
|
|
- Files must be workspace-relative regular files, at most 1 MiB each and 10 files
|
|
per stage. Traversal, absolute paths, symlinks, hidden paths, known database/key
|
|
paths, `secrets/`, and `node_modules/` are rejected. These guards are not a
|
|
content scanner/DLP system: only list files you intend to share.
|
|
- Notifications have a 30-second total budget, with no automatic retries. Their
|
|
receipts/failures appear in the selected run stage's **Metadata**, separately
|
|
from output. They do not change routing or convert a successful stage into a
|
|
failed run. A crash can leave a `sending` receipt with uncertain delivery;
|
|
notifications are not replayed during recovery. Tool access follows the stage's
|
|
declared tools; notification settings do not grant additional tools.
|
|
|
|
## Automatic run-attention notifications
|
|
|
|
Independently of any `notify` block, the server publishes one run-level
|
|
notification to the `system` destination when a run needs a person or ends and
|
|
was marked to notify. The run's notify mode decides:
|
|
|
|
| Run notify mode | Enters `waiting` | Fails / succeeds |
|
|
| --- | --- | --- |
|
|
| **on** ("Notify me" in the app, `--notify on` in the CLI) | `Run needs your decision` | `Run failed` / `Run succeeded` |
|
|
| **auto** (default) | `Run needs your decision` | silent, unless **Notify when any run ends** is on |
|
|
| **off** ("Silent") | silent | silent |
|
|
|
|
Queued, running, canceled and interrupted runs are always silent. The **Notify
|
|
when any run ends** setting (`run_finished`, off by default) makes every auto
|
|
run's end notify too; it is read in the same transaction that records the run's
|
|
final state, so a change applies to runs finishing after it. A stage `notify`
|
|
block is the pipeline's own opt-in and still publishes for an auto run; when it
|
|
covers the same event the run-level alert is not sent again.
|
|
The payload names the run and its pipeline. Run notifications and stage hooks
|
|
use the configured web run link described below.
|
|
|
|
### Open notifications in the web app
|
|
|
|
Set `PIPELINES_WEB_PUBLIC_URL` on the **Pipelines server** to the authenticated web
|
|
proxy's phone-reachable origin, for example `https://pipelines.example.com` or
|
|
`http://192.168.1.20:8082` on a trusted LAN. Notifications then carry
|
|
`https://pipelines.example.com/runs/<id>`, opening that exact run in the browser
|
|
(or installed web app). The existing sign-in gate remains; after login the web
|
|
proxy returns to the requested run.
|
|
|
|
Use the address you actually open on the phone, not localhost, the API port, or
|
|
the ntfy address. No credentials, query, fragment or subpath are allowed. This
|
|
does not expose a port or add authentication to the underlying control API.
|
|
An invalid configured origin omits the click action and logs a configuration
|
|
warning rather than leaking credentials. Without this setting, existing
|
|
Android-only installations retain `pipelines://run/<id>` for compatibility.
|
|
|
|
Native installs save a default from the web proxy's advertised LAN address/port.
|
|
Override it with `PIPELINES_WEB_PUBLIC_URL=https://pipelines.example.com ./install`;
|
|
reinstalls preserve that saved choice unless overridden. Custom services set the
|
|
environment variable in their service definition; `pipelines service enable`
|
|
preserves it. Docker Compose accepts the same variable, pointing to a separately
|
|
reachable authenticated web proxy. Apply service changes only after active work
|
|
is finished. Already-delivered notifications keep their original click target;
|
|
verify a new notification after configuration is installed.
|
|
|
|
A genuine event notifies once: a run that stays `waiting` is not republished on a
|
|
read or restart, and a stage `notify` policy that already covers the same event
|
|
(`on=waiting`, or a failure hook) suppresses the run-level default, so one event
|
|
means one notification. A failed delivery is logged, never fails the run, and is
|
|
not retried. The Android app shows the same information in one attention surface
|
|
ordered by importance — a run waiting for a decision, then a failed run, then a
|
|
run that finished since the operator last looked — derived from the run events the
|
|
server already pushes, so it needs no refresh and starts nothing.
|
|
|
|
## Tapping a notification
|
|
|
|
Run notifications (stage hooks and run-level waiting/failed/succeeded alerts)
|
|
set the ntfy `Click` to the app deep link `pipelines://run/<id>`, so a tap on the
|
|
phone opens the native Pipelines app at that run. When `PIPELINES_WEB_PUBLIC_URL`
|
|
is set to a valid HTTP(S) origin (no credentials, path, query or fragment), each
|
|
notification also gets one ntfy `view` action, **Open in browser**, pointing at
|
|
`<web>/runs/<id>`. Without it, or with an invalid value, only the app link is
|
|
sent. Queued run notifications derive both links from the run id at delivery,
|
|
so rows queued by older servers (which stored a web URL) also open the app.
|
|
|
|
## Chats that need you
|
|
|
|
When a chat asks the operator something — an unresolved question/form or a
|
|
permission (approval) request — the server publishes one notification to the
|
|
`system` destination: title `<chat title> needs your answer` (or `... needs your
|
|
approval` for permissions; `Chat` when the session has no title), the question or
|
|
permission text as the body, `Click` `pipelines://chat/<session id>`, and the
|
|
**Open in browser** action to `<web>/chat/<session id>` when
|
|
`PIPELINES_WEB_PUBLIC_URL` is set. These are controlled by the **Chat needs your
|
|
answer or approval** setting (`chat_needs_you`, on by default).
|
|
|
|
Finished replies notify only when **Chat reply finished**
|
|
(`chat_reply_finished`) is turned on (off by default): when an assistant turn
|
|
ends (the authoritative, non-intermediate step end), the server publishes
|
|
`<chat title> replied` (`Chat replied` without a title) with the first ~200
|
|
characters of the turn's final assistant text as the body, or `The agent
|
|
finished its reply.` when no text was retained, plus the same click and browser
|
|
action. Each turn notifies once: a repeated or replayed end with no turn
|
|
observed in between, or a turn id already announced, publishes nothing, and
|
|
reading history never publishes.
|
|
|
|
- Each form/request id notifies at most once per server process: duplicate
|
|
events, reconnects and replays of a still-pending question are deduplicated in
|
|
memory, and a form first seen already resolved or cancelled never notifies.
|
|
The record is not persisted, so a request still pending across a server
|
|
restart can notify once more.
|
|
- The server only observes a chat while at least one client is subscribed to
|
|
it, so questions raised while no client watches the session are not seen.
|
|
- Publishing is asynchronous within the same 30-second budget, never blocks or
|
|
fails chat handling, and failures are only logged. Nothing is sent when
|
|
`PIPELINES_NTFY_URL` is not configured.
|
|
|
|
## Notification settings
|
|
|
|
Global, server-side switches choose which of these events are published.
|
|
They are stored in the server database (`server_settings`), survive restarts and
|
|
apply to every device subscribed to the topic. Per-stage `notify` hooks and
|
|
explicit `type=ntfy` stages are not affected.
|
|
|
|
| Setting | Default | Effect |
|
|
| --- | --- | --- |
|
|
| `run_finished` — Notify when any run ends | off | every **auto**-mode run that fails or succeeds notifies; otherwise only runs started with **Notify me** (`on`) announce their end |
|
|
| `chat_needs_you` — Chat needs your answer or approval | on | chat question/form and approval requests notify |
|
|
| `chat_reply_finished` — Chat reply finished | off | each finished assistant turn in a chat notifies |
|
|
| `ntfy_enabled` — Publish to ntfy | on | when off, nothing is published to ntfy; the app notification stream (below) still delivers every notification |
|
|
|
|
Change them in the app under **Settings → Phone notifications** (checkboxes; a
|
|
refused change reverts and shows the server's reason). Over the websocket
|
|
protocol, `{"type":"notification_settings"}` reads them and
|
|
`{"type":"set_notification_settings","notification_settings":{"chat_reply_finished":true}}`
|
|
changes any subset; both reply with type `notification_settings` carrying every
|
|
key. Unknown keys and non-boolean values are refused with an `error`
|
|
reply and nothing is written.
|
|
|
|
`run_finished` changed meaning: it once meant "auto runs announce success" and
|
|
defaulted on. It is now stored under `notify.any_run_ended`, so a value saved
|
|
under the old meaning (`notify.run_finished`) is ignored and every server starts
|
|
from the new default (off).
|
|
|
|
## Clearing notifications across devices
|
|
|
|
The app's in-app attention state is shared through the server: dismissing an
|
|
entry or using **Clear all** in the Needs-you sheet, and opening a chat (which
|
|
clears its **New** badge), clears it on every connected browser and phone at
|
|
once, and on a disconnected device when it next connects. The protocol is
|
|
`{"type":"attention_acks"}` (read the whole set) and
|
|
`{"type":"ack_attention","attention_acks":{"dismissed":["42:Failed"],"chat_seen":{"ses_x":1730000000000}}}`
|
|
(add to it). Both reply `attention_acks`, and every other connection receives an
|
|
`attention_acks_changed` frame with the change. Acknowledgements for deleted runs
|
|
or sessions, or older than 30 days, are pruned.
|
|
|
|
This does **not** reach notifications the ntfy app has already shown on a phone:
|
|
once ntfy delivered a message, the server cannot retract it, so those must still
|
|
be swiped away in the ntfy app (or the Android notification shade).
|
|
|
|
## App notification stream (no ntfy app needed)
|
|
|
|
Every notification the server publishes to ntfy — run waiting/failed/succeeded
|
|
alerts from the run outbox, stage `notify` hook events, and chat
|
|
question/approval/reply alerts — is first recorded in the `app_notifications`
|
|
table (kept 7 days) and streamed over `/connect` to clients that subscribe, so
|
|
the Pipelines app can post them itself. The same settings decide what is
|
|
recorded, so the stream carries exactly the set ntfy gets, with the same title,
|
|
message and `click` deep link. A stage hook with several attachments is one
|
|
stream notification per step event; explicit `type=ntfy` stages are not
|
|
streamed.
|
|
|
|
```text
|
|
→ {"type":"subscribe_notifications","id":"1","after":"run:7:3:waiting"}
|
|
← {"type":"notifications_subscribed","id":"1","notifications":[N...],"ids":["<cleared id>",...]}
|
|
← {"type":"notification","notification":N} (push)
|
|
← {"type":"notifications_cleared","ids":["run:7:3:waiting"]} (push)
|
|
→ {"type":"clear_notifications","id":"2","ids":["run:7:3:waiting"]}
|
|
← {"type":"notifications_cleared","id":"2","ids":[<ids it cleared>]}
|
|
|
|
N = {"id":"run:7:3:waiting","kind":"run_waiting","title":"Run needs your decision",
|
|
"message":"Run #7 · release · waiting for a human decision at stage approve",
|
|
"click":"pipelines://run/7","priority":3,"tags":[],
|
|
"created_at":"2026-11-01T10:00:00Z","group":"run:7"}
|
|
```
|
|
|
|
- `id` is stable: `run:<run>:<step>:<event>` (event `waiting`, `failed`,
|
|
`succeeded`, or `success` for a mid-run stage hook) or
|
|
`chat:<session>:<form|permission|reply>:<id>`. `kind` is one of `run_waiting`,
|
|
`run_failed`, `run_succeeded`, `stage_failed`, `stage_succeeded`,
|
|
`chat_question`, `chat_approval`, `chat_reply`. `group` is `run:<id>` or
|
|
`chat:<id>`. `priority` is 3 and `tags` empty, matching what ntfy receives.
|
|
- The subscribe reply holds the active (uncleared) notifications newer than the
|
|
`after` cursor (the last id the client holds), oldest first, at most 100;
|
|
an empty, unknown or pruned cursor replays them all. Its `ids` lists
|
|
notifications cleared within the 7 days (newest first, at most 500) so a
|
|
reconnecting device can dismiss ones it still shows. Pushes follow in order;
|
|
subscribing again replaces the subscription.
|
|
- `clear_notifications` (the app's dismiss or tap) has the same effect as the
|
|
matching `ack_attention`: `run_waiting` dismisses `<run>:Decision`,
|
|
`run_failed`/`stage_failed` `<run>:Failed`, and a chat notification marks the
|
|
chat seen (clearing its other notifications too). Other subscribers receive
|
|
`notifications_cleared`; connections receive `attention_acks_changed`.
|
|
- Notifications are also cleared, with a push, by `ack_attention` from any
|
|
client, by a run leaving `waiting` (its waiting notifications), by deleting a
|
|
run, and by an answered or cancelled chat question or approval.
|
|
|
|
## Explicit delivery stage (optional)
|
|
|
|
The preceding stage must output three nonempty UTF-8 strings: `title`, `summary`,
|
|
and `markdown`. Title/summary must be single-line (256/2048 byte limits); Markdown
|
|
is limited to 1 MiB. Native delivery stages do not accept model prompts/tools or
|
|
inline credentials. They require `destination` and a `success` edge:
|
|
|
|
````markdown
|
|
# stage notify
|
|
|
|
```configs
|
|
type=ntfy
|
|
destination=system
|
|
```
|
|
|
|
```edges
|
|
success -> #success
|
|
error -> #failure
|
|
```
|
|
````
|
|
|
|
Configure the **pipeline server process** outside the pipeline file:
|
|
|
|
```sh
|
|
export PIPELINES_NTFY_URL='https://ntfy.your-domain.example/pipelines'
|
|
# Set PIPELINES_NTFY_TOKEN_FILE to the existing publishing token file.
|
|
go run ./cmd/pipelines-server
|
|
```
|
|
|
|
The URL includes one topic, without credentials, query parameters, or a fragment.
|
|
`destination=system` uses the same connection as hooks. Other destinations use
|
|
the named pattern: `destination=reports` reads
|
|
`PIPELINES_NTFY_REPORTS_URL` and `PIPELINES_NTFY_REPORTS_TOKEN` (or `_TOKEN_FILE`
|
|
pointing to a secret file). Set only one token source. Destination names
|
|
use lowercase letters, digits, and underscores. A token is optional only when
|
|
the configured server permits anonymous publishing; private deployments need it.
|
|
Never put tokens in a prompt, committed file, or chat. Environment credentials
|
|
are inherited by subprocesses: do not grant untrusted stages shell/code-execution
|
|
access. Prefer a dedicated publishing user with write-only access to this topic.
|
|
|
|
## Local server
|
|
|
|
The root `compose.yaml` runs **both the pipeline server and ntfy** with persistent
|
|
volumes, internal service networking, and a private topic. See the
|
|
[Docker + Android setup](docker.md) for bootstrap, credentials, and timezone
|
|
detection. The previous ntfy-only Compose file has been replaced. Isolated real
|
|
Docker health and publishing checks have passed; phone reception and the user's
|
|
remote Tailscale connectivity remain **not tested** locally.
|
|
|
|
Use a TLS reverse proxy or a private VPN (e.g. Tailscale). A phone cannot reach
|
|
your computer's `127.0.0.1`; that address is only for local publishing/proxying.
|
|
To bind the container to a particular LAN/VPN interface, explicitly set
|
|
`NTFY_BIND_ADDRESS` to that interface IP. Do not expose an unauthenticated server,
|
|
or send access tokens over public plain HTTP. Set `NTFY_BASE_URL` to the same
|
|
address your phone uses so attachment URLs resolve correctly. If the pipeline
|
|
server is another container, localhost is that container, not the ntfy host.
|
|
|
|
## Phone
|
|
|
|
### Tailscale address times out while LAN works
|
|
|
|
Use `http://<server-Tailscale-IP>:8090`, not port 80; enable Tailscale on both the
|
|
server and phone. The default Compose mapping binds ntfy to `0.0.0.0:8090`.
|
|
Binding to all interfaces does not bypass tailnet grants/ACLs, host firewalls,
|
|
or Docker forwarding rules. A timeout alone cannot identify which is blocking.
|
|
Do not disable firewalls or publicly forward this port as a workaround.
|
|
|
|
On the server, these are read-only diagnostics (the remote checks are not tested
|
|
from this development machine):
|
|
|
|
```sh
|
|
tailscale status
|
|
docker compose port ntfy 80
|
|
curl --connect-timeout 5 http://127.0.0.1:8090/v1/health
|
|
curl --connect-timeout 5 "http://$(tailscale ip -4):8090/v1/health"
|
|
```
|
|
|
|
If the host health checks succeed but the phone times out, check that the phone
|
|
is in the intended tailnet and that tailnet access rules/firewalls allow this
|
|
phone to reach server TCP 8090. Once reachable, log in as `phone-reader` for the
|
|
Tailscale server entry, separately from the LAN entry. Set `NTFY_BASE_URL` to a
|
|
phone-reachable address if attachment URLs currently point to a LAN-only address;
|
|
that setting is not a substitute for fixing a TCP timeout.
|
|
|
|
Install the ntfy app, add your own server, authenticate with the reader account,
|
|
and subscribe to `anime`. Ensure notifications and required background operation
|
|
are allowed. Outside your home network, both topic subscriptions and attachment
|
|
downloads need access to your server through VPN or public HTTPS.
|
|
|
|
For **iOS instant notifications**, ntfy's current documentation requires
|
|
`NTFY_UPSTREAM_BASE_URL=https://ntfy.sh` (or an APNS-connected upstream with a
|
|
custom app). The upstream sends wake-up requests; message content is fetched
|
|
from your instance. This is not completely independent of third-party push
|
|
infrastructure. Without upstream forwarding, delivery can be substantially
|
|
delayed. Phone delivery/background behavior has **not been tested** here.
|
|
|
|
## Execution and failure behavior
|
|
|
|
- Publishing is bounded to 20 seconds and does not follow redirects.
|
|
- No automatic publish retry: timeouts or malformed acknowledgments may mean the
|
|
notification was delivered but its acknowledgment was lost. Inspect ntfy
|
|
before retrying an ambiguous failure.
|
|
- For the optional native `type=ntfy` stage, a confirmed response adds `delivery` metadata (destination name, message ID,
|
|
attachment URL, filename) to the existing output. Failures take the error route
|
|
and are recorded in normal run/step audits, without remote response bodies or
|
|
credentials in diagnostics.
|
|
- Repeating the same run request key does not publish again. A new run or a graph
|
|
loop visiting delivery again can send another notification; content-level
|
|
"only new items" deduplication is not yet implemented. For the initial digest,
|
|
keep notification last with no feedback edge.
|
|
- Markdown reports remain available in pipeline run history even after ntfy expires its
|
|
attachment. Configure ntfy retention for your needs; its attachment directory
|
|
must be dedicated to ntfy, not shared with other files.
|
|
|
|
File-attachment bytes are stored by ntfy, not permanently archived in SQLite;
|
|
run metadata stores the delivered filename/URL, and workspace files may later
|
|
change. Download attachments before your ntfy retention expires.
|
|
|
|
Bundled starters do not require notification or schedule setup. Add a `notify`
|
|
block to the stages of your own workflow using the example above; configure the
|
|
system phone connection first. Adding a schedule activates it on publication,
|
|
so review it explicitly rather than copying a scheduled niche example.
|
|
|
|
References: [publishing](https://docs.ntfy.sh/publish/),
|
|
[server configuration](https://docs.ntfy.sh/config/),
|
|
[iOS self-hosting](https://docs.ntfy.sh/config/#ios-instant-notifications).
|