Notify on run end only for runs marked Notify me

Auto runs no longer alert on failure or success; only notify_mode on does,
unless the run_finished setting (now 'Notify when any run ends', default off)
opts into every run end. Waiting still notifies auto and on runs. The setting
is stored under a new key so a value saved under the old meaning is ignored.
This commit is contained in:
2026-10-31 16:52:04 +00:00
parent 4235cd858f
commit b109ab3e06
14 changed files with 188 additions and 98 deletions
@@ -4,15 +4,25 @@ import '../../core/api_gateway.dart';
/// One server-side ntfy notification choice, in the server's wire key.
class NtfySetting {
const NtfySetting(this.key, this.label);
const NtfySetting(this.key, this.label, {this.subtitle});
final String key;
final String label;
/// Optional one-line explanation shown under [label].
final String? subtitle;
}
/// The server's global notification choices, in display order.
///
/// `run_finished` (server default off) makes every run end notify; without it
/// only runs started with "Notify me" announce their success or failure.
const ntfySettings = <NtfySetting>[
NtfySetting('run_finished', 'Run finished'),
NtfySetting(
'run_finished',
'Notify when any run ends',
subtitle: 'Otherwise only runs started with Notify me',
),
NtfySetting('chat_needs_you', 'Chat needs your answer or approval'),
NtfySetting('chat_reply_finished', 'Chat reply finished'),
];
@@ -140,6 +140,7 @@ class _SettingsNtfySectionState extends State<SettingsNtfySection> {
return CupertinoListTile(
key: ntfySettingKey(setting.key),
title: Text(setting.label),
subtitle: setting.subtitle == null ? null : Text(setting.subtitle!),
// iOS settings rows toggle with a switch; an unchecked checkbox is
// nearly invisible on the grouped row.
trailing: CupertinoSwitch(value: value, onChanged: onChanged),
@@ -149,6 +150,7 @@ class _SettingsNtfySectionState extends State<SettingsNtfySection> {
return CheckboxListTile(
key: ntfySettingKey(setting.key),
title: Text(setting.label),
subtitle: setting.subtitle == null ? null : Text(setting.subtitle!),
value: value,
onChanged: onChanged == null ? null : (next) => onChanged(next!),
);
+2 -1
View File
@@ -2,7 +2,8 @@
Settings has a **Phone notifications** section beside the in-app
Notifications permission tile. It holds three checkboxes for the server's global
ntfy choices: **Run finished** (`run_finished`, default on), **Chat needs your
ntfy choices: **Notify when any run ends** (`run_finished`, default off,
subtitle "Otherwise only runs started with Notify me"), **Chat needs your
answer or approval** (`chat_needs_you`, default on) and **Chat reply finished**
(`chat_reply_finished`, default off), under one line saying they go through
ntfy to the subscribed devices. Material uses `CheckboxListTile`. Cupertino uses
@@ -65,7 +65,11 @@ void main() {
await tester.pumpWidget(_host(gateway, cupertino: cupertino));
await tester.pumpAndSettle();
expect(gateway.calls.single.$1, 'notification_settings');
expect(find.text('Run finished'), findsOneWidget);
expect(find.text('Notify when any run ends'), findsOneWidget);
expect(
find.text('Otherwise only runs started with Notify me'),
findsOneWidget,
);
expect(find.text('Chat needs your answer or approval'), findsOneWidget);
expect(find.text('Chat reply finished'), findsOneWidget);
expect(find.text(ntfySectionSubtitle), findsOneWidget);
@@ -77,6 +81,21 @@ void main() {
);
});
testWidgets('$style shows the server default: only marked runs', (
tester,
) async {
// The server's defaults: run ends notify only for "Notify me" runs.
final gateway = _Gateway((_, _) => _reply(false, true, false));
await tester.pumpWidget(_host(gateway, cupertino: cupertino));
await tester.pumpAndSettle();
expect(_checked(tester, 'run_finished', cupertino: cupertino), isFalse);
expect(_checked(tester, 'chat_needs_you', cupertino: cupertino), isTrue);
expect(
_checked(tester, 'chat_reply_finished', cupertino: cupertino),
isFalse,
);
});
testWidgets('$style toggling writes only that setting', (tester) async {
final gateway = _Gateway((type, fields) {
if (type == 'set_notification_settings') {
+5 -3
View File
@@ -681,9 +681,11 @@ appear with a `public/` prefix.
### Run notification modes and receipts
`--notify auto|on|off` records `notify_mode` on the run (also returned in
summaries and `run_event` frames). Auto retains stage policies and run-level
waiting/failure defaults. On also alerts terminal success without requiring a
stage hook. Off suppresses stage, run-level system and browser tray alerts.
summaries and `run_event` frames). Auto retains stage policies and the
run-level waiting alert; its failure or success is silent unless the server's
`run_finished` setting ("Notify when any run ends", off by default) is on. On
also alerts when the run fails or succeeds, without requiring a stage hook. Off
suppresses stage, run-level system and browser tray alerts.
`inspect --json` / `get_run` includes top-level `notifications` receipts with
`sequence`, `event`, `channel`, `status`, `attempts`, and optional `diagnostic`
+22 -11
View File
@@ -72,16 +72,22 @@ artifacts=markdown,files
## Automatic run-attention notifications
Independently of any `notify` block, the server publishes one run-level
notification to the `system` destination when a run needs a person: when it enters
`waiting` (a human stage asked for a decision), when it fails, and — while the
**Run finished** setting (`run_finished`, on by default) is enabled — when it
succeeds (`Run succeeded`). Queued, running, canceled and interrupted runs are
silent. The run's own notify mode still wins: a run started with notifications
**on** always notifies on success, a run started with notifications **off**
never notifies anything, and the default **auto** mode follows `run_finished`
for success (waiting/failed always notify). The setting is read in the same
transaction that records the run's final state, so a change applies to runs
finishing after it. The payload names the run and its pipeline. Run notifications and stage hooks
notification to the `system` destination when a run needs a person or ends and
was marked to notify. The run's notify mode decides:
| Run notify mode | Enters `waiting` | Fails / succeeds |
| --- | --- | --- |
| **on** ("Notify me" in the app, `--notify on` in the CLI) | `Run needs your decision` | `Run failed` / `Run succeeded` |
| **auto** (default) | `Run needs your decision` | silent, unless **Notify when any run ends** is on |
| **off** ("Silent") | silent | silent |
Queued, running, canceled and interrupted runs are always silent. The **Notify
when any run ends** setting (`run_finished`, off by default) makes every auto
run's end notify too; it is read in the same transaction that records the run's
final state, so a change applies to runs finishing after it. A stage `notify`
block is the pipeline's own opt-in and still publishes for an auto run; when it
covers the same event the run-level alert is not sent again.
The payload names the run and its pipeline. Run notifications and stage hooks
use the configured web run link described below.
### Open notifications in the web app
@@ -170,7 +176,7 @@ explicit `type=ntfy` stages are not affected.
| Setting | Default | Effect |
| --- | --- | --- |
| `run_finished` — Run finished | on | an **auto**-mode run that succeeds notifies `Run succeeded` |
| `run_finished` — Notify when any run ends | off | every **auto**-mode run that fails or succeeds notifies; otherwise only runs started with **Notify me** (`on`) announce their end |
| `chat_needs_you` — Chat needs your answer or approval | on | chat question/form and approval requests notify |
| `chat_reply_finished` — Chat reply finished | off | each finished assistant turn in a chat notifies |
@@ -182,6 +188,11 @@ changes any subset; both reply with type `notification_settings` carrying all
three keys. Unknown keys and non-boolean values are refused with an `error`
reply and nothing is written.
`run_finished` changed meaning: it once meant "auto runs announce success" and
defaulted on. It is now stored under `notify.any_run_ended`, so a value saved
under the old meaning (`notify.run_finished`) is ignored and every server starts
from the new default (off).
## Clearing notifications across devices
The app's in-app attention state is shared through the server: dismissing an
+19 -3
View File
@@ -13,6 +13,19 @@ import (
// default from protocol.DefaultNotificationSettings.
const notificationSettingPrefix = "notify."
// notificationStorageKeys maps a wire key to a different stored key when its
// meaning changed. run_finished used to mean "auto runs announce success"
// (default on); it now means "every run end notifies" (default off), so it is
// stored under a new key and any value saved under the old meaning is ignored.
var notificationStorageKeys = map[string]string{protocol.NotifyRunFinished: "any_run_ended"}
func notificationStorageKey(key string) string {
if stored, ok := notificationStorageKeys[key]; ok {
return stored
}
return key
}
type settingsQuerier interface {
QueryContext(context.Context, string, ...any) (*sql.Rows, error)
}
@@ -31,8 +44,11 @@ func readNotificationSettings(ctx context.Context, q settingsQuerier) (protocol.
if err := rows.Scan(&key, &value); err != nil {
return settings, err
}
if field := settings.Field(key[len(notificationSettingPrefix):]); field != nil {
*field = value == "true"
stored := key[len(notificationSettingPrefix):]
for _, wire := range protocol.NotificationSettingKeys {
if notificationStorageKey(wire) == stored {
*settings.Field(wire) = value == "true"
}
}
}
return settings, rows.Err()
@@ -58,7 +74,7 @@ func (s *Store) UpdateNotificationSettings(ctx context.Context, update map[strin
defer tx.Rollback()
for key, value := range values {
if _, err := tx.ExecContext(ctx, `INSERT INTO server_settings(key,value) VALUES(?,?)
ON CONFLICT(key) DO UPDATE SET value=excluded.value`, notificationSettingPrefix+key, fmt.Sprint(value)); err != nil {
ON CONFLICT(key) DO UPDATE SET value=excluded.value`, notificationSettingPrefix+notificationStorageKey(key), fmt.Sprint(value)); err != nil {
return protocol.NotificationSettings{}, err
}
}
+61 -50
View File
@@ -22,16 +22,16 @@ func TestNotificationSettingsDefaultPersistAndPartialUpdate(t *testing.T) {
if err != nil || got != protocol.DefaultNotificationSettings() {
t.Fatalf("defaults = %+v, %v", got, err)
}
if !got.RunFinished || !got.ChatNeedsYou || got.ChatReplyFinished {
if got.RunFinished || !got.ChatNeedsYou || got.ChatReplyFinished {
t.Fatalf("default values changed: %+v", got)
}
got, err = s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyChatReplyFinished: true})
want := protocol.NotificationSettings{RunFinished: true, ChatNeedsYou: true, ChatReplyFinished: true}
want := protocol.NotificationSettings{RunFinished: false, ChatNeedsYou: true, ChatReplyFinished: true}
if err != nil || got != want {
t.Fatalf("partial update = %+v, %v", got, err)
}
got, err = s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyRunFinished: false, protocol.NotifyChatNeedsYou: false})
want = protocol.NotificationSettings{RunFinished: false, ChatNeedsYou: false, ChatReplyFinished: true}
got, err = s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyRunFinished: true, protocol.NotifyChatNeedsYou: false})
want = protocol.NotificationSettings{RunFinished: true, ChatNeedsYou: false, ChatReplyFinished: true}
if err != nil || got != want {
t.Fatalf("second update = %+v, %v", got, err)
}
@@ -78,56 +78,67 @@ func TestNotificationSettingsRejectUnknownOrInvalid(t *testing.T) {
}
}
// A run reaching succeeded enqueues its generic alert according to its notify
// mode and, for auto, the run_finished setting read in the same transaction.
func TestSucceededRunEnqueueFollowsRunFinishedSetting(t *testing.T) {
for _, tc := range []struct {
mode string
runFinished bool
want int
}{
{"auto", true, 1}, {"auto", false, 0}, {"on", true, 1}, {"on", false, 1}, {"off", true, 0}, {"off", false, 0},
} {
t.Run(tc.mode+map[bool]string{true: "-enabled", false: "-disabled"}[tc.runFinished], func(t *testing.T) {
ctx := context.Background()
s, id := notificationTransitionRun(t, filepath.Join(t.TempDir(), "succeeded.db"), tc.mode)
defer s.Close()
if _, err := s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyRunFinished: tc.runFinished}); err != nil {
t.Fatal(err)
// A run's end (succeeded or failed) enqueues its generic alert only for a run
// marked "on", or for "auto" when run_finished (default off) opts into every
// run end. Waiting notifies auto and on runs regardless; "off" never notifies.
func TestRunEndEnqueueFollowsNotifyModeAndRunFinished(t *testing.T) {
for _, status := range []string{"succeeded", "failed", "waiting"} {
for _, tc := range []struct {
mode string
runFinished bool
want int
}{
{"auto", false, 0}, {"auto", true, 1}, {"on", false, 1}, {"on", true, 1}, {"off", false, 0}, {"off", true, 0},
} {
want := tc.want
if status == "waiting" && tc.mode == "auto" {
want = 1
}
if err := notificationTransition(ctx, s, id, "succeeded"); err != nil {
t.Fatal(err)
}
got, err := s.RunNotifications(ctx, id)
if err != nil || len(got) != tc.want {
t.Fatalf("notifications = %+v, %v; want %d", got, err, tc.want)
}
if tc.want == 1 {
n, err := s.ClaimRunNotification(ctx, time.Now().Unix())
if err != nil || n == nil || n.Event != "succeeded" || n.Title != "Run succeeded" || n.Click != "pipelines://run/"+strconv.FormatInt(id, 10) {
t.Fatalf("claimed = %+v, %v", n, err)
t.Run(status+"-"+tc.mode+map[bool]string{true: "-any", false: "-marked"}[tc.runFinished], func(t *testing.T) {
ctx := context.Background()
s, id := notificationTransitionRun(t, filepath.Join(t.TempDir(), "end.db"), tc.mode)
defer s.Close()
if tc.runFinished {
if _, err := s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyRunFinished: true}); err != nil {
t.Fatal(err)
}
}
}
})
if err := notificationTransition(ctx, s, id, status); err != nil {
t.Fatal(err)
}
got, err := s.RunNotifications(ctx, id)
if err != nil || len(got) != want {
t.Fatalf("notifications = %+v, %v; want %d", got, err, want)
}
if want == 1 {
title := map[string]string{"succeeded": "Run succeeded", "failed": "Run failed", "waiting": "Run needs your decision"}[status]
n, err := s.ClaimRunNotification(ctx, time.Now().Unix())
if err != nil || n == nil || n.Event != status || n.Title != title || n.Click != "pipelines://run/"+strconv.FormatInt(id, 10) {
t.Fatalf("claimed = %+v, %v", n, err)
}
}
})
}
}
}
// Waiting and failed alerts do not depend on run_finished.
func TestRunFinishedSettingLeavesAttentionAlone(t *testing.T) {
for _, status := range []string{"waiting", "failed"} {
t.Run(status, func(t *testing.T) {
ctx := context.Background()
s, id := notificationTransitionRun(t, filepath.Join(t.TempDir(), "attention.db"), "auto")
defer s.Close()
if _, err := s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyRunFinished: false}); err != nil {
t.Fatal(err)
}
if err := notificationTransition(ctx, s, id, status); err != nil {
t.Fatal(err)
}
if got, err := s.RunNotifications(ctx, id); err != nil || len(got) != 1 || got[0].Event != status {
t.Fatalf("notifications = %+v, %v", got, err)
}
})
// run_finished was stored as "notify.run_finished" when it meant "auto runs
// announce success" (default on). That stale value must not opt the operator
// into every run end under the new meaning.
func TestLegacyRunFinishedValueIsIgnored(t *testing.T) {
ctx := context.Background()
s, id := notificationTransitionRun(t, filepath.Join(t.TempDir(), "legacy.db"), "auto")
defer s.Close()
if _, err := s.db.ExecContext(ctx, `INSERT INTO server_settings(key,value) VALUES('notify.run_finished','true')`); err != nil {
t.Fatal(err)
}
if got, err := s.NotificationSettings(ctx); err != nil || got.RunFinished {
t.Fatalf("legacy value honoured: %+v, %v", got, err)
}
if err := notificationTransition(ctx, s, id, "failed"); err != nil {
t.Fatal(err)
}
if got, err := s.RunNotifications(ctx, id); err != nil || len(got) != 0 {
t.Fatalf("notifications = %+v, %v", got, err)
}
}
@@ -144,9 +144,10 @@ func TestRunNotificationTransitionModesAndCoverage(t *testing.T) {
if err != nil {
t.Fatal(err)
}
// run_finished defaults on, so an auto run announces success too.
// run_finished defaults off, so an auto run notifies only
// while waiting; its end is silent unless marked "on".
want := 1
if mode == "off" {
if mode == "off" || (mode == "auto" && status != "waiting") {
want = 0
}
if len(ns) != want {
+5 -8
View File
@@ -42,12 +42,9 @@ JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version
case "waiting":
title = "Run needs your decision"
body = "waiting for a human decision"
case "failed":
title = "Run failed"
body = "run failed"
case "succeeded":
// "on" always announces success; any other (auto/legacy) mode follows
// the operator's run_finished choice, read in this transaction.
case "failed", "succeeded":
// A run's end notifies only for runs marked "on", unless the operator
// opted into every run end via run_finished, read in this transaction.
if mode != "on" {
settings, err := readNotificationSettings(ctx, tx)
if err != nil {
@@ -57,8 +54,8 @@ JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version
return nil
}
}
title = "Run succeeded"
body = "run succeeded"
title = "Run " + status
body = "run " + status
default:
return nil
}
+4 -3
View File
@@ -27,8 +27,9 @@ const (
// NotificationSettings is the typed form of the global notification choices.
type NotificationSettings struct {
// RunFinished notifies when a run with notify mode "auto" succeeds. Runs
// started with notifications "on" always notify and "off" never do.
// RunFinished makes every run end (success or failure) notify, not only
// runs started with notify mode "on". Off by default: an "auto" run then
// notifies only when it waits for a human decision, and "off" never does.
RunFinished bool
// ChatNeedsYou notifies when a chat asks a question or requests approval.
ChatNeedsYou bool
@@ -41,7 +42,7 @@ var NotificationSettingKeys = []string{NotifyRunFinished, NotifyChatNeedsYou, No
// DefaultNotificationSettings are the values before the operator changes any.
func DefaultNotificationSettings() NotificationSettings {
return NotificationSettings{RunFinished: true, ChatNeedsYou: true, ChatReplyFinished: false}
return NotificationSettings{RunFinished: false, ChatNeedsYou: true, ChatReplyFinished: false}
}
// Map returns the wire form with every key present.
+9 -1
View File
@@ -334,12 +334,20 @@ func (r *runner) notifyRun(id int64, name, status string, sequence int, currentS
return
}
event := runAttentionEvent(status)
if status == "succeeded" && r.notifyMode(id) == "on" {
if status == "succeeded" {
event = "succeeded"
}
if event == "" {
return
}
// A run's end notifies only runs marked "on", unless the operator opted
// into every run end (run_finished); waiting notifies any non-off run.
if event != protocol.NotifyWaiting && r.notifyMode(id) != "on" {
settings, err := r.store.NotificationSettings(r.ctx)
if err != nil || !settings.RunFinished {
return
}
}
if r.attentionAlready(id, sequence, event) {
return
}
+4 -1
View File
@@ -331,7 +331,10 @@ func TestRunAttentionNotificationDelivery(t *testing.T) {
r.notifyRun(7, "release", protocol.RunStatusWaiting, 1, "approve", "")
// A run that stays waiting is never republished, on a later call or a re-request.
r.notifyRun(7, "release", protocol.RunStatusWaiting, 1, "approve", "")
// A failed run publishes once, with its diagnostic.
// An auto run's failure is silent: only runs marked "on" announce their end.
r.notifyRun(8, "release", "failed", 2, "deploy", "provider returned 500")
// A failed run marked "on" publishes once, with its diagnostic.
r.setNotifyMode(7, "on")
r.notifyRun(7, "release", "failed", 2, "deploy", "provider returned 500")
first := <-deliveries
+20 -12
View File
@@ -42,6 +42,8 @@ func TestRunNotificationPublisherFailureRetriesAfterReopen(t *testing.T) {
t.Setenv("PIPELINES_NTFY_TOKEN", "")
t.Setenv("PIPELINES_NTFY_TOKEN_FILE", "")
r := &runner{ctx: ctx, store: s}
// The run was started with "Notify me", so its failure is announced.
r.setNotifyMode(7, "on")
// Emulate a durable transition intent saved before the runner callback.
if err := s.EnqueueRunNotification(ctx, pipeline.RunNotification{RunID: 7, Sequence: 1, Event: "failed", Channel: "system", Title: "Run failed", Body: "Run 7", Click: "pipelines://run/7"}); err != nil {
t.Fatal(err)
@@ -60,6 +62,7 @@ func TestRunNotificationPublisherFailureRetriesAfterReopen(t *testing.T) {
}
defer s.Close()
r = &runner{ctx: ctx, store: s}
r.setNotifyMode(7, "on")
r.deliverRunNotifications(time.Now().Add(time.Minute))
r.notifyRun(7, "test", "failed", 1, "stage", "")
got, err = s.RunNotifications(ctx, 7)
@@ -250,14 +253,17 @@ func TestRunNotificationModesAndFinalStageDedupe(t *testing.T) {
for _, tc := range []struct {
mode string
hook bool
// quiet turns the global run_finished setting off (it defaults on).
quiet bool
want int32
// any turns the global run_finished setting on (it defaults off), so
// every run end notifies, not only runs marked "on".
any bool
want int32
}{
{"on", false, false, 1}, {"auto", false, false, 1}, {"auto", false, true, 0}, {"on", false, true, 1}, {"off", false, false, 0},
{"on", false, false, 1}, {"auto", false, false, 0}, {"auto", false, true, 1}, {"on", false, true, 1}, {"off", false, false, 0}, {"off", false, true, 0},
// A stage hook is the pipeline's own opt-in: it still publishes for an
// auto run, and never doubles up with the run alert.
{"on", true, false, 1}, {"auto", true, false, 1}, {"auto", true, true, 1}, {"off", true, false, 0},
} {
t.Run(tc.mode+map[bool]string{true: "-hook", false: "-no-hook"}[tc.hook]+map[bool]string{true: "-quiet", false: ""}[tc.quiet], func(t *testing.T) {
t.Run(tc.mode+map[bool]string{true: "-hook", false: "-no-hook"}[tc.hook]+map[bool]string{true: "-any", false: ""}[tc.any], func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
s, err := pipeline.Open(filepath.Join(t.TempDir(), "modes.db"))
@@ -265,8 +271,8 @@ func TestRunNotificationModesAndFinalStageDedupe(t *testing.T) {
t.Fatal(err)
}
defer s.Close()
if tc.quiet {
if _, err := s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyRunFinished: false}); err != nil {
if tc.any {
if _, err := s.UpdateNotificationSettings(ctx, map[string]any{protocol.NotifyRunFinished: true}); err != nil {
t.Fatal(err)
}
}
@@ -295,7 +301,7 @@ func TestRunNotificationModesAndFinalStageDedupe(t *testing.T) {
if err != nil || detail.Status != "succeeded" || detail.NotifyMode != tc.mode || calls.Load()-before != tc.want {
t.Fatalf("mode contract: %+v calls=%d %v", detail, calls.Load()-before, err)
}
if tc.want == 1 && !(tc.hook && tc.quiet) && (len(detail.Notifications) != 1 || (detail.Notifications[0].Status != "sent" && detail.Notifications[0].Status != "covered")) {
if tc.want == 1 && !(tc.hook && tc.mode == "auto" && !tc.any) && (len(detail.Notifications) != 1 || (detail.Notifications[0].Status != "sent" && detail.Notifications[0].Status != "covered")) {
t.Fatalf("missing final audit: %+v", detail.Notifications)
}
cancel()
@@ -328,14 +334,16 @@ func TestRunNotificationAttentionModesAndRetryBound(t *testing.T) {
r.notifyRun(id, "test", status, sequence, "stage", "")
}
}
if calls.Load() != 4 {
t.Fatalf("attention modes: %d calls, want 4", calls.Load())
// auto waiting, on waiting and on failed; an auto run's failure is silent
// because run_finished defaults off.
if calls.Load() != 3 {
t.Fatalf("attention modes: %d calls, want 3", calls.Load())
}
r.deliverRunNotifications(time.Now().Add(time.Minute))
r.deliverRunNotifications(time.Now().Add(2 * time.Minute))
r.deliverRunNotifications(time.Now().Add(3 * time.Minute))
if calls.Load() != 12 {
t.Fatalf("retry bound: %d calls, want 12", calls.Load())
if calls.Load() != 9 {
t.Fatalf("retry bound: %d calls, want 9", calls.Load())
}
for _, id := range []int64{1, 2, 3} {
got, err := s.RunNotifications(ctx, id)