Merge docs-event-driven: rewrite the workflow docs for the agent-driven model (no driver, no poll loop, no lanes)
This commit is contained in:
@@ -31,10 +31,14 @@ the single `docs-fold` writer rather than editing them per item: leave a
|
||||
|
||||
## Maintenance workflow
|
||||
|
||||
Top-level agents use the configured `Pipelines maintenance - plan implement review`
|
||||
maintenance workflow for code changes. Inspect the retained plan/implement/review
|
||||
outputs and diffs, execute the checks independently, and iterate on concrete
|
||||
failures rather than trusting stage summaries. An executing stage performs its own
|
||||
role directly and never recursively launches a workflow. If the workflow is
|
||||
unavailable, ask instead of silently bypassing it. Never hardcode a disposable
|
||||
server URL as a project default; the configured endpoint is authoritative.
|
||||
Top-level agents advance work through the `pipelines` CLI; **the agent is the brain**
|
||||
and there is no orchestration script, driver, polling loop or lane machinery, and none
|
||||
may be written. Choose the pipeline by the area the work is in — `Android`, `TUI`,
|
||||
`Server`, `Generic`, `Plan and land`, `UI sweep`, `Probe` — each reading the project's
|
||||
own map (`android/AGENTS.md`, `internal/tui/AGENTS.md`, `internal/server/AGENTS.md`)
|
||||
for how that technology is built, tested, run and captured. Inspect the retained
|
||||
plan/implement/review outputs and diffs, execute the checks independently, and iterate
|
||||
on concrete failures rather than trusting stage summaries. An executing stage performs
|
||||
its own role directly and never recursively launches a workflow. If a pipeline is
|
||||
unavailable, ask instead of silently bypassing it. Never hardcode a disposable server
|
||||
URL as a project default; the configured endpoint is authoritative.
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# Work documentation now describes the agent-driven model
|
||||
|
||||
## What this item changed
|
||||
|
||||
The workflow documentation was rewritten to describe how the system actually works:
|
||||
the agent (the session) is the brain and advances the queue by hand through the
|
||||
`pipelines` CLI. There is **no orchestration script, no driver, no polling loop and no
|
||||
lane machinery, and none may be written.**
|
||||
|
||||
- `internal/author/skill/references/work.md` — rewritten end to end. The queue-driver
|
||||
script (`drivers/workd.sh`), the claim-file ledger with atomic renames, the `PAUSED`
|
||||
marker and the poll loop are gone. The ledger is now the per-project markdown
|
||||
`queue.md` (authoritative for intent) plus `notes.md`; run state is read from the
|
||||
server (`pipelines inspect <run-id> --json`); the agent launches each run itself with
|
||||
`pipelines run <pipeline> <brief> --json --yes` from the item's worktree and is woken
|
||||
by a background `pipelines wait`.
|
||||
- `internal/author/skill/references/queue.md` — the landing gate now runs the command
|
||||
directly (`git merge-base --is-ancestor <branch> main`, exit 0 = landed) instead of a
|
||||
`land-check.sh` helper; the waiting section states the agent advances the queue by
|
||||
hand and there is no driver.
|
||||
- `internal/author/skill/references/orchestration.md` — the intro and "Running and
|
||||
watching" state the agent-driven, event-driven model; "Pipeline choice" now names the
|
||||
current set (`Android`, `TUI`, `Server`, `Generic`, `Plan and land`, `UI sweep`,
|
||||
`Probe`), chosen by area, and records the retired names.
|
||||
- `internal/author/skill/references/debugging.md` — "Duplicated work after a driver
|
||||
restart" became "…after a session restart", resuming from the queue file rather than a
|
||||
cursor file.
|
||||
- `internal/author/skill/SKILL.md` — the routing table row no longer says "queue driver".
|
||||
- `AGENTS.md` — the maintenance workflow section names the current pipeline set instead
|
||||
of the retired `Pipelines maintenance - plan implement review`.
|
||||
- `internal/tui/AGENTS.md` — the "do not restart … the lanes …" line no longer names
|
||||
lanes.
|
||||
|
||||
`delivery.md`, `operations.md` and `verification.md` were swept and needed no change:
|
||||
they contain no driver, poll loop, cursor file or lane guidance. Genuine device tooling
|
||||
(`android/scripts/capture_png.sh`) is left alone.
|
||||
|
||||
## Shared documents this change makes stale
|
||||
|
||||
No README or topic doc still teaches the scripted driver model. One topic doc needs a
|
||||
wording alignment for the event-driven model; the `docs-fold` writer should apply it
|
||||
when it next folds notes:
|
||||
|
||||
- `docs/cli.md` — its "Scripted runs" section frames runs as scripted automation, and
|
||||
its `--wait` paragraph is the only follow-a-run guidance. It should state that the
|
||||
agent (not a script) advances work, and that `pipelines wait <run-id> [run-id ...]`
|
||||
run as a **background task** is the event-driven wait. The `wait` command's own flag
|
||||
text is already recorded separately in `docs/notes/wait-command.md`.
|
||||
|
||||
## Do not change
|
||||
|
||||
- The dated evidence documents (`docs/pipeline-performance-review.md`,
|
||||
`docs/chat-verification.md`, `docs/app-integration-verification.md`) describe the
|
||||
scripted era as it was and stay as they are.
|
||||
- `internal/author/install.go`'s managed file inventory needs no change: no packaged
|
||||
reference file was added or removed.
|
||||
@@ -59,7 +59,7 @@ and `references/format.md` (in parallel when possible); for notifications also r
|
||||
| Task | Packaged reference |
|
||||
| --- | --- |
|
||||
| Briefs, work items, pipeline choice, running and watching | `references/orchestration.md` |
|
||||
| Across projects: work ledger, start/pause/resume/status, queue driver | `references/work.md` |
|
||||
| Across projects: work ledger, start/pause/resume/status, the queue | `references/work.md` |
|
||||
| The work queue, parallel items and landing | `references/queue.md` |
|
||||
| Diagnose a wedged, blocked or failed run | `references/debugging.md` |
|
||||
| Publish artefacts, LAN URLs, install boundaries | `references/delivery.md` |
|
||||
|
||||
@@ -130,13 +130,16 @@ Each playbook is *symptom → how to confirm → what to do*.
|
||||
approval. Put the capture helper in the repository so every UI run reuses one path.
|
||||
See the example in `references/delivery.md`.
|
||||
|
||||
## Duplicated work after a driver restart
|
||||
## Duplicated work after a session restart
|
||||
|
||||
- **Symptom** Items already completed run again after a queue driver
|
||||
restarts: the same brief is re-invoked, side effects repeat, or a commit is made
|
||||
- **Symptom** Items already completed run again after the session that was advancing the
|
||||
queue restarts: the same brief is re-invoked, side effects repeat, or a commit is made
|
||||
twice.
|
||||
- **Confirm** Compare the queue's items against their recorded state. A list alone
|
||||
has no resume point.
|
||||
- **Do** Record each item's state in the queue itself (`references/queue.md`) and resume from it. Record an item
|
||||
only once it has landed, and skip recorded items on startup. Never relaunch a
|
||||
duplicate of a run that is still active — reattach with its request key instead.
|
||||
- **Confirm** Compare the queue's items against their recorded state and the server's run
|
||||
history (`inspect RUN_ID --json`). A list alone has no resume point.
|
||||
- **Do** Record each item's state in the queue itself (`references/queue.md`) and resume
|
||||
from it: read the queue top to bottom, skip items already marked done, and settle the
|
||||
ones marked running by inspecting their run id. Record an item only once it has landed.
|
||||
Never relaunch a duplicate of a run that is still active — reattach with its request key
|
||||
instead. Do not build a driver or cursor file to remember position; the queue file is
|
||||
the resume point.
|
||||
|
||||
@@ -5,6 +5,11 @@ briefs and a set of managed runs, not the change itself. The implementer stage m
|
||||
the change; the reviewer stages check it; the committer stage commits it. Route all
|
||||
project work through Pipelines runs.
|
||||
|
||||
The agent advances this **by hand through the `pipelines` CLI**: it launches each run,
|
||||
is woken when it finishes, reconciles against the server, and starts the next item.
|
||||
There is no orchestration script, driver, polling loop or lane machinery, and none may
|
||||
be written.
|
||||
|
||||
## Stance
|
||||
|
||||
- **Reconnoitre first.** Read enough of the repository, its notes and recent history
|
||||
@@ -141,6 +146,24 @@ pipelines run "Docs fold"
|
||||
|
||||
## Pipeline choice
|
||||
|
||||
Choose by the **area the work is in**; each area pipeline reads the project's own map
|
||||
for its build, test, run and capture commands:
|
||||
|
||||
- **`Android`** — `android/`.
|
||||
- **`TUI`** — `internal/tui/`.
|
||||
- **`Server`** — `internal/server/`, `internal/stage/`, `internal/pipeline/` and
|
||||
`internal/protocol/`.
|
||||
- **`Generic`** — only when a change genuinely spans areas.
|
||||
- **`Plan and land`** — when the change needs a plan before implementation
|
||||
(contract-level work).
|
||||
- **`UI sweep`** — a full-screen review pass.
|
||||
- **`Probe`** — a harness smoke check.
|
||||
|
||||
The retired names (`Pipelines quick`/`maintenance`/`critical`, `Focused tweak`, `Quick
|
||||
tweak`, `Plan, implement, review`, `Harness probe`) stay in the registry because
|
||||
definitions are immutable, but nothing should start them. If a pipeline is unavailable,
|
||||
ask instead of silently bypassing it.
|
||||
|
||||
- Use a pipeline that keeps **implementer + reviewer + ui_reviewer + committer** for
|
||||
anything that matters. The planner is the only stage a leaner variant may drop.
|
||||
- **Never remove a review stage to go faster.** The reviewer stages are the
|
||||
@@ -162,9 +185,10 @@ pipelines run "Docs fold"
|
||||
- A run cannot approve its own work. When an item genuinely needs an operator gate
|
||||
before it continues or commits, insert a `type=human` decision stage
|
||||
(`references/format.md`); never invent a model stage that pretends to approve.
|
||||
- Choose per item: a logic, server or format change usually wants the planner
|
||||
(a plan/implement/review pipeline); a small, fully understood tweak may use the
|
||||
leaner implementer-first variant. Both keep all review stages.
|
||||
- Pick the pipeline whose area matches the change (above); `Generic` only when a change
|
||||
genuinely spans areas. A logic, server or format change usually wants the planner
|
||||
(`Plan and land`); a small, fully understood tweak may use the leaner area pipeline.
|
||||
Both keep all review stages.
|
||||
|
||||
## Running and watching
|
||||
|
||||
@@ -179,9 +203,13 @@ pipelines run "Docs fold"
|
||||
and are recorded on the run; `retry` reuses them. See `references/harnesses.md`
|
||||
for how a stage's model and its repair path both use the effective model.
|
||||
- Watch to a **terminal state** (`succeeded`, `failed`, `canceled`, `interrupted`).
|
||||
Poll `CLI_PREFIX inspect RUN_ID --json` (or use `--wait`) until terminal. A reached
|
||||
client wait limit is not a run failure: the run stays active; keep waiting or
|
||||
reattach with the original request key, never start a duplicate run.
|
||||
Progress is **event-driven, never polled**: `CLI_PREFIX wait RUN_ID […]` blocks until
|
||||
the runs are terminal and is run as a **background task**, so the session is woken
|
||||
exactly when the batch finishes; `CLI_PREFIX inspect RUN_ID --json` then reads the
|
||||
retained outcome. A reached client wait limit is not a run failure: the run stays
|
||||
active; keep waiting or reattach with the original request key, never start a
|
||||
duplicate run. Do not build a polling loop, a cursor file or a driver script to
|
||||
answer "is it done yet".
|
||||
- Read the **outcomes**, not just the status: the option each stage returned, the
|
||||
evidence it recorded, and any run or step diagnostic. Interpret diagnostics by
|
||||
category (`references/debugging.md`).
|
||||
|
||||
@@ -49,8 +49,9 @@ Serialize only when:
|
||||
|
||||
## The landing gate, and what must never happen
|
||||
|
||||
- After every item, and **before any cleanup or before starting the next item**, run
|
||||
`land-check.sh <branch>`. It must report the branch as an **ancestor of main**.
|
||||
- After every item, and **before any cleanup or before starting the next item**, verify
|
||||
the branch is an **ancestor of main** directly, with no helper script:
|
||||
`git merge-base --is-ancestor <branch> main` (exit 0 = landed).
|
||||
- If it refuses: do not remove the worktree, do not touch the branch, do not start the next item.
|
||||
Report the landing failure and the commits that are not in main.
|
||||
- **Never delete a branch.** Deleting a branch whose commits are not yet in main leaves them
|
||||
@@ -60,6 +61,9 @@ Serialize only when:
|
||||
|
||||
## Waiting without a scripting harness
|
||||
|
||||
- The agent advances the queue **by hand**: read it, create the item's worktree, and launch
|
||||
`pipelines run <pipeline> <brief> --json --yes` from that worktree. Nothing else advances it —
|
||||
there is no driver, and none may be written.
|
||||
- `pipelines wait <run-id> […]` blocks until the runs finish and exits; run it as a **background
|
||||
task** so the session is woken exactly when the batch is done. A run that failed is still a
|
||||
finished run: it exits 0 and says what happened.
|
||||
|
||||
@@ -1,256 +1,136 @@
|
||||
# Project work ledger and sessions
|
||||
|
||||
Work in a project is not a chat. It is a durable ledger: what is queued, what is
|
||||
running, what has landed, and what is blocked. A session — this conversation — starts,
|
||||
pauses, resumes and reports that ledger. Nothing important lives only in the
|
||||
conversation: running the skill again, in any project on this machine, reads the
|
||||
ledger and continues. This is the multi-project lifecycle on top of
|
||||
running, what has landed, and what is blocked. A session — this conversation — reads
|
||||
the ledger, launches the runs, is woken when they finish, and reports.
|
||||
|
||||
**The agent is the brain.** It advances the queue by hand through the `pipelines`
|
||||
CLI: it reads the queue, picks the item, launches the run(s) itself, is woken when they
|
||||
finish, reconciles against the server, and starts the next item. There is **no
|
||||
orchestration script, no driver, no polling loop and no lane machinery, and none may be
|
||||
written.** Where a check is needed, run the command directly.
|
||||
|
||||
State lives in exactly three places:
|
||||
|
||||
- the **server** is authoritative for run state — `pipelines inspect <run-id> --json`;
|
||||
- the **queue file** is authoritative for intent — what is queued, in what order, and
|
||||
what is finished;
|
||||
- the **conversation** holds this session's working memory.
|
||||
|
||||
Never keep progress in a script's own files. A session that dies and comes back reads
|
||||
the queue file and continues. This is the multi-project lifecycle on top of
|
||||
`references/orchestration.md` (work items, briefs, pipeline choice) and
|
||||
`references/queue.md` (parallel items).
|
||||
`references/queue.md` (parallel items, one worktree per item, landing).
|
||||
|
||||
## Where the ledger lives
|
||||
|
||||
Keep state **outside repositories**, under one work root, so it never collides with a
|
||||
Keep state **outside repositories**, under one work root, so it never collides with an
|
||||
item's worktree or a commit:
|
||||
|
||||
${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work/
|
||||
registry.json every project this machine knows
|
||||
registry.json every project this machine knows
|
||||
projects/<slug>/
|
||||
state.json project, root, endpoint, pipeline, workspace
|
||||
notes.md recon, decisions, open questions, blockers
|
||||
queue/<NNNN-slug>.brief pending briefs, read in filename order
|
||||
active/<id>.brief, <id>.claim claimed: the brief plus its run id and key
|
||||
done/<id> landed: run id, status, finished
|
||||
failed/<id> and <id>.brief failed twice: run id, status, diagnostic
|
||||
log/driver.log
|
||||
drivers/workd.sh this project's queue driver
|
||||
PAUSED marker: stop dispatching new items
|
||||
queue.md the project's markdown queue: items, briefs, state
|
||||
notes.md recon, decisions, open questions, blockers
|
||||
|
||||
- `<slug>` is `<basename>-<8 hex of sha256(absolute project root)>`. **Resolve a
|
||||
worktree to its main repository** so every item shares the project's one ledger.
|
||||
worktree to its main repository** so every item shares the project's one queue.
|
||||
- `PIPELINES_WORK_DIR` overrides the work root. `registry.json` maps projects to
|
||||
roots; add a project the first time it is worked on and keep it current.
|
||||
- `state.json` holds `project`, `root`, `endpoint` (the CLI's `--url`), `pipeline`
|
||||
(the default pipeline) and `workspace` (the default run directory).
|
||||
- The ledger is the operator's record: keep `notes.md` truthful about decisions taken
|
||||
and left open, never narrate it as a diary.
|
||||
- `queue.md` is the resume point (`references/queue.md`): read it top to bottom and
|
||||
take the first unfinished item. `notes.md` is the operator's record: keep it truthful
|
||||
about decisions taken and left open, never narrate it as a diary.
|
||||
- The CLI's configured endpoint and the area map's default pipeline come from the CLI
|
||||
and the project's own map, not from a script. Do not write a per-project state file
|
||||
for a script to read.
|
||||
|
||||
## Operations
|
||||
|
||||
Map one operator sentence to one operation. Do the operation, then answer with the
|
||||
status block — nothing else.
|
||||
status block — nothing else. **You launch the runs yourself**; nothing else does it.
|
||||
|
||||
| Operator says | Do |
|
||||
| --- | --- |
|
||||
| start / work on X | Reconnoitre, register the project, write small briefs into `queue/`, then run the driver. |
|
||||
| status / how is it going | Print the status block. |
|
||||
| queue "…" / add … | Reconnoitre, split into small items, append briefs; the driver picks them up. |
|
||||
| pause / stop for now | Write `PAUSED`. The item in flight finishes; no new one starts. |
|
||||
| resume / continue | Reconcile, remove `PAUSED`, run the driver. |
|
||||
| drop item N | Move its brief to `failed/` with a reason. Never silently delete work. |
|
||||
| done / stop everything | Pause and report what landed and what remains. |
|
||||
| start / work on X | Reconnoitre, register the project, write the queue file, then launch the first item's run yourself. |
|
||||
| status / how is it going | Read the server and the queue, then print the status block. |
|
||||
| queue "…" / add … | Reconnoitre, split into small items, append them to the queue file, launch what is ready. |
|
||||
| pause / stop for now | Launch nothing new. The runs already in flight finish; reconcile them when they do. |
|
||||
| resume / continue | Reconcile against the server, then launch the next item. |
|
||||
| drop item N | Mark its item failed in the queue file with a reason. Never silently delete work. |
|
||||
| done / stop everything | Stop launching, wait for what is in flight, and report what landed and what remains. |
|
||||
|
||||
**Reconcile before acting, always.** The server is authoritative for run state; the
|
||||
ledger is authoritative for intent. On every invocation: read `registry.json` and the
|
||||
project's ledger; for each `active/` claim, `CLI_PREFIX inspect <run id> --json` and
|
||||
settle it (running → keep waiting; succeeded → `done/`; failed/interrupted →
|
||||
`failed/`); only then do the requested operation. Never trust a claim file for a
|
||||
status, and never start a second run for an item whose request key is already
|
||||
admitted — that key is what makes a redispatch attach instead of duplicate.
|
||||
queue file is authoritative for intent. On every invocation: read the project's
|
||||
`queue.md`; for each item marked running, `pipelines inspect <run-id> --json` and settle
|
||||
it (running → keep waiting; succeeded → verify it landed and mark it done;
|
||||
failed/interrupted → mark it failed with the diagnostic). Only then do the requested
|
||||
operation. Never trust your memory of a run's status, and never start a second run for
|
||||
an item whose request key is already admitted — reuse the original `--request-key` so a
|
||||
redispatch attaches instead of duplicating.
|
||||
|
||||
Status block (these lines, no prose around them):
|
||||
|
||||
pipelines — active
|
||||
queue: 4 pending · 1 running (run 185 · reviewer · 6m) · 12 done · 2 failed
|
||||
next: 0005-graph-arrows on "Focused tweak - implement review commit"
|
||||
next: 0005-graph-arrows on "TUI"
|
||||
blocked: none
|
||||
|
||||
## The brief, and per-item directives
|
||||
|
||||
Write every brief exactly as `references/orchestration.md` describes. A brief file in
|
||||
`queue/` may open with up to three directive lines, stripped before the prompt is sent:
|
||||
Write every brief exactly as `references/orchestration.md` describes. An item in
|
||||
`queue.md` may open its brief with up to three directive lines, stripped before the
|
||||
prompt is sent:
|
||||
|
||||
# pipeline: Focused tweak - implement review commit
|
||||
# pipeline: TUI
|
||||
# workspace: /path/to/item-worktree
|
||||
# device: emulator-5556
|
||||
|
||||
Goal: …
|
||||
|
||||
`pipeline` overrides the default for that item; `workspace` routes an item to its own
|
||||
checkout (a driver only claims items for its own workspace); `device` is for the
|
||||
brief's own text. Number briefs `0001-…`, `0002-…` so order is explicit.
|
||||
`pipeline` overrides the default for that item; `workspace` names the item's own
|
||||
worktree (create it with `git worktree add -b <item> <path> main`, `references/queue.md`);
|
||||
`device` is for the brief's own text. Number items `0001-…`, `0002-…` so order is
|
||||
explicit.
|
||||
|
||||
## The driver
|
||||
## Advancing the queue by hand
|
||||
|
||||
A queue is advanced by a small driver, so work continues while the operator sleeps.
|
||||
Write the driver once per project to `drivers/workd.sh` and launch it; it is
|
||||
restart-safe (it resumes from `done/`, reattaches to `active/` runs by id, and never
|
||||
starts a duplicate), and it stops on the first item that fails twice. Run **one
|
||||
item at a time per worktree**; several items in their own worktrees advance in
|
||||
parallel because an item is claimed by an atomic rename.
|
||||
There is no driver. You advance the queue yourself, through the CLI:
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
# workd.sh - advance one project's Pipelines work ledger.
|
||||
#
|
||||
# sh workd.sh [PROJECT_ROOT] [WORKSPACE]
|
||||
#
|
||||
# PROJECT_ROOT identifies the ledger (default: the current directory).
|
||||
# WORKSPACE is the checkout this driver may run in (default: the ledger's
|
||||
# workspace). One item per worktree lets several items advance against one shared
|
||||
# queue: an item is claimed by an atomic rename, so it runs exactly once.
|
||||
#
|
||||
# Restart-safe. It resumes from done/, reattaches to active/ runs by id and
|
||||
# never starts a duplicate. PIPELINES_WORK_POLL sets the poll gap in seconds
|
||||
# (default 20); PIPELINES_CLI uses another client binary.
|
||||
set -u
|
||||
1. **Read the queue top to bottom** and take the first unfinished item.
|
||||
2. **Choose the pipeline by area** (`references/orchestration.md`): `Android`, `TUI`,
|
||||
`Server`, `Generic`, `Plan and land`, `UI sweep`, `Probe`. Each area pipeline reads
|
||||
the project's own map for its build, test, run and capture commands.
|
||||
3. **Create the item's worktree and branch**: `git worktree add -b <item> <path> main`.
|
||||
One worktree per item is what lets independent items run at once without the
|
||||
workspace guard becoming a bottleneck (`references/queue.md`).
|
||||
4. **Launch the run from that worktree**:
|
||||
`pipelines run <pipeline> <brief> --json --yes`. Keep the returned run id.
|
||||
5. **Be woken when it finishes — do not poll.** `pipelines wait <run-id> […]` blocks
|
||||
until the runs are terminal and is run as a **background task**, so the session is
|
||||
woken exactly when the batch finishes — or arm a timer and check on the tick. No
|
||||
polling loop, no cursor file, no driver script, ever.
|
||||
6. **Reconcile.** `pipelines inspect <run-id> --json`; read the outcomes and evidence;
|
||||
verify the branch landed with `git merge-base --is-ancestor <branch> main` (exit 0 =
|
||||
landed); mark the item in `queue.md`; then start the next item — or several
|
||||
independent items at once.
|
||||
|
||||
CLI=${PIPELINES_CLI:-pipelines}
|
||||
POLL=${PIPELINES_WORK_POLL:-20}
|
||||
ROOT=$(cd "${1:-$(pwd)}" && pwd)
|
||||
WORK_ROOT=${PIPELINES_WORK_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work}
|
||||
SLUG=$(basename "$ROOT")-$(printf '%s' "$ROOT" | shasum -a 256 | cut -c1-8)
|
||||
D=$WORK_ROOT/projects/$SLUG
|
||||
LOG=$D/log/driver.log
|
||||
mkdir -p "$D/queue" "$D/active" "$D/done" "$D/failed" "$D/log" 2>/dev/null
|
||||
**Parallelism is your judgement, bounded by real conflicts only**: two items editing
|
||||
the same files at once, the one capture device, or the provider's spend cap. Launch as
|
||||
many independent runs as the machine can carry without overload; serialise only for a
|
||||
real conflict and say which limit is binding. Never present serialisation as the
|
||||
operator's decision.
|
||||
|
||||
say() { printf '%s %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" | tee -a "$LOG"; }
|
||||
jfield() { python3 -c 'import json,sys
|
||||
try: d=json.load(open(sys.argv[1]))
|
||||
except Exception: d={}
|
||||
print(d.get(sys.argv[2]) or "")' "$1" "$2"; }
|
||||
dstatus() { "$CLI" inspect "$1" --json 2>/dev/null | python3 -c 'import json,sys
|
||||
try: d=json.load(sys.stdin)
|
||||
except Exception: print("unknown"); raise SystemExit
|
||||
print(d.get("status") or "unknown")'; }
|
||||
item_field() { sed -n "s/^# *$2 *: *//p" "$1" 2>/dev/null | head -1; } # from a brief
|
||||
claim_field() { sed -n "s/^$2=//p" "$1" 2>/dev/null | head -1; } # from a claim file
|
||||
prompt_of() { python3 -c '
|
||||
import re,sys
|
||||
lines=open(sys.argv[1],encoding="utf-8",errors="replace").read().splitlines()
|
||||
while lines and re.match(r"^#\s*(pipeline|workspace|device)\s*:", lines[0]):
|
||||
lines.pop(0)
|
||||
while lines and lines[0].strip()=="":
|
||||
lines.pop(0)
|
||||
print("\n".join(lines))' "$1"; }
|
||||
|
||||
[ -f "$D/state.json" ] || { echo "no ledger: $D/state.json" >&2; exit 2; }
|
||||
PIPELINE=$(jfield "$D/state.json" pipeline)
|
||||
ENDPOINT=$(jfield "$D/state.json" endpoint)
|
||||
WS=${2:-$(jfield "$D/state.json" workspace)}
|
||||
[ -n "$WS" ] || WS=$ROOT
|
||||
[ -n "$PIPELINE" ] || { echo "state.json has no pipeline" >&2; exit 2; }
|
||||
|
||||
wait_terminal() { # $1 run id -> terminal status (or "unknown" if unreadable)
|
||||
bad=0
|
||||
while :; do
|
||||
st=$(dstatus "$1")
|
||||
case "$st" in
|
||||
succeeded|failed|canceled|interrupted) echo "$st"; return 0;;
|
||||
unknown) bad=$((bad+1)); [ "$bad" -ge 5 ] && { echo "unknown"; return 0; };;
|
||||
*) bad=0;;
|
||||
esac
|
||||
sleep "$POLL"
|
||||
done
|
||||
}
|
||||
|
||||
dispatch() { # $1 brief file $2 pipeline $3 key -> run id (empty on failure)
|
||||
if [ -n "$ENDPOINT" ]; then
|
||||
out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" --url "$ENDPOINT" 2>&1)
|
||||
else
|
||||
out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" 2>&1)
|
||||
fi
|
||||
printf '%s' "$out" | python3 -c 'import json,sys
|
||||
try: d=json.load(sys.stdin)
|
||||
except Exception: print("")
|
||||
else: print((d.get("run_detail") or {}).get("id") or "")'
|
||||
}
|
||||
|
||||
record() { # $1 id $2 status $3 run id
|
||||
where=done; [ "$2" = succeeded ] || where=failed
|
||||
printf 'run_id=%s\nstatus=%s\nfinished=%s\n' "$3" "$2" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" > "$D/$where/$1"
|
||||
rm -f "$D/active/$1.brief" "$D/active/$1.claim"
|
||||
}
|
||||
|
||||
run_item() { # $1 id $2 brief $3 pipeline $4 key -> 0 succeeded / 1 failed
|
||||
rid=$(dispatch "$2" "$3" "$4")
|
||||
if [ -z "$rid" ]; then printf 'status=dispatch-failed\n' > "$D/active/$1.claim"; say "dispatch failed: $1"; return 1; fi
|
||||
printf 'run_id=%s\n' "$rid" > "$D/active/$1.claim"
|
||||
st=$(wait_terminal "$rid")
|
||||
if [ "$st" = succeeded ]; then record "$1" succeeded "$rid"; say "done $1 (run $rid)"; return 0; fi
|
||||
printf 'run_id=%s\nstatus=%s\n' "$rid" "$st" > "$D/active/$1.claim"
|
||||
say "failed $1 (run $rid, $st)"
|
||||
return 1
|
||||
}
|
||||
|
||||
reconcile() { # settle anything the previous process left claimed
|
||||
for a in "$D"/active/*; do
|
||||
[ -e "$a" ] || continue
|
||||
id=$(basename "$a"); id=${id%.brief}; id=${id%.claim}
|
||||
rid=$(claim_field "$D/active/$id.claim" run_id)
|
||||
if [ -z "$rid" ]; then
|
||||
say "requeueing undispatched $id"
|
||||
[ -e "$D/active/$id.brief" ] && mv "$D/active/$id.brief" "$D/queue/$id.brief"
|
||||
rm -f "$D/active/$id.claim"
|
||||
continue
|
||||
fi
|
||||
st=$(wait_terminal "$rid")
|
||||
record "$id" "$st" "$rid"
|
||||
say "reconciled $id as $st (run $rid)"
|
||||
done
|
||||
}
|
||||
|
||||
reconcile
|
||||
|
||||
while :; do
|
||||
if [ -e "$D/PAUSED" ]; then sleep "$POLL"; continue; fi
|
||||
next=""
|
||||
for q in $(ls -1 "$D"/queue/*.brief 2>/dev/null | sort); do
|
||||
id=$(basename "$q" .brief)
|
||||
[ -e "$D/done/$id" ] && continue
|
||||
[ -e "$D/failed/$id" ] && continue
|
||||
iws=$(item_field "$q" workspace)
|
||||
[ -n "$iws" ] && [ "$iws" != "$WS" ] && continue
|
||||
next=$q; break
|
||||
done
|
||||
if [ -z "$next" ]; then say "idle (no claimable items for $WS)"; exit 0; fi
|
||||
id=$(basename "$next" .brief)
|
||||
mv "$next" "$D/active/$id.brief" 2>/dev/null || continue
|
||||
ipipe=$(item_field "$D/active/$id.brief" pipeline)
|
||||
[ -n "$ipipe" ] || ipipe=$PIPELINE
|
||||
key="work-$SLUG-$id"
|
||||
if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key"; then continue; fi
|
||||
if [ "$(claim_field "$D/active/$id.claim" status)" = "dispatch-failed" ]; then
|
||||
say "STOP: could not dispatch $id"
|
||||
mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
|
||||
exit 1
|
||||
fi
|
||||
say "retrying $id once"
|
||||
if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key-r1"; then continue; fi
|
||||
rid=$(claim_field "$D/active/$id.claim" run_id)
|
||||
st=$(claim_field "$D/active/$id.claim" status)
|
||||
printf 'run_id=%s\nstatus=%s\nattempts=2\n' "$rid" "${st:-failed}" > "$D/failed/$id"
|
||||
mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
|
||||
rm -f "$D/active/$id.claim"
|
||||
say "STOP: $id failed twice; recorded under failed/"
|
||||
exit 1
|
||||
done
|
||||
```
|
||||
|
||||
Launch it detached and keep its log:
|
||||
|
||||
nohup sh <work root>/projects/<slug>/drivers/workd.sh <project root> >/dev/null 2>&1 &
|
||||
|
||||
The driver only *advances* a prepared queue; it never writes briefs. Preparing a good
|
||||
queue and settling a failure is the agent's work.
|
||||
**Landing stays the pipeline's job**: the committer stage merges and lands its own
|
||||
work. There is no merger, no resolver and nothing to hand to a script. You verify the
|
||||
landing gate (`references/queue.md`); you do not perform the merge.
|
||||
|
||||
## Sleeping and waking
|
||||
|
||||
- Before the operator sleeps: reconcile, make sure the queue is full enough to cover
|
||||
the night, and confirm the driver is running. A drained queue is fine; it idles.
|
||||
- Configure a **failure notification** (`references/notifications.md`) so a stopped
|
||||
driver is not silent. On an overnight failure the driver stops after one retry; that
|
||||
is by design — do not add retry loops to make it look busy.
|
||||
- Before the operator sleeps: reconcile, make sure the queue has enough items to cover
|
||||
the night, and launch what is ready with a background `pipelines wait`.
|
||||
- Configure a **failure notification** (`references/notifications.md`) so a failed run
|
||||
is not silent. A run that fails is recorded and the next item starts; do not add
|
||||
retry loops to make it look busy.
|
||||
- On waking, reconcile first, then answer with the status block: what landed, what
|
||||
failed and why, what is next.
|
||||
|
||||
@@ -262,7 +142,7 @@ The operator wants the result, not the shift report.
|
||||
two sentences; never answer a narrow question with a sectioned report.
|
||||
- Answer the question that was asked first. Put any extra fact that changes a decision in
|
||||
a single clause, not in a new section.
|
||||
- Answer with the status block for status; do not narrate polling or re-print the
|
||||
- Answer with the status block for status; do not narrate waiting or re-print the
|
||||
queue when nothing changed.
|
||||
- Report a run by its outcome and evidence (`references/verification.md`), not the
|
||||
steps you took to watch it.
|
||||
|
||||
@@ -42,5 +42,5 @@ say why is a review finding.
|
||||
you checked.
|
||||
- Terminal width matters: verify at a narrow width as well as a wide one, and never assume colour
|
||||
support (`plain-nocolor-*` exists for a reason).
|
||||
- Do not restart the deployed server, the lanes or ntfy to test a TUI change: run the binary against a
|
||||
- Do not restart the deployed server or ntfy to test a TUI change: run the binary against a
|
||||
scratch server or use the fixtures.
|
||||
|
||||
Reference in New Issue
Block a user