Files
pipelines/internal/author/skill/references/work.md
T

272 lines
12 KiB
Markdown

# Project work ledger and sessions
Work in a project is not a chat. It is a durable ledger: what is queued, what is
running, what has landed, and what is blocked. A session — this conversation — starts,
pauses, resumes and reports that ledger. Nothing important lives only in the
conversation: running the skill again, in any project on this machine, reads the
ledger and continues. This is the multi-project lifecycle on top of
`references/orchestration.md` (work items, briefs, pipeline choice) and
`references/lanes.md` (parallel workspaces).
## Where the ledger lives
Keep state **outside repositories**, under one work root, so it never collides with a
lane's tree or a commit:
${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work/
registry.json every project this machine knows
projects/<slug>/
state.json project, root, endpoint, pipeline, workspace
notes.md recon, decisions, open questions, blockers
queue/<NNNN-slug>.brief pending briefs, read in filename order
active/<id>.brief, <id>.claim claimed: the brief plus its run id and key
done/<id> landed: run id, status, finished
failed/<id> and <id>.brief failed twice: run id, status, diagnostic
log/driver.log
drivers/workd.sh this project's queue driver
PAUSED marker: stop dispatching new items
- `<slug>` is `<basename>-<8 hex of sha256(absolute project root)>`. **Resolve a
worktree to its main repository** so a lane shares the project's one ledger.
- `PIPELINES_WORK_DIR` overrides the work root. `registry.json` maps projects to
roots; add a project the first time it is worked on and keep it current.
- `state.json` holds `project`, `root`, `endpoint` (the CLI's `--url`), `pipeline`
(the default pipeline) and `workspace` (the default run directory).
- The ledger is the operator's record: keep `notes.md` truthful about decisions taken
and left open, never narrate it as a diary.
## Operations
Map one operator sentence to one operation. Do the operation, then answer with the
status block — nothing else.
| Operator says | Do |
| --- | --- |
| start / work on X | Reconnoitre, register the project, write small briefs into `queue/`, then run the driver. |
| status / how is it going | Print the status block. |
| queue "…" / add … | Reconnoitre, split into small items, append briefs; the driver picks them up. |
| pause / stop for now | Write `PAUSED`. The item in flight finishes; no new one starts. |
| resume / continue | Reconcile, remove `PAUSED`, run the driver. |
| drop item N | Move its brief to `failed/` with a reason. Never silently delete work. |
| done / stop everything | Pause and report what landed and what remains. |
**Reconcile before acting, always.** The server is authoritative for run state; the
ledger is authoritative for intent. On every invocation: read `registry.json` and the
project's ledger; for each `active/` claim, `CLI_PREFIX inspect <run id> --json` and
settle it (running → keep waiting; succeeded → `done/`; failed/interrupted →
`failed/`); only then do the requested operation. Never trust a claim file for a
status, and never start a second run for an item whose request key is already
admitted — that key is what makes a redispatch attach instead of duplicate.
Status block (these lines, no prose around them):
pipelines — active
queue: 4 pending · 1 running (run 185 · reviewer · 6m) · 12 done · 2 failed
next: 0005-graph-arrows on "Focused tweak - implement review commit"
blocked: none
## The brief, and per-item directives
Write every brief exactly as `references/orchestration.md` describes. A brief file in
`queue/` may open with up to three directive lines, stripped before the prompt is sent:
# pipeline: Focused tweak - implement review commit
# workspace: /path/to/lane-app2
# device: emulator-5556
Goal: …
`pipeline` overrides the default for that item; `workspace` routes an item to a lane's
checkout (a driver only claims items for its own workspace); `device` is for the
brief's own text. Number briefs `0001-…`, `0002-…` so order is explicit.
## The driver
A queue is advanced by a small driver, so work continues while the operator sleeps.
Write the driver once per project to `drivers/workd.sh` and launch it; it is
restart-safe (it resumes from `done/`, reattaches to `active/` runs by id, and never
starts a duplicate), and it stops on the first item that fails twice. Run **one
driver per workspace**; several drivers sharing one queue advance different lanes in
parallel because an item is claimed by an atomic rename.
```sh
#!/bin/sh
# workd.sh - advance one project's Pipelines work ledger.
#
# sh workd.sh [PROJECT_ROOT] [WORKSPACE]
#
# PROJECT_ROOT identifies the ledger (default: the current directory).
# WORKSPACE is the checkout this driver may run in (default: the ledger's
# workspace). One driver per workspace lets several lanes advance one shared
# queue: an item is claimed by an atomic rename, so it runs exactly once.
#
# Restart-safe. It resumes from done/, reattaches to active/ runs by id and
# never starts a duplicate. PIPELINES_WORK_POLL sets the poll gap in seconds
# (default 20); PIPELINES_CLI uses another client binary.
set -u
CLI=${PIPELINES_CLI:-pipelines}
POLL=${PIPELINES_WORK_POLL:-20}
ROOT=$(cd "${1:-$(pwd)}" && pwd)
WORK_ROOT=${PIPELINES_WORK_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work}
SLUG=$(basename "$ROOT")-$(printf '%s' "$ROOT" | shasum -a 256 | cut -c1-8)
D=$WORK_ROOT/projects/$SLUG
LOG=$D/log/driver.log
mkdir -p "$D/queue" "$D/active" "$D/done" "$D/failed" "$D/log" 2>/dev/null
say() { printf '%s %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" | tee -a "$LOG"; }
jfield() { python3 -c 'import json,sys
try: d=json.load(open(sys.argv[1]))
except Exception: d={}
print(d.get(sys.argv[2]) or "")' "$1" "$2"; }
dstatus() { "$CLI" inspect "$1" --json 2>/dev/null | python3 -c 'import json,sys
try: d=json.load(sys.stdin)
except Exception: print("unknown"); raise SystemExit
print(d.get("status") or "unknown")'; }
item_field() { sed -n "s/^# *$2 *: *//p" "$1" 2>/dev/null | head -1; } # from a brief
claim_field() { sed -n "s/^$2=//p" "$1" 2>/dev/null | head -1; } # from a claim file
prompt_of() { python3 -c '
import re,sys
lines=open(sys.argv[1],encoding="utf-8",errors="replace").read().splitlines()
while lines and re.match(r"^#\s*(pipeline|workspace|device)\s*:", lines[0]):
lines.pop(0)
while lines and lines[0].strip()=="":
lines.pop(0)
print("\n".join(lines))' "$1"; }
[ -f "$D/state.json" ] || { echo "no ledger: $D/state.json" >&2; exit 2; }
PIPELINE=$(jfield "$D/state.json" pipeline)
ENDPOINT=$(jfield "$D/state.json" endpoint)
WS=${2:-$(jfield "$D/state.json" workspace)}
[ -n "$WS" ] || WS=$ROOT
[ -n "$PIPELINE" ] || { echo "state.json has no pipeline" >&2; exit 2; }
wait_terminal() { # $1 run id -> terminal status (or "unknown" if unreadable)
bad=0
while :; do
st=$(dstatus "$1")
case "$st" in
succeeded|failed|canceled|interrupted) echo "$st"; return 0;;
unknown) bad=$((bad+1)); [ "$bad" -ge 5 ] && { echo "unknown"; return 0; };;
*) bad=0;;
esac
sleep "$POLL"
done
}
dispatch() { # $1 brief file $2 pipeline $3 key -> run id (empty on failure)
if [ -n "$ENDPOINT" ]; then
out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" --url "$ENDPOINT" 2>&1)
else
out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" 2>&1)
fi
printf '%s' "$out" | python3 -c 'import json,sys
try: d=json.load(sys.stdin)
except Exception: print("")
else: print((d.get("run_detail") or {}).get("id") or "")'
}
record() { # $1 id $2 status $3 run id
where=done; [ "$2" = succeeded ] || where=failed
printf 'run_id=%s\nstatus=%s\nfinished=%s\n' "$3" "$2" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" > "$D/$where/$1"
rm -f "$D/active/$1.brief" "$D/active/$1.claim"
}
run_item() { # $1 id $2 brief $3 pipeline $4 key -> 0 succeeded / 1 failed
rid=$(dispatch "$2" "$3" "$4")
if [ -z "$rid" ]; then printf 'status=dispatch-failed\n' > "$D/active/$1.claim"; say "dispatch failed: $1"; return 1; fi
printf 'run_id=%s\n' "$rid" > "$D/active/$1.claim"
st=$(wait_terminal "$rid")
if [ "$st" = succeeded ]; then record "$1" succeeded "$rid"; say "done $1 (run $rid)"; return 0; fi
printf 'run_id=%s\nstatus=%s\n' "$rid" "$st" > "$D/active/$1.claim"
say "failed $1 (run $rid, $st)"
return 1
}
reconcile() { # settle anything the previous process left claimed
for a in "$D"/active/*; do
[ -e "$a" ] || continue
id=$(basename "$a"); id=${id%.brief}; id=${id%.claim}
rid=$(claim_field "$D/active/$id.claim" run_id)
if [ -z "$rid" ]; then
say "requeueing undispatched $id"
[ -e "$D/active/$id.brief" ] && mv "$D/active/$id.brief" "$D/queue/$id.brief"
rm -f "$D/active/$id.claim"
continue
fi
st=$(wait_terminal "$rid")
record "$id" "$st" "$rid"
say "reconciled $id as $st (run $rid)"
done
}
reconcile
while :; do
if [ -e "$D/PAUSED" ]; then sleep "$POLL"; continue; fi
next=""
for q in $(ls -1 "$D"/queue/*.brief 2>/dev/null | sort); do
id=$(basename "$q" .brief)
[ -e "$D/done/$id" ] && continue
[ -e "$D/failed/$id" ] && continue
iws=$(item_field "$q" workspace)
[ -n "$iws" ] && [ "$iws" != "$WS" ] && continue
next=$q; break
done
if [ -z "$next" ]; then say "idle (no claimable items for $WS)"; exit 0; fi
id=$(basename "$next" .brief)
mv "$next" "$D/active/$id.brief" 2>/dev/null || continue
ipipe=$(item_field "$D/active/$id.brief" pipeline)
[ -n "$ipipe" ] || ipipe=$PIPELINE
key="work-$SLUG-$id"
if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key"; then continue; fi
if [ "$(claim_field "$D/active/$id.claim" status)" = "dispatch-failed" ]; then
say "STOP: could not dispatch $id"
mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
exit 1
fi
say "retrying $id once"
if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key-r1"; then continue; fi
rid=$(claim_field "$D/active/$id.claim" run_id)
st=$(claim_field "$D/active/$id.claim" status)
printf 'run_id=%s\nstatus=%s\nattempts=2\n' "$rid" "${st:-failed}" > "$D/failed/$id"
mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
rm -f "$D/active/$id.claim"
say "STOP: $id failed twice; recorded under failed/"
exit 1
done
```
Launch it detached and keep its log:
nohup sh <work root>/projects/<slug>/drivers/workd.sh <project root> >/dev/null 2>&1 &
The driver only *advances* a prepared queue; it never writes briefs. Preparing a good
queue and settling a failure is the agent's work.
## Sleeping and waking
- Before the operator sleeps: reconcile, make sure the queue is full enough to cover
the night, and confirm the driver is running. A drained queue is fine; it idles.
- Configure a **failure notification** (`references/notifications.md`) so a stopped
driver is not silent. On an overnight failure the driver stops after one retry; that
is by design — do not add retry loops to make it look busy.
- On waking, reconcile first, then answer with the status block: what landed, what
failed and why, what is next.
## Conciseness
The operator wants the result, not the shift report.
- Match the answer's length to the question. A yes/no or one-line question gets one or
two sentences; never answer a narrow question with a sectioned report.
- Answer the question that was asked first. Put any extra fact that changes a decision in
a single clause, not in a new section.
- Answer with the status block for status; do not narrate polling or re-print the
queue when nothing changed.
- Report a run by its outcome and evidence (`references/verification.md`), not the
steps you took to watch it.
- Ask only for a decision the operator owns (accept, publish, restart, install). Batch
the day's questions rather than asking per item.
- Never claim a check, build or screenshot you did not run.