272 lines
12 KiB
Markdown
272 lines
12 KiB
Markdown
# Project work ledger and sessions
|
|
|
|
Work in a project is not a chat. It is a durable ledger: what is queued, what is
|
|
running, what has landed, and what is blocked. A session — this conversation — starts,
|
|
pauses, resumes and reports that ledger. Nothing important lives only in the
|
|
conversation: running the skill again, in any project on this machine, reads the
|
|
ledger and continues. This is the multi-project lifecycle on top of
|
|
`references/orchestration.md` (work items, briefs, pipeline choice) and
|
|
`references/lanes.md` (parallel workspaces).
|
|
|
|
## Where the ledger lives
|
|
|
|
Keep state **outside repositories**, under one work root, so it never collides with a
|
|
lane's tree or a commit:
|
|
|
|
${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work/
|
|
registry.json every project this machine knows
|
|
projects/<slug>/
|
|
state.json project, root, endpoint, pipeline, workspace
|
|
notes.md recon, decisions, open questions, blockers
|
|
queue/<NNNN-slug>.brief pending briefs, read in filename order
|
|
active/<id>.brief, <id>.claim claimed: the brief plus its run id and key
|
|
done/<id> landed: run id, status, finished
|
|
failed/<id> and <id>.brief failed twice: run id, status, diagnostic
|
|
log/driver.log
|
|
drivers/workd.sh this project's queue driver
|
|
PAUSED marker: stop dispatching new items
|
|
|
|
- `<slug>` is `<basename>-<8 hex of sha256(absolute project root)>`. **Resolve a
|
|
worktree to its main repository** so a lane shares the project's one ledger.
|
|
- `PIPELINES_WORK_DIR` overrides the work root. `registry.json` maps projects to
|
|
roots; add a project the first time it is worked on and keep it current.
|
|
- `state.json` holds `project`, `root`, `endpoint` (the CLI's `--url`), `pipeline`
|
|
(the default pipeline) and `workspace` (the default run directory).
|
|
- The ledger is the operator's record: keep `notes.md` truthful about decisions taken
|
|
and left open, never narrate it as a diary.
|
|
|
|
## Operations
|
|
|
|
Map one operator sentence to one operation. Do the operation, then answer with the
|
|
status block — nothing else.
|
|
|
|
| Operator says | Do |
|
|
| --- | --- |
|
|
| start / work on X | Reconnoitre, register the project, write small briefs into `queue/`, then run the driver. |
|
|
| status / how is it going | Print the status block. |
|
|
| queue "…" / add … | Reconnoitre, split into small items, append briefs; the driver picks them up. |
|
|
| pause / stop for now | Write `PAUSED`. The item in flight finishes; no new one starts. |
|
|
| resume / continue | Reconcile, remove `PAUSED`, run the driver. |
|
|
| drop item N | Move its brief to `failed/` with a reason. Never silently delete work. |
|
|
| done / stop everything | Pause and report what landed and what remains. |
|
|
|
|
**Reconcile before acting, always.** The server is authoritative for run state; the
|
|
ledger is authoritative for intent. On every invocation: read `registry.json` and the
|
|
project's ledger; for each `active/` claim, `CLI_PREFIX inspect <run id> --json` and
|
|
settle it (running → keep waiting; succeeded → `done/`; failed/interrupted →
|
|
`failed/`); only then do the requested operation. Never trust a claim file for a
|
|
status, and never start a second run for an item whose request key is already
|
|
admitted — that key is what makes a redispatch attach instead of duplicate.
|
|
|
|
Status block (these lines, no prose around them):
|
|
|
|
pipelines — active
|
|
queue: 4 pending · 1 running (run 185 · reviewer · 6m) · 12 done · 2 failed
|
|
next: 0005-graph-arrows on "Focused tweak - implement review commit"
|
|
blocked: none
|
|
|
|
## The brief, and per-item directives
|
|
|
|
Write every brief exactly as `references/orchestration.md` describes. A brief file in
|
|
`queue/` may open with up to three directive lines, stripped before the prompt is sent:
|
|
|
|
# pipeline: Focused tweak - implement review commit
|
|
# workspace: /path/to/lane-app2
|
|
# device: emulator-5556
|
|
|
|
Goal: …
|
|
|
|
`pipeline` overrides the default for that item; `workspace` routes an item to a lane's
|
|
checkout (a driver only claims items for its own workspace); `device` is for the
|
|
brief's own text. Number briefs `0001-…`, `0002-…` so order is explicit.
|
|
|
|
## The driver
|
|
|
|
A queue is advanced by a small driver, so work continues while the operator sleeps.
|
|
Write the driver once per project to `drivers/workd.sh` and launch it; it is
|
|
restart-safe (it resumes from `done/`, reattaches to `active/` runs by id, and never
|
|
starts a duplicate), and it stops on the first item that fails twice. Run **one
|
|
driver per workspace**; several drivers sharing one queue advance different lanes in
|
|
parallel because an item is claimed by an atomic rename.
|
|
|
|
```sh
|
|
#!/bin/sh
|
|
# workd.sh - advance one project's Pipelines work ledger.
|
|
#
|
|
# sh workd.sh [PROJECT_ROOT] [WORKSPACE]
|
|
#
|
|
# PROJECT_ROOT identifies the ledger (default: the current directory).
|
|
# WORKSPACE is the checkout this driver may run in (default: the ledger's
|
|
# workspace). One driver per workspace lets several lanes advance one shared
|
|
# queue: an item is claimed by an atomic rename, so it runs exactly once.
|
|
#
|
|
# Restart-safe. It resumes from done/, reattaches to active/ runs by id and
|
|
# never starts a duplicate. PIPELINES_WORK_POLL sets the poll gap in seconds
|
|
# (default 20); PIPELINES_CLI uses another client binary.
|
|
set -u
|
|
|
|
CLI=${PIPELINES_CLI:-pipelines}
|
|
POLL=${PIPELINES_WORK_POLL:-20}
|
|
ROOT=$(cd "${1:-$(pwd)}" && pwd)
|
|
WORK_ROOT=${PIPELINES_WORK_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work}
|
|
SLUG=$(basename "$ROOT")-$(printf '%s' "$ROOT" | shasum -a 256 | cut -c1-8)
|
|
D=$WORK_ROOT/projects/$SLUG
|
|
LOG=$D/log/driver.log
|
|
mkdir -p "$D/queue" "$D/active" "$D/done" "$D/failed" "$D/log" 2>/dev/null
|
|
|
|
say() { printf '%s %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" | tee -a "$LOG"; }
|
|
jfield() { python3 -c 'import json,sys
|
|
try: d=json.load(open(sys.argv[1]))
|
|
except Exception: d={}
|
|
print(d.get(sys.argv[2]) or "")' "$1" "$2"; }
|
|
dstatus() { "$CLI" inspect "$1" --json 2>/dev/null | python3 -c 'import json,sys
|
|
try: d=json.load(sys.stdin)
|
|
except Exception: print("unknown"); raise SystemExit
|
|
print(d.get("status") or "unknown")'; }
|
|
item_field() { sed -n "s/^# *$2 *: *//p" "$1" 2>/dev/null | head -1; } # from a brief
|
|
claim_field() { sed -n "s/^$2=//p" "$1" 2>/dev/null | head -1; } # from a claim file
|
|
prompt_of() { python3 -c '
|
|
import re,sys
|
|
lines=open(sys.argv[1],encoding="utf-8",errors="replace").read().splitlines()
|
|
while lines and re.match(r"^#\s*(pipeline|workspace|device)\s*:", lines[0]):
|
|
lines.pop(0)
|
|
while lines and lines[0].strip()=="":
|
|
lines.pop(0)
|
|
print("\n".join(lines))' "$1"; }
|
|
|
|
[ -f "$D/state.json" ] || { echo "no ledger: $D/state.json" >&2; exit 2; }
|
|
PIPELINE=$(jfield "$D/state.json" pipeline)
|
|
ENDPOINT=$(jfield "$D/state.json" endpoint)
|
|
WS=${2:-$(jfield "$D/state.json" workspace)}
|
|
[ -n "$WS" ] || WS=$ROOT
|
|
[ -n "$PIPELINE" ] || { echo "state.json has no pipeline" >&2; exit 2; }
|
|
|
|
wait_terminal() { # $1 run id -> terminal status (or "unknown" if unreadable)
|
|
bad=0
|
|
while :; do
|
|
st=$(dstatus "$1")
|
|
case "$st" in
|
|
succeeded|failed|canceled|interrupted) echo "$st"; return 0;;
|
|
unknown) bad=$((bad+1)); [ "$bad" -ge 5 ] && { echo "unknown"; return 0; };;
|
|
*) bad=0;;
|
|
esac
|
|
sleep "$POLL"
|
|
done
|
|
}
|
|
|
|
dispatch() { # $1 brief file $2 pipeline $3 key -> run id (empty on failure)
|
|
if [ -n "$ENDPOINT" ]; then
|
|
out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" --url "$ENDPOINT" 2>&1)
|
|
else
|
|
out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" 2>&1)
|
|
fi
|
|
printf '%s' "$out" | python3 -c 'import json,sys
|
|
try: d=json.load(sys.stdin)
|
|
except Exception: print("")
|
|
else: print((d.get("run_detail") or {}).get("id") or "")'
|
|
}
|
|
|
|
record() { # $1 id $2 status $3 run id
|
|
where=done; [ "$2" = succeeded ] || where=failed
|
|
printf 'run_id=%s\nstatus=%s\nfinished=%s\n' "$3" "$2" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" > "$D/$where/$1"
|
|
rm -f "$D/active/$1.brief" "$D/active/$1.claim"
|
|
}
|
|
|
|
run_item() { # $1 id $2 brief $3 pipeline $4 key -> 0 succeeded / 1 failed
|
|
rid=$(dispatch "$2" "$3" "$4")
|
|
if [ -z "$rid" ]; then printf 'status=dispatch-failed\n' > "$D/active/$1.claim"; say "dispatch failed: $1"; return 1; fi
|
|
printf 'run_id=%s\n' "$rid" > "$D/active/$1.claim"
|
|
st=$(wait_terminal "$rid")
|
|
if [ "$st" = succeeded ]; then record "$1" succeeded "$rid"; say "done $1 (run $rid)"; return 0; fi
|
|
printf 'run_id=%s\nstatus=%s\n' "$rid" "$st" > "$D/active/$1.claim"
|
|
say "failed $1 (run $rid, $st)"
|
|
return 1
|
|
}
|
|
|
|
reconcile() { # settle anything the previous process left claimed
|
|
for a in "$D"/active/*; do
|
|
[ -e "$a" ] || continue
|
|
id=$(basename "$a"); id=${id%.brief}; id=${id%.claim}
|
|
rid=$(claim_field "$D/active/$id.claim" run_id)
|
|
if [ -z "$rid" ]; then
|
|
say "requeueing undispatched $id"
|
|
[ -e "$D/active/$id.brief" ] && mv "$D/active/$id.brief" "$D/queue/$id.brief"
|
|
rm -f "$D/active/$id.claim"
|
|
continue
|
|
fi
|
|
st=$(wait_terminal "$rid")
|
|
record "$id" "$st" "$rid"
|
|
say "reconciled $id as $st (run $rid)"
|
|
done
|
|
}
|
|
|
|
reconcile
|
|
|
|
while :; do
|
|
if [ -e "$D/PAUSED" ]; then sleep "$POLL"; continue; fi
|
|
next=""
|
|
for q in $(ls -1 "$D"/queue/*.brief 2>/dev/null | sort); do
|
|
id=$(basename "$q" .brief)
|
|
[ -e "$D/done/$id" ] && continue
|
|
[ -e "$D/failed/$id" ] && continue
|
|
iws=$(item_field "$q" workspace)
|
|
[ -n "$iws" ] && [ "$iws" != "$WS" ] && continue
|
|
next=$q; break
|
|
done
|
|
if [ -z "$next" ]; then say "idle (no claimable items for $WS)"; exit 0; fi
|
|
id=$(basename "$next" .brief)
|
|
mv "$next" "$D/active/$id.brief" 2>/dev/null || continue
|
|
ipipe=$(item_field "$D/active/$id.brief" pipeline)
|
|
[ -n "$ipipe" ] || ipipe=$PIPELINE
|
|
key="work-$SLUG-$id"
|
|
if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key"; then continue; fi
|
|
if [ "$(claim_field "$D/active/$id.claim" status)" = "dispatch-failed" ]; then
|
|
say "STOP: could not dispatch $id"
|
|
mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
|
|
exit 1
|
|
fi
|
|
say "retrying $id once"
|
|
if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key-r1"; then continue; fi
|
|
rid=$(claim_field "$D/active/$id.claim" run_id)
|
|
st=$(claim_field "$D/active/$id.claim" status)
|
|
printf 'run_id=%s\nstatus=%s\nattempts=2\n' "$rid" "${st:-failed}" > "$D/failed/$id"
|
|
mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
|
|
rm -f "$D/active/$id.claim"
|
|
say "STOP: $id failed twice; recorded under failed/"
|
|
exit 1
|
|
done
|
|
```
|
|
|
|
Launch it detached and keep its log:
|
|
|
|
nohup sh <work root>/projects/<slug>/drivers/workd.sh <project root> >/dev/null 2>&1 &
|
|
|
|
The driver only *advances* a prepared queue; it never writes briefs. Preparing a good
|
|
queue and settling a failure is the agent's work.
|
|
|
|
## Sleeping and waking
|
|
|
|
- Before the operator sleeps: reconcile, make sure the queue is full enough to cover
|
|
the night, and confirm the driver is running. A drained queue is fine; it idles.
|
|
- Configure a **failure notification** (`references/notifications.md`) so a stopped
|
|
driver is not silent. On an overnight failure the driver stops after one retry; that
|
|
is by design — do not add retry loops to make it look busy.
|
|
- On waking, reconcile first, then answer with the status block: what landed, what
|
|
failed and why, what is next.
|
|
|
|
## Conciseness
|
|
|
|
The operator wants the result, not the shift report.
|
|
|
|
- Match the answer's length to the question. A yes/no or one-line question gets one or
|
|
two sentences; never answer a narrow question with a sectioned report.
|
|
- Answer the question that was asked first. Put any extra fact that changes a decision in
|
|
a single clause, not in a new section.
|
|
- Answer with the status block for status; do not narrate polling or re-print the
|
|
queue when nothing changed.
|
|
- Report a run by its outcome and evidence (`references/verification.md`), not the
|
|
steps you took to watch it.
|
|
- Ask only for a decision the operator owns (accept, publish, restart, install). Batch
|
|
the day's questions rather than asking per item.
|
|
- Never claim a check, build or screenshot you did not run.
|