Files
pipelines/internal/author/skill/references/work.md
T

12 KiB

Project work ledger and sessions

Work in a project is not a chat. It is a durable ledger: what is queued, what is running, what has landed, and what is blocked. A session — this conversation — starts, pauses, resumes and reports that ledger. Nothing important lives only in the conversation: running the skill again, in any project on this machine, reads the ledger and continues. This is the multi-project lifecycle on top of references/orchestration.md (work items, briefs, pipeline choice) and references/lanes.md (parallel workspaces).

Where the ledger lives

Keep state outside repositories, under one work root, so it never collides with a lane's tree or a commit:

${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work/
  registry.json                   every project this machine knows
  projects/<slug>/
    state.json                    project, root, endpoint, pipeline, workspace
    notes.md                      recon, decisions, open questions, blockers
    queue/<NNNN-slug>.brief       pending briefs, read in filename order
    active/<id>.brief, <id>.claim claimed: the brief plus its run id and key
    done/<id>                     landed: run id, status, finished
    failed/<id> and <id>.brief    failed twice: run id, status, diagnostic
    log/driver.log
    drivers/workd.sh              this project's queue driver
    PAUSED                        marker: stop dispatching new items
  • <slug> is <basename>-<8 hex of sha256(absolute project root)>. Resolve a worktree to its main repository so a lane shares the project's one ledger.
  • PIPELINES_WORK_DIR overrides the work root. registry.json maps projects to roots; add a project the first time it is worked on and keep it current.
  • state.json holds project, root, endpoint (the CLI's --url), pipeline (the default pipeline) and workspace (the default run directory).
  • The ledger is the operator's record: keep notes.md truthful about decisions taken and left open, never narrate it as a diary.

Operations

Map one operator sentence to one operation. Do the operation, then answer with the status block — nothing else.

Operator says Do
start / work on X Reconnoitre, register the project, write small briefs into queue/, then run the driver.
status / how is it going Print the status block.
queue "…" / add … Reconnoitre, split into small items, append briefs; the driver picks them up.
pause / stop for now Write PAUSED. The item in flight finishes; no new one starts.
resume / continue Reconcile, remove PAUSED, run the driver.
drop item N Move its brief to failed/ with a reason. Never silently delete work.
done / stop everything Pause and report what landed and what remains.

Reconcile before acting, always. The server is authoritative for run state; the ledger is authoritative for intent. On every invocation: read registry.json and the project's ledger; for each active/ claim, CLI_PREFIX inspect <run id> --json and settle it (running → keep waiting; succeeded → done/; failed/interrupted → failed/); only then do the requested operation. Never trust a claim file for a status, and never start a second run for an item whose request key is already admitted — that key is what makes a redispatch attach instead of duplicate.

Status block (these lines, no prose around them):

pipelines — active
queue: 4 pending · 1 running (run 185 · reviewer · 6m) · 12 done · 2 failed
next: 0005-graph-arrows on "Focused tweak - implement review commit"
blocked: none

The brief, and per-item directives

Write every brief exactly as references/orchestration.md describes. A brief file in queue/ may open with up to three directive lines, stripped before the prompt is sent:

# pipeline: Focused tweak - implement review commit
# workspace: /path/to/lane-app2
# device: emulator-5556

Goal: …

pipeline overrides the default for that item; workspace routes an item to a lane's checkout (a driver only claims items for its own workspace); device is for the brief's own text. Number briefs 0001-…, 0002-… so order is explicit.

The driver

A queue is advanced by a small driver, so work continues while the operator sleeps. Write the driver once per project to drivers/workd.sh and launch it; it is restart-safe (it resumes from done/, reattaches to active/ runs by id, and never starts a duplicate), and it stops on the first item that fails twice. Run one driver per workspace; several drivers sharing one queue advance different lanes in parallel because an item is claimed by an atomic rename.

#!/bin/sh
# workd.sh - advance one project's Pipelines work ledger.
#
#   sh workd.sh [PROJECT_ROOT] [WORKSPACE]
#
# PROJECT_ROOT identifies the ledger (default: the current directory).
# WORKSPACE is the checkout this driver may run in (default: the ledger's
# workspace). One driver per workspace lets several lanes advance one shared
# queue: an item is claimed by an atomic rename, so it runs exactly once.
#
# Restart-safe. It resumes from done/, reattaches to active/ runs by id and
# never starts a duplicate. PIPELINES_WORK_POLL sets the poll gap in seconds
# (default 20); PIPELINES_CLI uses another client binary.
set -u

CLI=${PIPELINES_CLI:-pipelines}
POLL=${PIPELINES_WORK_POLL:-20}
ROOT=$(cd "${1:-$(pwd)}" && pwd)
WORK_ROOT=${PIPELINES_WORK_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/pipelines/work}
SLUG=$(basename "$ROOT")-$(printf '%s' "$ROOT" | shasum -a 256 | cut -c1-8)
D=$WORK_ROOT/projects/$SLUG
LOG=$D/log/driver.log
mkdir -p "$D/queue" "$D/active" "$D/done" "$D/failed" "$D/log" 2>/dev/null

say() { printf '%s %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" | tee -a "$LOG"; }
jfield() { python3 -c 'import json,sys
try: d=json.load(open(sys.argv[1]))
except Exception: d={}
print(d.get(sys.argv[2]) or "")' "$1" "$2"; }
dstatus() { "$CLI" inspect "$1" --json 2>/dev/null | python3 -c 'import json,sys
try: d=json.load(sys.stdin)
except Exception: print("unknown"); raise SystemExit
print(d.get("status") or "unknown")'; }
item_field() { sed -n "s/^# *$2 *: *//p" "$1" 2>/dev/null | head -1; }   # from a brief
claim_field() { sed -n "s/^$2=//p" "$1" 2>/dev/null | head -1; }         # from a claim file
prompt_of() { python3 -c '
import re,sys
lines=open(sys.argv[1],encoding="utf-8",errors="replace").read().splitlines()
while lines and re.match(r"^#\s*(pipeline|workspace|device)\s*:", lines[0]):
    lines.pop(0)
while lines and lines[0].strip()=="":
    lines.pop(0)
print("\n".join(lines))' "$1"; }

[ -f "$D/state.json" ] || { echo "no ledger: $D/state.json" >&2; exit 2; }
PIPELINE=$(jfield "$D/state.json" pipeline)
ENDPOINT=$(jfield "$D/state.json" endpoint)
WS=${2:-$(jfield "$D/state.json" workspace)}
[ -n "$WS" ] || WS=$ROOT
[ -n "$PIPELINE" ] || { echo "state.json has no pipeline" >&2; exit 2; }

wait_terminal() { # $1 run id -> terminal status (or "unknown" if unreadable)
  bad=0
  while :; do
    st=$(dstatus "$1")
    case "$st" in
      succeeded|failed|canceled|interrupted) echo "$st"; return 0;;
      unknown) bad=$((bad+1)); [ "$bad" -ge 5 ] && { echo "unknown"; return 0; };;
      *) bad=0;;
    esac
    sleep "$POLL"
  done
}

dispatch() { # $1 brief file  $2 pipeline  $3 key -> run id (empty on failure)
  if [ -n "$ENDPOINT" ]; then
    out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" --url "$ENDPOINT" 2>&1)
  else
    out=$("$CLI" run "$2" "$(prompt_of "$1")" --json --yes --workspace "$WS" --request-key "$3" 2>&1)
  fi
  printf '%s' "$out" | python3 -c 'import json,sys
try: d=json.load(sys.stdin)
except Exception: print("")
else: print((d.get("run_detail") or {}).get("id") or "")'
}

record() { # $1 id  $2 status  $3 run id
  where=done; [ "$2" = succeeded ] || where=failed
  printf 'run_id=%s\nstatus=%s\nfinished=%s\n' "$3" "$2" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" > "$D/$where/$1"
  rm -f "$D/active/$1.brief" "$D/active/$1.claim"
}

run_item() { # $1 id  $2 brief  $3 pipeline  $4 key -> 0 succeeded / 1 failed
  rid=$(dispatch "$2" "$3" "$4")
  if [ -z "$rid" ]; then printf 'status=dispatch-failed\n' > "$D/active/$1.claim"; say "dispatch failed: $1"; return 1; fi
  printf 'run_id=%s\n' "$rid" > "$D/active/$1.claim"
  st=$(wait_terminal "$rid")
  if [ "$st" = succeeded ]; then record "$1" succeeded "$rid"; say "done $1 (run $rid)"; return 0; fi
  printf 'run_id=%s\nstatus=%s\n' "$rid" "$st" > "$D/active/$1.claim"
  say "failed $1 (run $rid, $st)"
  return 1
}

reconcile() { # settle anything the previous process left claimed
  for a in "$D"/active/*; do
    [ -e "$a" ] || continue
    id=$(basename "$a"); id=${id%.brief}; id=${id%.claim}
    rid=$(claim_field "$D/active/$id.claim" run_id)
    if [ -z "$rid" ]; then
      say "requeueing undispatched $id"
      [ -e "$D/active/$id.brief" ] && mv "$D/active/$id.brief" "$D/queue/$id.brief"
      rm -f "$D/active/$id.claim"
      continue
    fi
    st=$(wait_terminal "$rid")
    record "$id" "$st" "$rid"
    say "reconciled $id as $st (run $rid)"
  done
}

reconcile

while :; do
  if [ -e "$D/PAUSED" ]; then sleep "$POLL"; continue; fi
  next=""
  for q in $(ls -1 "$D"/queue/*.brief 2>/dev/null | sort); do
    id=$(basename "$q" .brief)
    [ -e "$D/done/$id" ] && continue
    [ -e "$D/failed/$id" ] && continue
    iws=$(item_field "$q" workspace)
    [ -n "$iws" ] && [ "$iws" != "$WS" ] && continue
    next=$q; break
  done
  if [ -z "$next" ]; then say "idle (no claimable items for $WS)"; exit 0; fi
  id=$(basename "$next" .brief)
  mv "$next" "$D/active/$id.brief" 2>/dev/null || continue
  ipipe=$(item_field "$D/active/$id.brief" pipeline)
  [ -n "$ipipe" ] || ipipe=$PIPELINE
  key="work-$SLUG-$id"
  if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key"; then continue; fi
  if [ "$(claim_field "$D/active/$id.claim" status)" = "dispatch-failed" ]; then
    say "STOP: could not dispatch $id"
    mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
    exit 1
  fi
  say "retrying $id once"
  if run_item "$id" "$D/active/$id.brief" "$ipipe" "$key-r1"; then continue; fi
  rid=$(claim_field "$D/active/$id.claim" run_id)
  st=$(claim_field "$D/active/$id.claim" status)
  printf 'run_id=%s\nstatus=%s\nattempts=2\n' "$rid" "${st:-failed}" > "$D/failed/$id"
  mv "$D/active/$id.brief" "$D/failed/$id.brief" 2>/dev/null
  rm -f "$D/active/$id.claim"
  say "STOP: $id failed twice; recorded under failed/"
  exit 1
done

Launch it detached and keep its log:

nohup sh <work root>/projects/<slug>/drivers/workd.sh <project root> >/dev/null 2>&1 &

The driver only advances a prepared queue; it never writes briefs. Preparing a good queue and settling a failure is the agent's work.

Sleeping and waking

  • Before the operator sleeps: reconcile, make sure the queue is full enough to cover the night, and confirm the driver is running. A drained queue is fine; it idles.
  • Configure a failure notification (references/notifications.md) so a stopped driver is not silent. On an overnight failure the driver stops after one retry; that is by design — do not add retry loops to make it look busy.
  • On waking, reconcile first, then answer with the status block: what landed, what failed and why, what is next.

Conciseness

The operator wants the result, not the shift report.

  • Match the answer's length to the question. A yes/no or one-line question gets one or two sentences; never answer a narrow question with a sectioned report.
  • Answer the question that was asked first. Put any extra fact that changes a decision in a single clause, not in a new section.
  • Answer with the status block for status; do not narrate polling or re-print the queue when nothing changed.
  • Report a run by its outcome and evidence (references/verification.md), not the steps you took to watch it.
  • Ask only for a decision the operator owns (accept, publish, restart, install). Batch the day's questions rather than asking per item.
  • Never claim a check, build or screenshot you did not run.