01d5dad9b9
Rewrite the workflow documentation to describe how the system actually works: the agent (the session) is the brain and advances the queue by hand through the `pipelines` CLI. There is no orchestration script, no driver, no polling loop and no lane machinery, and none may be written. - references/work.md: rewritten end to end. The queue-driver script (drivers/workd.sh), the claim-file ledger with atomic renames, the PAUSED marker and the poll loop are gone. State is now the server (authoritative for run state), the queue file (authoritative for intent) and the conversation. The agent reads the queue, picks the item, creates its worktree/branch and launches it with `pipelines run <pipeline> <brief> --json --yes`; progress is event-driven via a background `pipelines wait <run-id> [...]` (never polled); on completion the agent reconciles with `pipelines inspect <run-id> --json`, verifies it landed and starts the next item. - references/queue.md: the landing gate runs the command directly (`git merge-base --is-ancestor <branch> main`, exit 0 = landed) instead of a `land-check.sh` helper; the never-delete-branch / never-remove-worktree rules are kept, and the waiting section states the agent advances the queue by hand. - references/orchestration.md: intro and "Running and watching" state the agent-driven, event-driven model; "Pipeline choice" names the current set (Android, TUI, Server, Generic, Plan and land, UI sweep, Probe) by area and records the retired names. - references/debugging.md: "Duplicated work after a driver restart" became "…after a session restart", resuming from the queue file not a cursor file. - SKILL.md: the routing row no longer says "queue driver". - AGENTS.md: the maintenance workflow names the current pipeline set instead of the retired `Pipelines maintenance - plan implement review`. - internal/tui/AGENTS.md: the restart line no longer names lanes. - docs/notes/agent-driven-work-model.md: the docs-fold note recording that docs/cli.md needs its "Scripted runs"/`--wait` framing aligned to the event-driven model, and that the dated evidence docs stay as they are. Checked before committing: no packaged template directory exists in this repo, so none needed updating; internal/author/install.go's managed file inventory needs no change (no packaged reference was added or removed — the 13 listed files all match internal/author/skill/references/); no stale `queue driver`/`workd.sh`/ `claim-file` references remain anywhere under internal/author. Genuine device tooling (android/scripts/capture_png.sh-style) and server-side polling prose were left alone. Checks: `go test ./internal/author -count=1` ok; `go build ./...` clean; `go vet ./...` clean; `gofmt -l .` clean.
146 lines
9.0 KiB
Markdown
146 lines
9.0 KiB
Markdown
# Debugging wedged and failed runs
|
|
|
|
Debug from **retained evidence**, never from an agent's summary. Open the run with
|
|
`CLI_PREFIX inspect RUN_ID --json` and read the exact stage outputs, outcomes,
|
|
diagnostics, file changes and artifacts. Each completed step also carries a
|
|
`verdict`: the chosen option (`outcome → target`), a one-line reason with its
|
|
provenance (`model`, `fallback` or `absent`), and the recorded artifacts the reason
|
|
refers to. In the TUI, selecting a stage shows that verdict at the top of the
|
|
stage's result, above its output and evidence. Treat the reason as a claim to check
|
|
against the step's output, never as evidence in itself. Identify evidence-supported
|
|
causes and say what remains unknown. Never fix a run's symptom by editing the
|
|
project by hand: fix the pipeline definition or the Pipelines/harness service, or
|
|
record the failure and continue.
|
|
|
|
Each playbook is *symptom → how to confirm → what to do*.
|
|
|
|
## HARNESS TRANSPORT errors
|
|
|
|
- **Symptom** A stage or run fails with the sanitized category `HARNESS TRANSPORT`;
|
|
the model may have done work but no final result was read.
|
|
- **Confirm** `inspect --json` shows the category on the failed stage. A request
|
|
failure, an active-session poll failure and a final-readback failure all carry this
|
|
one category. A request that failed while the context was still in flight shows
|
|
both the sanitized transport detail and `context deadline exceeded`; an execution
|
|
error names the sanitized error name/tag and any HTTP status, not a provider body.
|
|
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
|
|
pre-admission reach failure (connecting or creating the session; nothing executed)
|
|
from a post-admission one (tools may have run). A pre-admission reach failure
|
|
replays nothing, so it is retried automatically within `retries` for any stage; a
|
|
post-admission failure is retried only on an explicit read-only stage. A stage
|
|
whose tools include `edit`/`shell`, or that has no tools block, **never replays
|
|
admitted work**. Do not re-run a full agent stage merely to recover a response's
|
|
formatting. If it recurs, check the service and harness health
|
|
(`references/operations.md`); do not weaken the stage to get past it.
|
|
|
|
## Malformed model results
|
|
|
|
- **Symptom** The category `FINAL RESULT VALIDATION` names a duplicate nested field
|
|
or malformed/ambiguous JSON with a byte offset.
|
|
- **Confirm** Read the diagnostic and the retained final result: the stage's result
|
|
does not match its `output` contract. A retained transcript can be a valid bounded
|
|
preview envelope, not the complete history.
|
|
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
|
|
appends missing closers, drops one trailing comma, or inserts the single
|
|
comma/colon the parser expected; only if that is ambiguous does one deny-all
|
|
formatter request run, with a deadline derived from the payload size, the stage
|
|
timeout and the attempt's remaining budget, clamped so it never extends the
|
|
stage's own deadline. Each path records a `repaired`, `partial` or
|
|
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
|
|
is never accepted as complete. On an explicit read-only stage a malformed result
|
|
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
|
|
is never replayed, though its malformed result can still be repaired by the
|
|
deterministic pass or the fresh deny-all formatter session. Never accept prose,
|
|
flattened output or a tool-call envelope as success. If one stage keeps
|
|
producing invalid JSON, tighten its `output`/`prompt` in the **pipeline
|
|
definition** (`references/format.md`) — a pipeline fix, not a project edit.
|
|
|
|
## A reviewer or visual reviewer that blocks, rejects repeatedly, or hangs
|
|
|
|
- **Symptom** The reviewer returns `rejected` on every visit, returns `blocked`, or
|
|
the run stalls inside a reviewer stage.
|
|
- **Confirm** Inspect every visit and read what the reviewer actually ran and saw. A
|
|
rejection loop is bounded by the stage `limits` (`max_visits`); at the cap the
|
|
rejection is retained and routes to `#failure`. `blocked` means the reviewer could
|
|
not obtain the evidence it needed.
|
|
- **Do**
|
|
- A visual reviewer that returns `blocked` with no screenshots is right to refuse.
|
|
Fix the capture path (below) rather than removing it.
|
|
- Rejections that repeat the same finding mean either the implementer is not fixing
|
|
it or the acceptance criteria contradict the brief. Read the retained findings,
|
|
correct the brief or the pipeline's criteria, and re-run.
|
|
- **Never** remove a review stage, relax an acceptance check, or approve the review
|
|
yourself to get past it (`references/orchestration.md`).
|
|
- A reviewer that hangs on an unbounded command (for example a filesystem-root
|
|
scan) can leave a process in uninterruptible sleep and stall the server or
|
|
Docker. Cancel the run, stop the offending command, and add a scope instruction
|
|
to the brief or pipeline: bounded, relevant paths only.
|
|
|
|
## Workspace-guard refusals
|
|
|
|
- **Symptom** A run is refused with "run N is already active in workspace ...".
|
|
- **Confirm** The message names the conflicting run and its workspace; the guard is
|
|
working as designed — one active run per resolved workspace.
|
|
- **Do** Wait for the conflicting run, or start this one in a different workspace
|
|
(another worktree/checkout, which is what a second item's worktree is for — `references/queue.md`).
|
|
Do not delete the active run, the database or sessions to clear it. An **orphan**
|
|
active session left by a restart with no owning active run still refuses new work
|
|
until it is interrupted; reusing the active run's original request key reattaches
|
|
idempotently.
|
|
|
|
## A client older than the server dropping fields
|
|
|
|
- **Symptom** `inspect --json` or `ping --json` is missing fields this skill
|
|
describes, or a flag is rejected as unknown.
|
|
- **Confirm** The client now reports this itself. A server response carrying a
|
|
top-level field this client does not model prints a one-time
|
|
`server sent unrecognized top-level field(s) ...` notice; a same-protocol build
|
|
mismatch prints one `warning:` naming both builds on stderr; and a server
|
|
speaking a newer protocol fails the command non-zero. `ping --json` shows the
|
|
server build identity and `version --json` the client; `ping`/`connection`
|
|
also print the client/server relationship. An older server omits `build`.
|
|
- **Do** Treat it as a version mismatch, not a run failure. State both versions and
|
|
the mismatch. **Do not infer** that a source fix or an updated skill is deployed.
|
|
Use only flags the installed help shows; do not restart or reinstall a service to
|
|
"fix" it without the operator (`references/operations.md`). Rebuild or reinstall
|
|
the CLI so it matches the server; never hand-edit an installed binary.
|
|
|
|
## A server restart killing in-flight runs
|
|
|
|
- **Symptom** Runs that were active when the server restarted are now `interrupted`,
|
|
with every step's `agent_active=false`.
|
|
- **Confirm** The status is `interrupted`, not `failed`, and there is **no automatic
|
|
replay**.
|
|
- **Do** Recovery is explicit. Inspect what completed; retry only when the server is
|
|
healthy and the previous attempt's cleanup has stopped. Never restart a server that
|
|
has runs in flight — quiesce first (`references/queue.md`). Keep the interrupted run
|
|
as history; never describe a restart, retry or cancellation as rollback.
|
|
|
|
## Screenshot capture that silently produces non-images
|
|
|
|
- **Symptom** A run's "screenshot" artifact, or an image cited in an approval, is not
|
|
an image at all — for example 100+ KB of text beginning with a
|
|
`[Warning] Multiple displays were found` line — yet the capture command exited 0.
|
|
- **Confirm** Check the **bytes** (magic bytes or the reported content type), not the
|
|
`.png` name. A file that is not a real image is not evidence.
|
|
- **Do** Capture on the device to a file and pull it — never write `exec-out` or a
|
|
shell redirect straight to a `.png`; name the display explicitly when more than one
|
|
exists; validate the pulled bytes before citing them. A capture that fails
|
|
validation is deleted and the stage returns `blocked`; never cite it to support an
|
|
approval. Put the capture helper in the repository so every UI run reuses one path.
|
|
See the example in `references/delivery.md`.
|
|
|
|
## Duplicated work after a session restart
|
|
|
|
- **Symptom** Items already completed run again after the session that was advancing the
|
|
queue restarts: the same brief is re-invoked, side effects repeat, or a commit is made
|
|
twice.
|
|
- **Confirm** Compare the queue's items against their recorded state and the server's run
|
|
history (`inspect RUN_ID --json`). A list alone has no resume point.
|
|
- **Do** Record each item's state in the queue itself (`references/queue.md`) and resume
|
|
from it: read the queue top to bottom, skip items already marked done, and settle the
|
|
ones marked running by inspecting their run id. Record an item only once it has landed.
|
|
Never relaunch a duplicate of a run that is still active — reattach with its request key
|
|
instead. Do not build a driver or cursor file to remember position; the queue file is
|
|
the resume point.
|