Files
pipelines/internal/author/skill/references/debugging.md
T
alan 1b4674d22c Show the derived stage verdict atop the stage view in the run detail
Move a completed stage visit's derived verdict out of the Progress summary
and into the top of the selected stage's Result section, so the chosen
option, its concise reason and the referenced artifacts are visible where
the stage is inspected.

- Add stageVerdictLines, which renders one line "Verdict: <option> · <reason>"
  (still stating the option when the reason is absent) plus a following
  "Referenced artifacts:" line only when artifacts exist, and returns nil
  for a nil verdict so nothing extra is shown.
- Prepend those lines in runSectionLines after the stage state and before
  the stage output and evidence, without adding a section or changing the
  panel layout.
- Update the TUI test to render through the run-detail view, asserting the
  option, reason and artifact names appear, and that a nil verdict adds
  nothing; document the TUI placement in the debugging skill reference.

Omni-search work already present in the working tree (including the
runDuration→protocol.Duration refactor in run_detail.go) is unrelated to
this change and is left uncommitted.
2026-10-16 15:12:23 +01:00

132 lines
7.8 KiB
Markdown

# Debugging wedged and failed runs
Debug from **retained evidence**, never from an agent's summary. Open the run with
`CLI_PREFIX inspect RUN_ID --json` and read the exact stage outputs, outcomes,
diagnostics, file changes and artifacts. Each completed step also carries a
`verdict`: the chosen option (`outcome → target`), a one-line reason with its
provenance (`model`, `fallback` or `absent`), and the recorded artifacts the reason
refers to. In the TUI, selecting a stage shows that verdict at the top of the
stage's result, above its output and evidence. Treat the reason as a claim to check
against the step's output, never as evidence in itself. Identify evidence-supported
causes and say what remains unknown. Never fix a run's symptom by editing the
project by hand: fix the pipeline definition or the Pipelines/harness service, or
record the failure and continue.
Each playbook is *symptom → how to confirm → what to do*.
## HARNESS TRANSPORT errors
- **Symptom** A stage or run fails with the sanitized category `HARNESS TRANSPORT`;
the model may have done work but no final result was read.
- **Confirm** `inspect --json` shows the category on the failed stage. A request
failure, an active-session poll failure and a final-readback failure all carry this
one category.
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
pre-admission failure (nothing executed) from a post-admission one (tools may have
run). An explicit read-only stage may retry; a stage whose tools include `edit`/
`shell`, or that has no tools block, **never replays admitted work**. Do not re-run
a full agent stage merely to recover a response's formatting. If it recurs, check
the service and harness health (`references/operations.md`); do not weaken the
stage to get past it.
## Malformed model results
- **Symptom** The category `FINAL RESULT VALIDATION` names a duplicate nested field
or malformed/ambiguous JSON with a byte offset.
- **Confirm** Read the diagnostic and the retained final result: the stage's result
does not match its `output` contract. A retained transcript can be a valid bounded
preview envelope, not the complete history.
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
appends missing closers, drops one trailing comma, or inserts the single
comma/colon the parser expected; only if that is ambiguous does one deny-all
formatter request run, with a deadline derived from the payload size and the
attempt's remaining budget. Each path records a `repaired`, `partial` or
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
is never accepted as complete. On an explicit read-only stage a malformed result
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
is never replayed, though its malformed result can still be repaired by the
deterministic pass or the fresh deny-all formatter session. Never accept prose,
flattened output or a tool-call envelope as success. If one stage keeps
producing invalid JSON, tighten its `output`/`prompt` in the **pipeline
definition** (`references/format.md`) — a pipeline fix, not a project edit.
## A reviewer or visual reviewer that blocks, rejects repeatedly, or hangs
- **Symptom** The reviewer returns `rejected` on every visit, returns `blocked`, or
the run stalls inside a reviewer stage.
- **Confirm** Inspect every visit and read what the reviewer actually ran and saw. A
rejection loop is bounded by the stage `limits` (`max_visits`); at the cap the
rejection is retained and routes to `#failure`. `blocked` means the reviewer could
not obtain the evidence it needed.
- **Do**
- A visual reviewer that returns `blocked` with no screenshots is right to refuse.
Fix the capture path (below) rather than removing it.
- Rejections that repeat the same finding mean either the implementer is not fixing
it or the acceptance criteria contradict the brief. Read the retained findings,
correct the brief or the pipeline's criteria, and re-run.
- **Never** remove a review stage, relax an acceptance check, or approve the review
yourself to get past it (`references/orchestration.md`).
- A reviewer that hangs on an unbounded command (for example a filesystem-root
scan) can leave a process in uninterruptible sleep and stall the server or
Docker. Cancel the run, stop the offending command, and add a scope instruction
to the brief or pipeline: bounded, relevant paths only.
## Workspace-guard refusals
- **Symptom** A run is refused with "run N is already active in workspace ...".
- **Confirm** The message names the conflicting run and its workspace; the guard is
working as designed — one active run per resolved workspace.
- **Do** Wait for the conflicting run, or start this one in a different workspace
(another worktree/checkout, which is what a lane is for — `references/lanes.md`).
Do not delete the active run, the database or sessions to clear it. An **orphan**
active session left by a restart with no owning active run still refuses new work
until it is interrupted; reusing the active run's original request key reattaches
idempotently.
## A client older than the server dropping fields
- **Symptom** `inspect --json` or `ping --json` is missing fields this skill
describes, or a flag is rejected as unknown.
- **Confirm** `ping --json` for the server build identity and `version --json` for the
client. An older server omits `build`; an older client may not send newer fields.
- **Do** Treat it as a version mismatch, not a run failure. State both versions and
the mismatch. **Do not infer** that a source fix or an updated skill is deployed.
Use only flags the installed help shows; do not restart or reinstall a service to
"fix" it without the operator (`references/operations.md`).
## A server restart killing in-flight runs
- **Symptom** Runs that were active when the server restarted are now `interrupted`,
with every step's `agent_active=false`.
- **Confirm** The status is `interrupted`, not `failed`, and there is **no automatic
replay**.
- **Do** Recovery is explicit. Inspect what completed; retry only when the server is
healthy and the previous attempt's cleanup has stopped. Never restart a server that
has runs in flight — quiesce first (`references/lanes.md`). Keep the interrupted run
as history; never describe a restart, retry or cancellation as rollback.
## Screenshot capture that silently produces non-images
- **Symptom** A run's "screenshot" artifact, or an image cited in an approval, is not
an image at all — for example 100+ KB of text beginning with a
`[Warning] Multiple displays were found` line — yet the capture command exited 0.
- **Confirm** Check the **bytes** (magic bytes or the reported content type), not the
`.png` name. A file that is not a real image is not evidence.
- **Do** Capture on the device to a file and pull it — never write `exec-out` or a
shell redirect straight to a `.png`; name the display explicitly when more than one
exists; validate the pulled bytes before citing them. A capture that fails
validation is deleted and the stage returns `blocked`; never cite it to support an
approval. Put the capture helper in the repository so every UI run reuses one path.
See the example in `references/delivery.md`.
## Duplicated work after a lane or chain restart
- **Symptom** Items already completed run again after a lane or chain script
restarts: the same brief is re-invoked, side effects repeat, or a commit is made
twice.
- **Confirm** Compare the lane's list against its completed-items record. A list alone
has no resume point.
- **Do** Persist a cursor and resume from it (`references/lanes.md`). Record an item
only once it has landed, and skip recorded items on startup. Never relaunch a
duplicate of a run that is still active — reattach with its request key instead.