9f88e4ff71
A model stage whose final result was malformed failed the whole run even when
the work and evidence had survived, because the only recovery was an isolated
formatter call on a fixed 30-second deadline that could time out on its own
(run 55, ui_reviewer of pipeline 9 v6).
Add an ordered, deterministic local repair pass (internal/stage/result_repair.go)
that runs before any model call: (a) append missing closing delimiters via the
existing closeJSONContainers; (b) remove one unambiguous trailing comma before a
closing } or ]; (c) insert exactly the single comma/colon the JSON parser reported
at its byte offset. Every candidate must pass losslessSyntaxRepair(original,
candidate) and decodeModelResult before it is accepted, so no field value is
invented, removed, reordered or rewritten. Ambiguous payloads (an unescaped quote
or colon inside a string value, or a stray delimiter with no single completion)
return not-ok and fall through to the bounded model re-emit rather than guessing.
stageResultText still strips surrounding prose and a single code fence first;
model_prompt.go now also strips prose around a single balanced-but-malformed
object span so the deterministic pass sees just the object.
Replace the fixed 30-second formatter deadline with repairDeadline(payloadBytes,
remaining), which grows monotonically with payload size, has a floor, and is
clamped to the attempt's remaining budget minus a safety margin, so the stage
context remains the outer bound and a large result is not guaranteed to time out.
The one-request-per-attempt, 32 KiB bound and the deny-all, retained, new-session
property of RepairFormat are unchanged for OpenCodeClient and PiClient.
Record an explicit recovery status on every terminal path: repaired, partial or
unrecoverable, plus a byte offset. A candidate that is valid JSON but still fails
envelope validation is marked partial and never returned as a completed result;
the unrecoverable path retains the bounded original payload, the precise parse
error (including its byte offset) and surfaces the status in the FINAL RESULT
VALIDATION diagnostic.
safeToReplay is intentionally unchanged. The deterministic local repair and the
fresh deny-all, no-tools, retained formatter session are independent of the
stage's tool set and never re-prompt or replay the admitted agent session, so a
shell-enabled review stage gains malformed-result recovery without any replay of
admitted work. Risk: no change to mutation safety; the only added cost is one
bounded formatter call, already budgeted. Replayability is not extended for shell
stages.
Docs and packaged skill references (docs/model-results.md,
internal/author/skill/references/{format,debugging,verification}.md) describe the
three-tier recovery, the status words and the derived deadline; the safeToReplay
decision and its risk are recorded there alongside the commit.
Tests add mocked, offline coverage for the deterministic and partial paths, a
repairDeadline monotonicity/floor/clamp table, and a shell-stage regression
proving a recoverable malformed result does not fail the run or replay tool work.
127 lines
7.4 KiB
Markdown
127 lines
7.4 KiB
Markdown
# Debugging wedged and failed runs
|
|
|
|
Debug from **retained evidence**, never from an agent's summary. Open the run with
|
|
`CLI_PREFIX inspect RUN_ID --json` and read the exact stage outputs, outcomes,
|
|
diagnostics, file changes and artifacts. Identify evidence-supported causes and say
|
|
what remains unknown. Never fix a run's symptom by editing the project by hand: fix
|
|
the pipeline definition or the Pipelines/harness service, or record the failure and
|
|
continue.
|
|
|
|
Each playbook is *symptom → how to confirm → what to do*.
|
|
|
|
## HARNESS TRANSPORT errors
|
|
|
|
- **Symptom** A stage or run fails with the sanitized category `HARNESS TRANSPORT`;
|
|
the model may have done work but no final result was read.
|
|
- **Confirm** `inspect --json` shows the category on the failed stage. A request
|
|
failure, an active-session poll failure and a final-readback failure all carry this
|
|
one category.
|
|
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
|
|
pre-admission failure (nothing executed) from a post-admission one (tools may have
|
|
run). An explicit read-only stage may retry; a stage whose tools include `edit`/
|
|
`shell`, or that has no tools block, **never replays admitted work**. Do not re-run
|
|
a full agent stage merely to recover a response's formatting. If it recurs, check
|
|
the service and harness health (`references/operations.md`); do not weaken the
|
|
stage to get past it.
|
|
|
|
## Malformed model results
|
|
|
|
- **Symptom** The category `FINAL RESULT VALIDATION` names a duplicate nested field
|
|
or malformed/ambiguous JSON with a byte offset.
|
|
- **Confirm** Read the diagnostic and the retained final result: the stage's result
|
|
does not match its `output` contract. A retained transcript can be a valid bounded
|
|
preview envelope, not the complete history.
|
|
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
|
|
appends missing closers, drops one trailing comma, or inserts the single
|
|
comma/colon the parser expected; only if that is ambiguous does one deny-all
|
|
formatter request run, with a deadline derived from the payload size and the
|
|
attempt's remaining budget. Each path records a `repaired`, `partial` or
|
|
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
|
|
is never accepted as complete. On an explicit read-only stage a malformed result
|
|
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
|
|
is never replayed, though its malformed result can still be repaired by the
|
|
deterministic pass or the fresh deny-all formatter session. Never accept prose,
|
|
flattened output or a tool-call envelope as success. If one stage keeps
|
|
producing invalid JSON, tighten its `output`/`prompt` in the **pipeline
|
|
definition** (`references/format.md`) — a pipeline fix, not a project edit.
|
|
|
|
## A reviewer or visual reviewer that blocks, rejects repeatedly, or hangs
|
|
|
|
- **Symptom** The reviewer returns `rejected` on every visit, returns `blocked`, or
|
|
the run stalls inside a reviewer stage.
|
|
- **Confirm** Inspect every visit and read what the reviewer actually ran and saw. A
|
|
rejection loop is bounded by the stage `limits` (`max_visits`); at the cap the
|
|
rejection is retained and routes to `#failure`. `blocked` means the reviewer could
|
|
not obtain the evidence it needed.
|
|
- **Do**
|
|
- A visual reviewer that returns `blocked` with no screenshots is right to refuse.
|
|
Fix the capture path (below) rather than removing it.
|
|
- Rejections that repeat the same finding mean either the implementer is not fixing
|
|
it or the acceptance criteria contradict the brief. Read the retained findings,
|
|
correct the brief or the pipeline's criteria, and re-run.
|
|
- **Never** remove a review stage, relax an acceptance check, or approve the review
|
|
yourself to get past it (`references/orchestration.md`).
|
|
- A reviewer that hangs on an unbounded command (for example a filesystem-root
|
|
scan) can leave a process in uninterruptible sleep and stall the server or
|
|
Docker. Cancel the run, stop the offending command, and add a scope instruction
|
|
to the brief or pipeline: bounded, relevant paths only.
|
|
|
|
## Workspace-guard refusals
|
|
|
|
- **Symptom** A run is refused with "run N is already active in workspace ...".
|
|
- **Confirm** The message names the conflicting run and its workspace; the guard is
|
|
working as designed — one active run per resolved workspace.
|
|
- **Do** Wait for the conflicting run, or start this one in a different workspace
|
|
(another worktree/checkout, which is what a lane is for — `references/lanes.md`).
|
|
Do not delete the active run, the database or sessions to clear it. An **orphan**
|
|
active session left by a restart with no owning active run still refuses new work
|
|
until it is interrupted; reusing the active run's original request key reattaches
|
|
idempotently.
|
|
|
|
## A client older than the server dropping fields
|
|
|
|
- **Symptom** `inspect --json` or `ping --json` is missing fields this skill
|
|
describes, or a flag is rejected as unknown.
|
|
- **Confirm** `ping --json` for the server build identity and `version --json` for the
|
|
client. An older server omits `build`; an older client may not send newer fields.
|
|
- **Do** Treat it as a version mismatch, not a run failure. State both versions and
|
|
the mismatch. **Do not infer** that a source fix or an updated skill is deployed.
|
|
Use only flags the installed help shows; do not restart or reinstall a service to
|
|
"fix" it without the operator (`references/operations.md`).
|
|
|
|
## A server restart killing in-flight runs
|
|
|
|
- **Symptom** Runs that were active when the server restarted are now `interrupted`,
|
|
with every step's `agent_active=false`.
|
|
- **Confirm** The status is `interrupted`, not `failed`, and there is **no automatic
|
|
replay**.
|
|
- **Do** Recovery is explicit. Inspect what completed; retry only when the server is
|
|
healthy and the previous attempt's cleanup has stopped. Never restart a server that
|
|
has runs in flight — quiesce first (`references/lanes.md`). Keep the interrupted run
|
|
as history; never describe a restart, retry or cancellation as rollback.
|
|
|
|
## Screenshot capture that silently produces non-images
|
|
|
|
- **Symptom** A run's "screenshot" artifact, or an image cited in an approval, is not
|
|
an image at all — for example 100+ KB of text beginning with a
|
|
`[Warning] Multiple displays were found` line — yet the capture command exited 0.
|
|
- **Confirm** Check the **bytes** (magic bytes or the reported content type), not the
|
|
`.png` name. A file that is not a real image is not evidence.
|
|
- **Do** Capture on the device to a file and pull it — never write `exec-out` or a
|
|
shell redirect straight to a `.png`; name the display explicitly when more than one
|
|
exists; validate the pulled bytes before citing them. A capture that fails
|
|
validation is deleted and the stage returns `blocked`; never cite it to support an
|
|
approval. Put the capture helper in the repository so every UI run reuses one path.
|
|
See the example in `references/delivery.md`.
|
|
|
|
## Duplicated work after a lane or chain restart
|
|
|
|
- **Symptom** Items already completed run again after a lane or chain script
|
|
restarts: the same brief is re-invoked, side effects repeat, or a commit is made
|
|
twice.
|
|
- **Confirm** Compare the lane's list against its completed-items record. A list alone
|
|
has no resume point.
|
|
- **Do** Persist a cursor and resume from it (`references/lanes.md`). Record an item
|
|
only once it has landed, and skip recorded items on startup. Never relaunch a
|
|
duplicate of a run that is still active — reattach with its request key instead.
|