8c1eea53a9
Exchange a build/protocol identity so an older CLI can no longer talk to a newer native server silently: - protocol carries ProtocolVersion and every server frame (including the unsolicited welcome) is stamped with the server build and protocol, so a client that never pings still learns the server identity. - buildinfo.Compare classifies the relationship as InSync/Warn/Refuse and names both builds; the client Watch reports one warning per connection and returns a hard error when the server speaks a newer, uninterpretable protocol so the caller never acts on the frame. - client.ReceiveNotice decodes the raw frame so an unmodeled top-level field produces a one-time "unrecognized field(s)" notice instead of being dropped silently (the case that hid the artifacts field). - CLI routes warnings to stderr, keeps machine JSON stdout clean and fails non-zero on a refused mismatch; ping/connection print the client/server relationship. The TUI surfaces the same notices through the shared log. - scripts/setup-native.py now warns when an older `pipelines` earlier on PATH shadows prefix/bin/pipelines, naming both paths and versions, without installing or modifying anything else. - Docs, the packaged debugging skill reference and README no longer imply the installed CLI always matches the server. Tests cover the comparison table, unknown-field detection and caching, the server identity stamping on pong and welcome, the CLI/TUI reporting paths and the installer PATH shadowing warning.
138 lines
8.3 KiB
Markdown
138 lines
8.3 KiB
Markdown
# Debugging wedged and failed runs
|
|
|
|
Debug from **retained evidence**, never from an agent's summary. Open the run with
|
|
`CLI_PREFIX inspect RUN_ID --json` and read the exact stage outputs, outcomes,
|
|
diagnostics, file changes and artifacts. Each completed step also carries a
|
|
`verdict`: the chosen option (`outcome → target`), a one-line reason with its
|
|
provenance (`model`, `fallback` or `absent`), and the recorded artifacts the reason
|
|
refers to. In the TUI, selecting a stage shows that verdict at the top of the
|
|
stage's result, above its output and evidence. Treat the reason as a claim to check
|
|
against the step's output, never as evidence in itself. Identify evidence-supported
|
|
causes and say what remains unknown. Never fix a run's symptom by editing the
|
|
project by hand: fix the pipeline definition or the Pipelines/harness service, or
|
|
record the failure and continue.
|
|
|
|
Each playbook is *symptom → how to confirm → what to do*.
|
|
|
|
## HARNESS TRANSPORT errors
|
|
|
|
- **Symptom** A stage or run fails with the sanitized category `HARNESS TRANSPORT`;
|
|
the model may have done work but no final result was read.
|
|
- **Confirm** `inspect --json` shows the category on the failed stage. A request
|
|
failure, an active-session poll failure and a final-readback failure all carry this
|
|
one category.
|
|
- **Do** Treat it as a transport failure, not a task failure. Distinguish a
|
|
pre-admission failure (nothing executed) from a post-admission one (tools may have
|
|
run). An explicit read-only stage may retry; a stage whose tools include `edit`/
|
|
`shell`, or that has no tools block, **never replays admitted work**. Do not re-run
|
|
a full agent stage merely to recover a response's formatting. If it recurs, check
|
|
the service and harness health (`references/operations.md`); do not weaken the
|
|
stage to get past it.
|
|
|
|
## Malformed model results
|
|
|
|
- **Symptom** The category `FINAL RESULT VALIDATION` names a duplicate nested field
|
|
or malformed/ambiguous JSON with a byte offset.
|
|
- **Confirm** Read the diagnostic and the retained final result: the stage's result
|
|
does not match its `output` contract. A retained transcript can be a valid bounded
|
|
preview envelope, not the complete history.
|
|
- **Do** Recovery is bounded and work-preserving. A deterministic local pass first
|
|
appends missing closers, drops one trailing comma, or inserts the single
|
|
comma/colon the parser expected; only if that is ambiguous does one deny-all
|
|
formatter request run, with a deadline derived from the payload size and the
|
|
attempt's remaining budget. Each path records a `repaired`, `partial` or
|
|
`unrecoverable` status in `format_repairs`; a partial or unrecoverable payload
|
|
is never accepted as complete. On an explicit read-only stage a malformed result
|
|
may be re-emitted up to `retries`; a stage with `edit`/`shell` or no tools block
|
|
is never replayed, though its malformed result can still be repaired by the
|
|
deterministic pass or the fresh deny-all formatter session. Never accept prose,
|
|
flattened output or a tool-call envelope as success. If one stage keeps
|
|
producing invalid JSON, tighten its `output`/`prompt` in the **pipeline
|
|
definition** (`references/format.md`) — a pipeline fix, not a project edit.
|
|
|
|
## A reviewer or visual reviewer that blocks, rejects repeatedly, or hangs
|
|
|
|
- **Symptom** The reviewer returns `rejected` on every visit, returns `blocked`, or
|
|
the run stalls inside a reviewer stage.
|
|
- **Confirm** Inspect every visit and read what the reviewer actually ran and saw. A
|
|
rejection loop is bounded by the stage `limits` (`max_visits`); at the cap the
|
|
rejection is retained and routes to `#failure`. `blocked` means the reviewer could
|
|
not obtain the evidence it needed.
|
|
- **Do**
|
|
- A visual reviewer that returns `blocked` with no screenshots is right to refuse.
|
|
Fix the capture path (below) rather than removing it.
|
|
- Rejections that repeat the same finding mean either the implementer is not fixing
|
|
it or the acceptance criteria contradict the brief. Read the retained findings,
|
|
correct the brief or the pipeline's criteria, and re-run.
|
|
- **Never** remove a review stage, relax an acceptance check, or approve the review
|
|
yourself to get past it (`references/orchestration.md`).
|
|
- A reviewer that hangs on an unbounded command (for example a filesystem-root
|
|
scan) can leave a process in uninterruptible sleep and stall the server or
|
|
Docker. Cancel the run, stop the offending command, and add a scope instruction
|
|
to the brief or pipeline: bounded, relevant paths only.
|
|
|
|
## Workspace-guard refusals
|
|
|
|
- **Symptom** A run is refused with "run N is already active in workspace ...".
|
|
- **Confirm** The message names the conflicting run and its workspace; the guard is
|
|
working as designed — one active run per resolved workspace.
|
|
- **Do** Wait for the conflicting run, or start this one in a different workspace
|
|
(another worktree/checkout, which is what a lane is for — `references/lanes.md`).
|
|
Do not delete the active run, the database or sessions to clear it. An **orphan**
|
|
active session left by a restart with no owning active run still refuses new work
|
|
until it is interrupted; reusing the active run's original request key reattaches
|
|
idempotently.
|
|
|
|
## A client older than the server dropping fields
|
|
|
|
- **Symptom** `inspect --json` or `ping --json` is missing fields this skill
|
|
describes, or a flag is rejected as unknown.
|
|
- **Confirm** The client now reports this itself. A server response carrying a
|
|
top-level field this client does not model prints a one-time
|
|
`server sent unrecognized top-level field(s) ...` notice; a same-protocol build
|
|
mismatch prints one `warning:` naming both builds on stderr; and a server
|
|
speaking a newer protocol fails the command non-zero. `ping --json` shows the
|
|
server build identity and `version --json` the client; `ping`/`connection`
|
|
also print the client/server relationship. An older server omits `build`.
|
|
- **Do** Treat it as a version mismatch, not a run failure. State both versions and
|
|
the mismatch. **Do not infer** that a source fix or an updated skill is deployed.
|
|
Use only flags the installed help shows; do not restart or reinstall a service to
|
|
"fix" it without the operator (`references/operations.md`). Rebuild or reinstall
|
|
the CLI so it matches the server; never hand-edit an installed binary.
|
|
|
|
## A server restart killing in-flight runs
|
|
|
|
- **Symptom** Runs that were active when the server restarted are now `interrupted`,
|
|
with every step's `agent_active=false`.
|
|
- **Confirm** The status is `interrupted`, not `failed`, and there is **no automatic
|
|
replay**.
|
|
- **Do** Recovery is explicit. Inspect what completed; retry only when the server is
|
|
healthy and the previous attempt's cleanup has stopped. Never restart a server that
|
|
has runs in flight — quiesce first (`references/lanes.md`). Keep the interrupted run
|
|
as history; never describe a restart, retry or cancellation as rollback.
|
|
|
|
## Screenshot capture that silently produces non-images
|
|
|
|
- **Symptom** A run's "screenshot" artifact, or an image cited in an approval, is not
|
|
an image at all — for example 100+ KB of text beginning with a
|
|
`[Warning] Multiple displays were found` line — yet the capture command exited 0.
|
|
- **Confirm** Check the **bytes** (magic bytes or the reported content type), not the
|
|
`.png` name. A file that is not a real image is not evidence.
|
|
- **Do** Capture on the device to a file and pull it — never write `exec-out` or a
|
|
shell redirect straight to a `.png`; name the display explicitly when more than one
|
|
exists; validate the pulled bytes before citing them. A capture that fails
|
|
validation is deleted and the stage returns `blocked`; never cite it to support an
|
|
approval. Put the capture helper in the repository so every UI run reuses one path.
|
|
See the example in `references/delivery.md`.
|
|
|
|
## Duplicated work after a lane or chain restart
|
|
|
|
- **Symptom** Items already completed run again after a lane or chain script
|
|
restarts: the same brief is re-invoked, side effects repeat, or a commit is made
|
|
twice.
|
|
- **Confirm** Compare the lane's list against its completed-items record. A list alone
|
|
has no resume point.
|
|
- **Do** Persist a cursor and resume from it (`references/lanes.md`). Record an item
|
|
only once it has landed, and skip recorded items on startup. Never relaunch a
|
|
duplicate of a run that is still active — reattach with its request key instead.
|