Files
pipelines/internal/author/skill/references/debugging.md
T
alan 8c1eea53a9 Report a stale client and refuse an unreadable newer server
Exchange a build/protocol identity so an older CLI can no longer talk to a
newer native server silently:

- protocol carries ProtocolVersion and every server frame (including the
  unsolicited welcome) is stamped with the server build and protocol, so a
  client that never pings still learns the server identity.
- buildinfo.Compare classifies the relationship as InSync/Warn/Refuse and
  names both builds; the client Watch reports one warning per connection and
  returns a hard error when the server speaks a newer, uninterpretable
  protocol so the caller never acts on the frame.
- client.ReceiveNotice decodes the raw frame so an unmodeled top-level field
  produces a one-time "unrecognized field(s)" notice instead of being dropped
  silently (the case that hid the artifacts field).
- CLI routes warnings to stderr, keeps machine JSON stdout clean and fails
  non-zero on a refused mismatch; ping/connection print the client/server
  relationship. The TUI surfaces the same notices through the shared log.
- scripts/setup-native.py now warns when an older `pipelines` earlier on PATH
  shadows prefix/bin/pipelines, naming both paths and versions, without
  installing or modifying anything else.
- Docs, the packaged debugging skill reference and README no longer imply the
  installed CLI always matches the server.

Tests cover the comparison table, unknown-field detection and caching, the
server identity stamping on pong and welcome, the CLI/TUI reporting paths and
the installer PATH shadowing warning.
2026-10-16 18:37:01 +01:00

8.3 KiB

Debugging wedged and failed runs

Debug from retained evidence, never from an agent's summary. Open the run with CLI_PREFIX inspect RUN_ID --json and read the exact stage outputs, outcomes, diagnostics, file changes and artifacts. Each completed step also carries a verdict: the chosen option (outcome → target), a one-line reason with its provenance (model, fallback or absent), and the recorded artifacts the reason refers to. In the TUI, selecting a stage shows that verdict at the top of the stage's result, above its output and evidence. Treat the reason as a claim to check against the step's output, never as evidence in itself. Identify evidence-supported causes and say what remains unknown. Never fix a run's symptom by editing the project by hand: fix the pipeline definition or the Pipelines/harness service, or record the failure and continue.

Each playbook is symptom → how to confirm → what to do.

HARNESS TRANSPORT errors

  • Symptom A stage or run fails with the sanitized category HARNESS TRANSPORT; the model may have done work but no final result was read.
  • Confirm inspect --json shows the category on the failed stage. A request failure, an active-session poll failure and a final-readback failure all carry this one category.
  • Do Treat it as a transport failure, not a task failure. Distinguish a pre-admission failure (nothing executed) from a post-admission one (tools may have run). An explicit read-only stage may retry; a stage whose tools include edit/ shell, or that has no tools block, never replays admitted work. Do not re-run a full agent stage merely to recover a response's formatting. If it recurs, check the service and harness health (references/operations.md); do not weaken the stage to get past it.

Malformed model results

  • Symptom The category FINAL RESULT VALIDATION names a duplicate nested field or malformed/ambiguous JSON with a byte offset.
  • Confirm Read the diagnostic and the retained final result: the stage's result does not match its output contract. A retained transcript can be a valid bounded preview envelope, not the complete history.
  • Do Recovery is bounded and work-preserving. A deterministic local pass first appends missing closers, drops one trailing comma, or inserts the single comma/colon the parser expected; only if that is ambiguous does one deny-all formatter request run, with a deadline derived from the payload size and the attempt's remaining budget. Each path records a repaired, partial or unrecoverable status in format_repairs; a partial or unrecoverable payload is never accepted as complete. On an explicit read-only stage a malformed result may be re-emitted up to retries; a stage with edit/shell or no tools block is never replayed, though its malformed result can still be repaired by the deterministic pass or the fresh deny-all formatter session. Never accept prose, flattened output or a tool-call envelope as success. If one stage keeps producing invalid JSON, tighten its output/prompt in the pipeline definition (references/format.md) — a pipeline fix, not a project edit.

A reviewer or visual reviewer that blocks, rejects repeatedly, or hangs

  • Symptom The reviewer returns rejected on every visit, returns blocked, or the run stalls inside a reviewer stage.
  • Confirm Inspect every visit and read what the reviewer actually ran and saw. A rejection loop is bounded by the stage limits (max_visits); at the cap the rejection is retained and routes to #failure. blocked means the reviewer could not obtain the evidence it needed.
  • Do
    • A visual reviewer that returns blocked with no screenshots is right to refuse. Fix the capture path (below) rather than removing it.
    • Rejections that repeat the same finding mean either the implementer is not fixing it or the acceptance criteria contradict the brief. Read the retained findings, correct the brief or the pipeline's criteria, and re-run.
    • Never remove a review stage, relax an acceptance check, or approve the review yourself to get past it (references/orchestration.md).
    • A reviewer that hangs on an unbounded command (for example a filesystem-root scan) can leave a process in uninterruptible sleep and stall the server or Docker. Cancel the run, stop the offending command, and add a scope instruction to the brief or pipeline: bounded, relevant paths only.

Workspace-guard refusals

  • Symptom A run is refused with "run N is already active in workspace ...".
  • Confirm The message names the conflicting run and its workspace; the guard is working as designed — one active run per resolved workspace.
  • Do Wait for the conflicting run, or start this one in a different workspace (another worktree/checkout, which is what a lane is for — references/lanes.md). Do not delete the active run, the database or sessions to clear it. An orphan active session left by a restart with no owning active run still refuses new work until it is interrupted; reusing the active run's original request key reattaches idempotently.

A client older than the server dropping fields

  • Symptom inspect --json or ping --json is missing fields this skill describes, or a flag is rejected as unknown.
  • Confirm The client now reports this itself. A server response carrying a top-level field this client does not model prints a one-time server sent unrecognized top-level field(s) ... notice; a same-protocol build mismatch prints one warning: naming both builds on stderr; and a server speaking a newer protocol fails the command non-zero. ping --json shows the server build identity and version --json the client; ping/connection also print the client/server relationship. An older server omits build.
  • Do Treat it as a version mismatch, not a run failure. State both versions and the mismatch. Do not infer that a source fix or an updated skill is deployed. Use only flags the installed help shows; do not restart or reinstall a service to "fix" it without the operator (references/operations.md). Rebuild or reinstall the CLI so it matches the server; never hand-edit an installed binary.

A server restart killing in-flight runs

  • Symptom Runs that were active when the server restarted are now interrupted, with every step's agent_active=false.
  • Confirm The status is interrupted, not failed, and there is no automatic replay.
  • Do Recovery is explicit. Inspect what completed; retry only when the server is healthy and the previous attempt's cleanup has stopped. Never restart a server that has runs in flight — quiesce first (references/lanes.md). Keep the interrupted run as history; never describe a restart, retry or cancellation as rollback.

Screenshot capture that silently produces non-images

  • Symptom A run's "screenshot" artifact, or an image cited in an approval, is not an image at all — for example 100+ KB of text beginning with a [Warning] Multiple displays were found line — yet the capture command exited 0.
  • Confirm Check the bytes (magic bytes or the reported content type), not the .png name. A file that is not a real image is not evidence.
  • Do Capture on the device to a file and pull it — never write exec-out or a shell redirect straight to a .png; name the display explicitly when more than one exists; validate the pulled bytes before citing them. A capture that fails validation is deleted and the stage returns blocked; never cite it to support an approval. Put the capture helper in the repository so every UI run reuses one path. See the example in references/delivery.md.

Duplicated work after a lane or chain restart

  • Symptom Items already completed run again after a lane or chain script restarts: the same brief is re-invoked, side effects repeat, or a commit is made twice.
  • Confirm Compare the lane's list against its completed-items record. A list alone has no resume point.
  • Do Persist a cursor and resume from it (references/lanes.md). Record an item only once it has landed, and skip recorded items on startup. Never relaunch a duplicate of a run that is still active — reattach with its request key instead.