e37b864c40
Each model visit now gets <run>/stages/<seq>-<stage>/ with input.md, memory.md, output.md, private/ and a shared run-level public/. The model reads the two input files, follows the stage's instructions verbatim and returns complete markdown; only the # outcome section is parsed. - add internal/turn layout, render and output parsing - write input.md/memory.md before a visit and output.md from the final message; inventory all turn files as stage artifacts - accumulate completed turns and system-written public file additions in memory, including across retries and resume - remove the JSON envelope, strict field validation, line format and the repair ladder for model stages - republish pipeline authoring prompts and docs in the new style Checks: go build ./..., go vet ./..., go test -count=1 ./..., gofmt -l .
610 lines
39 KiB
Markdown
610 lines
39 KiB
Markdown
# Verification, accounting and real-provider evidence
|
||
|
||
Historical note: model stages now use file-based turns and parse only the
|
||
`# outcome` section. The JSON/line-format envelope and format-repair ladder
|
||
described in the dated sections below are retired; see `docs/model-results.md`
|
||
for the current contract. The dated evidence remains as a record.
|
||
|
||
Structural validation is not inference, tool execution, correct task completion,
|
||
phone delivery or correct billing. Distinguish observed artifacts/command results
|
||
from model-written reports. State the command/check actually run and say **not
|
||
tested** for anything not executed. Never present mocked results as live-provider
|
||
evidence or a different backend/model as already validated.
|
||
|
||
## Authoring useful verification
|
||
|
||
Preserve the facts later stages need in free-form output headings; only the
|
||
`# outcome` section is interpreted. Implementer/reviewer stages
|
||
can use appropriate permitted commands; do not accept a preceding stage's success
|
||
summary as independent proof. Actual file bytes, command exit status, retained tool
|
||
parts and server-captured diffs are stronger evidence than prose. A read failure
|
||
can prove a deliberately missing input, but is not a successful read. Tool names
|
||
mentioned in text do not prove tools executed. A reviewer stage must inspect the
|
||
actual files and run its own checks; a source-only review runs no commands and
|
||
must state that runtime behavior was not tested. Structural, mocked or
|
||
authoring-time checks are not deployed or live-provider evidence.
|
||
|
||
Do not add technology-specific checks unless the task/repository requires them.
|
||
Keep runtime verification separate from authoring. Execute relevant code and fixture
|
||
checks within the authorized task; ask before unrelated chargeable calls or notifications.
|
||
Do not weaken an acceptance check just to get a passing run.
|
||
`provider-check` must run on the intended execution computer/container with its
|
||
credentials; a client-side check is not proof of a remote server's provider health.
|
||
|
||
## Budgets and backend caveats
|
||
|
||
Defaults remain two retries and one hour per stage attempt; a run-level wall-clock
|
||
deadline is not added. Budgets are optional, positive and independent USD/token
|
||
reported-usage limits shared across attempts and repeat visits.
|
||
Accounting failures stop budget-limited stages. In-flight calls can overshoot;
|
||
these are **not provider-side hard caps or a guaranteed bill ceiling**. Reported
|
||
USD may validly be zero; do not fabricate prices/accounting from output length.
|
||
|
||
Per-run usage is recorded as typed step evidence from the usage the stage already
|
||
measured for its budget (no second harness read and no transcript parse), and
|
||
totalled for the run and over all history. A stage revisited within a run shares
|
||
one budget, so each step records only that visit's increment of it and the run
|
||
total is not the cumulative stage budget counted again; a revisit that reports
|
||
nothing new (a failure or timeout before any usage poll, or a Pi visit that never
|
||
reaches a usage stat) records no usage for that visit rather than a known zero.
|
||
`CLI_PREFIX inspect RUN_ID --json`
|
||
exposes each model step's `tokens_total`, `tokens_in`/`tokens_out` (only when
|
||
`split_known`), `cost_usd` (only when `cost_known`) and the model it ran with,
|
||
plus the totalled `usage`; `CLI_PREFIX stats --json` adds the history-wide totals,
|
||
which equal the sum of the per-run totals. A stage that reported no cost exposes
|
||
its tokens and marks the cost unavailable rather than showing zero; a harness that
|
||
reports only a total (Pi) exposes the total and marks the in/out split
|
||
unavailable rather than presenting the total as input tokens. An upgraded or
|
||
historical database gains an empty usage table with no backfill, so old runs omit
|
||
`usage` instead of showing zero. This is current source, not deployment evidence.
|
||
|
||
Dated current-source evidence (2026-10-17, this machine). A throwaway read-only
|
||
reader (`internal` store API) over a scratch copy of
|
||
`~/.local/share/pipelines/data/pipelines.db`, taken with
|
||
`sqlite3 "file:$HOME/.local/share/pipelines/data/pipelines.db?mode=ro" ".backup <scratch>"`
|
||
so the live DB is never mutated, opened 228 runs and found **no** `step_usage`
|
||
rows: every stored run predates this change, so per-run figures are unavailable
|
||
rather than zero. Run 314 (this item's own run) and run 239 (whose implementer
|
||
step ran `opencode-go/kimi-k2.7-code`, a stage model different from the pinned
|
||
`opencode-go/deepseek-v4.1-flash`) both report `usage=(none recorded)`, and
|
||
`stats` omits the history-wide totals. The recorded figures themselves are
|
||
covered by the passing current-source round trips — `internal/pipeline/usage_test.go`,
|
||
`internal/server/usage_test.go` (including `inspect --json` over a websocket) and
|
||
the pre-upgrade/read-bounded/stats-sum cases — not by a live run, because the
|
||
deployed server predates this change and was not restarted. No live model call,
|
||
deploy or rebuild produced this evidence.
|
||
|
||
A live V2 trial found `/generate` returned text while its retained session had no
|
||
messages and zero counters. Current source uses tracked `/prompt` → bounded
|
||
active-session polling → message readback for **budgeted V2 text-only calls, with
|
||
all tools denied**. Completed token-budgeted responses reporting zero tokens fail
|
||
closed. No new definition flags or agent tools are required for this fix.
|
||
Unbudgeted tool-free V2 generation can still have empty backend message history;
|
||
pipeline inputs, results and command audit remain retained. An unbudgeted V2
|
||
audited agent stage makes exactly one extra bounded final session-aggregate read
|
||
to record any reported USD/token usage best-effort: missing or invalid usage stays
|
||
unknown and never turns a valid run into a failure, cumulative sessions/retries
|
||
are keyed per session so they cannot double count, and totals are never inferred
|
||
from final-assistant-only or native-partial data. Budgeted settlement and native
|
||
V1 enforcement are unchanged. Native V1 uses its existing path.
|
||
|
||
The V2 `.../experimental/session/{id}/wait` request is **no longer used**: a dated
|
||
Docker log shows it timing out at the CLI's own request timeout after about six
|
||
minutes (run started 2026-10-10T23:07:16.547Z, `TimeoutError` logged at
|
||
2026-10-10T23:13:17.262Z, server `http.status=499`), which aborted a long
|
||
implementation stage. Instead current source polls `GET /api/session/active`
|
||
about once per second with a context-aware delay and no busy loop, until the
|
||
admitted session is absent (inactive). Missing/null/non-object `data`, or a state
|
||
other than the documented `running` type, fails closed and is never treated as
|
||
inactive. This is **mocked regression coverage** in `internal/stage` (fake CLI),
|
||
not live-provider evidence; V1 native settlement is unchanged.
|
||
|
||
Current source reads the V2 final result with a single bounded descending
|
||
final-assistant readback (`order=desc&limit=1&type=assistant`) after the settled
|
||
poll, for both audited agent stages and budgeted V2 text-only calls, so large
|
||
retained history (including prior tool-read contents) is no longer fetched just
|
||
to select the completed result. A truncated/incomplete readback, an empty data
|
||
list, or a latest message that is neither completed `stop` nor a completed
|
||
structured tool-call fails closed with a sanitized diagnostic that omits
|
||
provider content; retained audit previews are **not** executable final results.
|
||
Session/debug evidence now reuses the same single latest-assistant readback
|
||
instead of a recent-20 all-message fetch, and returns each payload as a labelled
|
||
bounded, credential-redacted envelope with explicit preview/truncation/missing
|
||
diagnostics; a small final assistant result is therefore selected even when large
|
||
prior tool content exists. Pi evidence remains documented JSONL and is never
|
||
parsed as one JSON object. This describes current source behavior, not a guarantee
|
||
that a user's installed server includes the fix. An updated skill/client is not
|
||
deployment evidence; never rebuild/restart services or log into providers without
|
||
specific authorization.
|
||
|
||
Retained model-stage failures now carry a fixed sanitized category. A strictly
|
||
invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a
|
||
byte offset) is labeled `FINAL RESULT VALIDATION`; request, V2 active-session poll
|
||
and final-readback failures are labeled `HARNESS TRANSPORT`. A pre-admission
|
||
request failure replays nothing and is retried for any stage; a post-admission
|
||
failure is retried only on an explicit read-only stage (see the 2026-10-17 section
|
||
below). A validation failure
|
||
preserves the admitted session, transcript and evidence and performs no second tool
|
||
execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history
|
||
readback of exactly 262144 bytes of incomplete JSON (`invalid agent messages`), a
|
||
reviewer result with a duplicate nested `performance_review` field, and an
|
||
onboarding visual-review result with `invalid character 'z' after object key:value
|
||
pair at byte 3968` where unsafe lossless repair was rejected. These are **mocked
|
||
`internal/stage` fixtures**: executable source-level evidence, not live-provider
|
||
evidence and not a deployment. OpenCode V2 prose `output_contract` does not enforce
|
||
structured output; V2 results are validated locally and no unsupported V2 format
|
||
field is sent. A review-only fresh execution is a future recovery idea, **not
|
||
implemented**. Onboarding UI visual approval remains unresolved and must not be
|
||
reported complete.
|
||
|
||
## Bounded malformed-result recovery: 2026-10-16
|
||
|
||
Source-level, mocked `internal/stage` regression coverage (not live-provider
|
||
evidence and not a deployment): malformed final-result recovery now tries a
|
||
deterministic local pass first (append missing closers, drop one trailing comma,
|
||
or insert the single comma/colon the parser expected), then at most one deny-all
|
||
formatter request whose deadline is derived from the payload size, the stage
|
||
timeout and the attempt's remaining budget rather than a fixed 30-second
|
||
constant. Every path
|
||
records `repaired`, `partial` or `unrecoverable` in `format_repairs`; a `partial`
|
||
or `unrecoverable` payload is never returned as a completed result, and an
|
||
ambiguous payload (an unescaped quote/colon inside a string value) is never
|
||
guessed at. A shell-enabled stage's malformed but deterministically recoverable
|
||
result now completes without replaying admitted tool work; `safeToReplay` is
|
||
unchanged. The run-55 `ui_reviewer` payload was retrieved read-only with
|
||
`pipelines inspect 55 --json` and reproduced its exact byte offset 7837; it is the
|
||
ambiguous error class, so it is reported unrecoverable rather than repaired. This
|
||
does not update or restart any running server.
|
||
|
||
## Harness transport reach retry and repair deadline: 2026-10-17
|
||
|
||
Source-level, mocked `internal/stage`, `internal/opencode` and `internal/server`
|
||
regression coverage (not live-provider evidence and not a deployment): the
|
||
harness transport path now distinguishes a pre-admission reach failure
|
||
(connecting or creating the session) from a post-admission readback/poll/agent
|
||
failure under the one `HARNESS TRANSPORT` category. A pre-admission reach failure
|
||
replays nothing, so it is retried within `retries` for any stage with a fresh
|
||
session and no replayed prompt; a post-admission failure is retried only on an
|
||
explicit read-only stage, and an `edit`/`shell`/legacy stage still never replays
|
||
admitted work. A context that expires during a request keeps both the sanitized
|
||
transport detail (operation/status/exit) and
|
||
`errors.Is(err, context.DeadlineExceeded)`; a bare attempt-boundary deadline is
|
||
not treated as a reach failure and stays terminal for a mutating stage. An
|
||
unrecognized provider execution-error name is reported as a bounded, sanitized
|
||
identifier plus any HTTP status. The isolated formatter deadline is now
|
||
proportional to the stage timeout (floored, capped, and clamped to
|
||
`remaining - margin` so it can never extend the stage/attempt deadline). These are
|
||
mocked fixtures; a deployed server is not updated by this change.
|
||
|
||
## Failure-handling source changes: 2026-10-15
|
||
|
||
Source-level, mocked `internal/stage` and `internal/server` regression coverage
|
||
(not live-provider evidence): an explicit read-only tools stage now actually
|
||
retries a post-admission stage deadline or harness transport failure, creating a
|
||
fresh bounded session per attempt, while a stage with `edit`/`shell` or no tools
|
||
block still never replays admitted work; a malformed final result is re-emitted
|
||
up to `retries` only on such a read-only stage with its `format_repairs` evidence
|
||
retained. A timed-out agent stage settles knowable V2 usage and retains one
|
||
bounded, redacted latest-assistant readback plus a `retained_session` pointer in
|
||
its stored transcript. A declared outcome routed to `#failure` with no step
|
||
diagnostic now yields a run diagnostic like `stage <key> returned outcome
|
||
"blocked"`; an existing step diagnostic is still used verbatim.
|
||
|
||
## Dated source-level coverage: 2026-10-11
|
||
|
||
Source-level/mocked regression coverage (not live-provider evidence):
|
||
`resolve-releases` now uses a single bounded
|
||
`GET .../git/matching-refs/tags/v2` response that the GitHub REST docs describe as
|
||
complete with no pagination; offline fixtures cover a 100+ ref array,
|
||
`MAX_REFS`/`MAX_BODY` caps, ignored prerelease/build tags, malformed refs and a
|
||
missing newest-tag digest failing closed with no older fallback, with no
|
||
`tags/list` cross-check. Picker tests cover blink/index preservation,
|
||
ANSI-readable selection without a colliding background, and
|
||
40x16/80x24/120x36 height budgets. The compiled picker PTY check (opt-in via
|
||
`PIPELINES_HARNESS_PTY_DIR`) waits through blink intervals and asserts exact
|
||
saved/arrow-scrolled models. `setup-docker.py` accepts the actual
|
||
`opencode v2.0.26` probe token and compares source identity only on a full
|
||
install; model/login-only reconfiguration compares the verified release image
|
||
identity. Python fixtures bind temporary HOME directories so the suite passes
|
||
under the container default `/home/pipelines` without weakening mount guards.
|
||
The Docker authoring transport is also covered at source level: the shared
|
||
prefix picks `/usr/local/bin/pipelines` for a Docker-linked profile. Loopback is
|
||
recognized only via `net.ParseIP(...).IsLoopback()` and an exact case-insensitive
|
||
`localhost`, so a remote DNS name such as `127.remote.example` stays remote with
|
||
no probe. A single bounded `docker compose port` probe parses complete
|
||
`host:port` bindings, and scoped source/target aliases are embedded only when a
|
||
binding covers the requested loopback address and port; malformed or non-matching
|
||
bindings and a loopback `wss` endpoint fail closed, and `client.Dial` rewrites
|
||
only an exact source match. `provider-check` dispatches through a Docker-linked
|
||
profile's existing service with `-T`, the original cwd/model/timeout, exact
|
||
stdout/exit propagation and `--json` passed only in JSON mode (fixture tests, no
|
||
real provider), while explicit native profiles keep the host generator. The
|
||
picker uses a true `x/term` TTY check so `/dev/null` and pipes fail fast, with a
|
||
short context deadline so a regression fails rather than hangs. Real Docker
|
||
rebuild/reconfigure against live registries, a real
|
||
40x16/80x24/120x36 terminal capture, and real model inference remain **not tested**
|
||
in this change.
|
||
|
||
### 2026-10-11 Pi sign-in handoff (source + Root PTY observation)
|
||
|
||
Root separately exercised current-source `pipelines harness setup pi --login`
|
||
under a real Docker PTY with copied harness preferences. Root observed that
|
||
Pi 1.1.0 defaults to fullscreen and clears the Pipelines login instructions, so
|
||
the user sees a coding interface without the `/login` guidance, and that
|
||
`/login` first shows "Select authentication method" (account / API key / Radius)
|
||
rather than a provider picker. The same handler run with a temporary executable
|
||
wrapper passing `pi --tui-mode regular` retained the Pipelines header and
|
||
instructions above Pi's interface. Installed `pi --help` documents the supported
|
||
`--tui-mode regular` flag and the fullscreen default; `pi auth --help` confirms
|
||
there is no standalone login (only print/check, never run). These are source/root
|
||
observations with copied preferences: **no provider was selected, no key was
|
||
entered and no inference occurred**, and this source is **not deployed** and is
|
||
not live-provider evidence. A first reviewer capture at 80x30 found the longer
|
||
handoff plus Pi's own banner scrolled the Pipelines header off the top, so the
|
||
copy was shortened to a title plus three short lines (existing-credential reuse
|
||
folded into the `/login` line, with the account/API key/Radius method choice and
|
||
`ctrl+c`/`ctrl+d` exit keys kept) and the redundant trailing blank removed.
|
||
Independent PTY re-capture at 80x30/100x36 after this shortening is **not
|
||
tested** in this change. The matching exact login-argv and copy assertions are
|
||
covered by fixture tests in `internal/cli`/`internal/harness`.
|
||
|
||
### 2026-10-11 server CLI fixture isolation (source-level + Root-reported run)
|
||
|
||
Root-reported runtime evidence supplied for this change (not re-executed by the
|
||
authoring stages): the deterministic seeded-profile regression
|
||
`go test ./tmp/preview-verification -run TestServerCLIFixtureDoesNotInheritUserDockerProfile -count=1`
|
||
seeds a synthetic Docker-linked harness profile in a child
|
||
`go test ./internal/server -run '^TestCLIHeadlessRunInspectionCancellationAndRPC$'`.
|
||
It previously failed because the child inherited the caller's real
|
||
`PIPELINES_HARNESS_CONFIG`, so the CLI `debug` prefix failed closed
|
||
("refusing to redirect") against the test's ephemeral httptest endpoint. Root's
|
||
four independent preview tests for the bounded transcript helpers pass; Root's
|
||
full host race run produced no race warning and only this fixture failure, and
|
||
`go vet ./...` passed. Current source therefore isolates the server CLI fixture
|
||
with the same missing-temp-file
|
||
`t.Setenv("PIPELINES_HARNESS_CONFIG", filepath.Join(t.TempDir(), "harnesses.json"))`
|
||
pattern already used by `internal/cli`'s setup, so the host executable stays
|
||
authoritative and no Docker probe runs for an ephemeral test endpoint. The
|
||
production fail-closed redirect guard in `internal/harness/prefix.go` and its
|
||
`refusing to redirect` / loopback-wss regressions are unchanged; the failing
|
||
verification was fixed rather than removed.
|
||
|
||
This is a current-source and Root-reported observation. At the time of that
|
||
observation, independent re-execution of the full `go build`/`go vet`/`go test`
|
||
matrix (including Docker-only race results) and the final installed-host Docker
|
||
verification were **pending**; they are now recorded in the 2026-10-11 FINAL
|
||
INSTALLED section below, so this phase-specific wording is superseded by that dated
|
||
install. The packaged reference describes source behavior; a source fix or updated
|
||
skill is not deployment evidence.
|
||
|
||
### 2026-10-11 retained run11 reviewer re-capture (source-level, not a final install)
|
||
|
||
Independently reading the retained run11 verification at
|
||
`tmp/maintenance-run11-verification.json` records the source-level review
|
||
sequence: an earlier reviewer PTY capture rejected the longer handoff because at
|
||
80x30 it scrolled the Pipelines header off the top; the implementer then
|
||
shortened `AuthHandoff` to a title plus three short lines; and the re-run
|
||
`reviewer` and `ui_reviewer` stages approved from real current-source Pipelines
|
||
PTY captures at 80x30 and 100x36 that showed the header and handoff visible above
|
||
installed Pi 1.1.0's regular interface. A labelled controlled fixture that
|
||
stripped `--tui-mode regular` reproduced the pre-fix fullscreen clear, and the
|
||
review stages report no provider selected, no key entered and no inference. This
|
||
is source-level reviewer evidence, read from retained output; it was not a
|
||
deployed/installed state at that time, and the final full Docker install
|
||
verification was then **pending** — it is now recorded in the 2026-10-11 FINAL
|
||
INSTALLED section below, so this phase-specific wording is superseded by that dated
|
||
install.
|
||
|
||
Root-reported actual runs (source-level observations supplied by Root for this
|
||
change and not re-verified here): run7 showed both configured providers' exact
|
||
handoff markers; runs6/9/10 exercised live maintenance with final-assistant
|
||
polling/readback succeeding. These are Root observations, not mocked-regression
|
||
coverage and not a completed installation; no final deployed/installed state is
|
||
claimed.
|
||
|
||
### 2026-10-11 run13 interruption, Docker race limitation and state separation
|
||
|
||
Four states are kept explicitly separate and must not be conflated:
|
||
|
||
1. **Current source** (this checkout, not deployed): bounded 32 KiB
|
||
(`transcriptBound = 32<<10`) redacted transcript previews produce a valid JSON
|
||
envelope with redaction applied before truncation and a rune-safe 8 KiB preview;
|
||
session/debug evidence is scoped `latest_bounded_evidence` with the single
|
||
bounded latest-assistant readback and at most the 3 most recent session-bearing
|
||
steps; an unbudgeted V2 audited agent stage records reported usage best-effort
|
||
through exactly one 2s-bounded final session-aggregate read, leaving
|
||
missing/invalid usage unknown; and the isolated server CLI fixture uses the
|
||
missing-temp-file `PIPELINES_HARNESS_CONFIG` pattern while the production
|
||
fail-closed redirect guard in `internal/harness/prefix.go` is unchanged. Pi JSONL
|
||
is never parsed as one JSON object; native V1 keeps its existing path.
|
||
2. **Host verification evidence independently recorded by Root**: after the fixture
|
||
fix, Root ran the full `go test -race -count=1 ./...` and `go vet ./...`; after
|
||
three harness-neutral evidence labels and Step wording changes Root re-ran the
|
||
affected `tui`/`author`/`cli` race and vet checks, the Python suite (80) and the
|
||
opt-in compiled picker PTY check (4). These are Root-recorded host results, not
|
||
re-executed by the authoring stages.
|
||
3. **Deployed old image**: the running Docker image predates these source changes
|
||
and does not include them.
|
||
4. **Final install (completed 2026-10-11)**: the final install and installed
|
||
authoring/provider verification were **pending** when this separation was
|
||
written and are now recorded in the 2026-10-11 FINAL INSTALLED section below; an
|
||
updated skill/client or source checkout is still not deployment evidence on its
|
||
own.
|
||
|
||
Docker race limitation (confirmed): the Docker runtime is built with
|
||
`CGO_ENABLED=0` and has **no C compiler** (the Alpine runtime stage installs no
|
||
GCC/musl-dev), so `go test -race` cannot run there. In-Docker reviewers must use
|
||
ordinary `go build`/`go test`/`go vet` and rely on independently executed host
|
||
`-race` runs. Race checks must not be attempted in Docker and no compiler may be
|
||
installed or searched for.
|
||
|
||
Run13 interruption (dated evidence, not a fix): the previous reviewer ran `find /`
|
||
looking for GCC; the tool timeout left a process in disk sleep, Docker could not
|
||
stop the container, and an approved Desktop restart recovered run13 as interrupted
|
||
(`executing=false`, every step `agent_active=false`) with **no automatic replay**.
|
||
This is an environment/interruption observation only: no general cancellation fix
|
||
is inferred and process termination is unchanged. Docker startup recovery already
|
||
sets running steps to `interrupted` with `agent_active=0`.
|
||
|
||
### 2026-10-11 short-lived CLI inherited-pipe cancellation (current source)
|
||
|
||
Current source (not deployed) adds one shared, dependency-free constructor in
|
||
`internal/opencode`: `CommandContext(ctx, executable, args...)` sets the single
|
||
finite `os/exec.Cmd.WaitDelay` constant `ShortLivedWaitDelay = 250ms` and is the
|
||
only place that value is defined. It is reused by the `internal/stage`
|
||
`OpenCodeClient.call()` API/agent path (including `Interrupt`), `opencode.Detect()`
|
||
`--help` and the native `Connection.Messages()` export fallback. It bounds
|
||
**draining of inherited output pipes only**: after the child exits os/exec waits at
|
||
most the grace for descendants holding the inherited stdout/stderr pipes, then
|
||
closes them and reports `exec.ErrWaitDelay`; it does not terminate or confirm
|
||
cleanup of descendants.
|
||
|
||
`context.Canceled`/`context.DeadlineExceeded` identities, `exec.ErrWaitDelay` on
|
||
the sanitized errors, the ≤64 KiB request / ≤1 MiB response / 8 KiB stderr bounds,
|
||
`boundedBuffer`/`boundedOutput` drain-with-overflow behavior, redaction, strict
|
||
result validation and cleanup-unconfirmed active ownership (`ErrStageCleanup`, no
|
||
replay) are unchanged; no new module dependency is added and a parent that exits
|
||
successfully while a descendant still holds partial stdout is never treated as
|
||
success.
|
||
|
||
Dated synthetic reproduction (`tmp/interrupt-verification`, synthetic
|
||
real-subprocess diagnostics, **not provider evidence**): with Root waiting for the
|
||
descendant itself to start before cancelling public
|
||
`OpenCodeClient{Backend:"v2"}.Interrupt`, two Docker/Linux returns took
|
||
3996/3995ms despite `context.Canceled` (`descendant-started.jsonl`); the stronger
|
||
host reproduction took 4003/4000ms (`reproduction.log`). Added source-level
|
||
regressions use real `os/exec` fixtures with start-marker barriers, bounded
|
||
watchdogs and naturally terminating children for cancel-after-descendant,
|
||
deadline identity, descendant-held pipes → `ErrWaitDelay`, ordinary
|
||
output/exit/redaction, failed-interrupt ownership/no replay, plus `Detect`/export
|
||
wait-delay and cancellation cases. This is **current source, not deployed**:
|
||
descendant termination, confirmed remote interruption and any run13 Docker disk
|
||
sleep fix are **not claimed**, and Docker race remains unavailable (no C
|
||
compiler).
|
||
|
||
### 2026-10-11 FINAL INSTALLED verification (friend-ready preview)
|
||
|
||
This is the dated receipt for the installed runtime, recorded by Root from
|
||
`tmp/friend-final-verification/summary.json`; it supersedes the earlier
|
||
"pending final install" statements above, which stay as phase-specific history.
|
||
A normal interactive `./install --model opencode-go/deepseek-v4.1-flash
|
||
--harness-model opencode-go/deepseek-v4.1-flash` exited **0**, resolving the current
|
||
releases live and building fresh dirty source into actual image
|
||
`sha256:1810ab50f3ea3991e7f3548c79250e65a114bb1821eac52efd872b91cf7c3136`
|
||
(OpenCode 2.0.26 / Pi 1.1.0 / Node 24.18.1) with real tool-free provider
|
||
validation, ntfy authenticated publish, server + host health, home/workspace entry,
|
||
preserved authoritative endpoint and `ok` pre-install database/home backup
|
||
integrity. Post-install preservation all passed (original definitions 1–4, history
|
||
1–16, host/container default Pi and both models/profiles, logical endpoint,
|
||
service URLs/timezone/ports/models, Pi credential hash, OpenCode connected accounts,
|
||
ntfy token); neither original-repo commit/push nor credential/data cleanup ran.
|
||
Installed host-CLI authoring with **both** Docker harnesses (separate retained
|
||
drafts, each changing only the worker prompt) performed real `ping` → `pong` and
|
||
`validate` → `validated` through the generated Linux `/usr/local/bin/pipelines` and
|
||
scoped transport aliases preserving the logical `8081` endpoint and draft metadata,
|
||
independently inspected from native OpenCode tool outputs and retained Pi JSONL
|
||
`toolResults`, with no draft submit/run/publication. Docker-proxy submit on
|
||
verification-only definition 5 published v2, a second stale v1 draft was rejected as
|
||
conflict, and **Run17** succeeded in 7 seconds with the exact
|
||
`pipelines-docker-verified` markers in OpenCode then Pi, both inactive and exactly
|
||
one completed marker read verified from retained native history (not the
|
||
latest-assistant preview); best-effort unbudgeted usage was OpenCode 4011
|
||
tokens/$0.00029769599999999996 and Pi 1071 tokens/$0.000142002, and
|
||
`/api/session/active` was empty after exit. Installed-UI captures
|
||
(`installed-pi-*`, `pi-signin-*`, `installed-pi-models-*`) cover the actual run17
|
||
80x30/120x36 neutral-Pi/green-both-stage renders, the Pi auth handoff/method menu
|
||
viewed/canceled/exit0 with no sign-in, and the model wizard using copied
|
||
preferences only while production preferences stayed preserved. Root's independent
|
||
checks passed: full host `go test -race -count=1 ./...` and `go vet ./...`,
|
||
Python 80 tests (1 optional PTY skipped), Node policy, isolated developer lifecycle
|
||
fixture (no inference/deploy) and author package tests; earlier independent
|
||
reviewers ran ordinary Docker build/vet/tests and focused subprocess regressions
|
||
(no Docker race, no GCC search, no global `find`). **UNVERIFIED** here: fresh
|
||
OAuth/provider login completion, phone delivery, other real providers, the
|
||
D-disk-sleep fix and active lease expiry; authentication menus and
|
||
existing-credential inference are not fresh sign-in proof. The stronger synthetic
|
||
PUBLIC `Interrupt` child-start barrier (host 252/252 ms, Docker 250/251 ms, context
|
||
canceled true, bounded true vs ~4 s before) is not proof of descendant termination,
|
||
remote cleanup or a Docker VM disk fix. Archive extraction, build-identity and
|
||
compile verification remain pending until Root executes them.
|
||
|
||
### 2026-10-11 generated release receipt ignore (source-hygiene regression)
|
||
|
||
Current source adds two root-anchored `.gitignore` rules for the installer's
|
||
machine-local generated artifacts: `/.pipelines-releases.json` (the receipt path
|
||
from `setup-docker.py` `release_receipt_path()`) and `/.pipelines-releases-*` (the
|
||
`tempfile.mkstemp` `prefix=".pipelines-releases-"` in `record_release_receipt()`);
|
||
no installer/runtime code changed and the pre-existing receipt and other files are
|
||
retained. A new `scripts/test_build_metadata.py` case is a controlled, isolated Git
|
||
fixture using the repository's actual `.gitignore`: a clean tracked source plus a
|
||
representative generated receipt/`.pipelines-releases-*` temporary file leave `git
|
||
status --porcelain` empty and `build_metadata.docker_args` at
|
||
`PIPELINES_DIRTY=false` with no `-dirty` version; editing the tracked source then
|
||
makes the status non-empty and `docker_args` report `PIPELINES_DIRTY=true` with a
|
||
`-dirty` version. This is **offline source-level regression coverage**, not the
|
||
recorded actual new-account install/provider login above and not live-provider or
|
||
deployment evidence; this docs-and-ignore change has no rendered UI.
|
||
|
||
## Dated live reference validation: 2026-10-10
|
||
|
||
Environment: Docker Linux aarch64, OpenCode v2.0.20,
|
||
**opencode-go/deepseek-v4.1-flash**. Isolated databases/fixtures and local HTTP
|
||
notification endpoints; normal published pipelines/repositories were not changed.
|
||
|
||
- Incident extraction → report: exact IDs/severities/counts, unchanged handoff,
|
||
retained session readback and one local success receipt passed (6.63s).
|
||
- File edit → verification: exact final release-file bytes, unchanged fixtures,
|
||
independently executed `cmp`, completed read/edit/shell evidence, retained edit
|
||
diff and no verifier edits passed (14.60s).
|
||
- Missing required input: blocked route ended failed; missing file was not invented;
|
||
retained read attempt and one local failure receipt passed (4.21s).
|
||
- Accounting/one-token budget initially failed its acceptance check because session
|
||
zeros let the run succeed. After the tracked-prompt fix, a focused live rerun
|
||
passed: **952 reported tokens, $0.00016440**, budget-exceeded failure, stopped
|
||
owned session (2.43s). This is not an independently reconciled provider bill.
|
||
|
||
The earlier three workflows were rechecked from retained artifacts with current
|
||
assertions, without more inference. There was **no fresh full-suite live run of
|
||
all four workflows after the fix**. Do not generalize these results to all five
|
||
starter templates, other models/providers, or compatible native schema/tool usage.
|
||
Real ntfy authentication/storage, phone reception, provider outages/timeouts,
|
||
long-stream interruption and promptness of cleanup were **not tested** by this trial.
|
||
The trial ran rebuilt test binaries; it did not deploy the fix to the ordinary server.
|
||
|
||
## Dated live reference validation: 2026-10-11 (run7)
|
||
|
||
One existing real run7 (2026-10-11) is the retained live evidence dated for this
|
||
change: it executed both configured harnesses, OpenCode 2.0.26 and Pi 1.1.0, with
|
||
`opencode-go/deepseek-v4.1-flash` stages on the Docker server, and both steps
|
||
succeeded (6s; the retained Pi evidence panel reports 1054/10000 tokens). The
|
||
captures are retained at `tmp/friend-ui-verification/source-pi-80-*` and
|
||
`source-pi-120-*`, and a current-source TUI inspection of that run recorded actual
|
||
80x30 and 120x36 renderings, including the run graph and the bounded evidence
|
||
panel.
|
||
|
||
This is a single real run, **not** a fresh full reference suite. Other providers,
|
||
real outages/long-stream interruption, fresh OAuth and phone delivery remain
|
||
**not tested**; the new-source installer and installed-authoring verification were
|
||
separately completed by the dated 2026-10-11 FINAL INSTALLED section above. The
|
||
[2026-10-10 OpenCode v2.0.20 trial](#dated-live-reference-validation-2026-10-10)
|
||
above remains dated history.
|
||
|
||
## Reference suite when explicitly requested
|
||
|
||
The source checkout's `TestReferenceWorkflowsLive` uses six model stage attempts
|
||
plus possible compatibility/format-repair calls. Retries are disabled and stages
|
||
have two-minute timeouts. It uses real existing credentials and retains private
|
||
`runs.db`, pinned definitions, fixture directories, run/session JSON and report.
|
||
The accounting case has a one-token/$0.25 budget; other cases have no spending
|
||
limit. Test notifications use a local endpoint, not the configured phone topic.
|
||
|
||
Live inference requires explicit `PIPELINES_REFERENCE_LIVE=1`, a chosen
|
||
`PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL`, and an existing private
|
||
`PIPELINES_REFERENCE_DIR`. Get consent for chargeable inference and confirm which
|
||
execution computer/account to use. Ordinary tests skip live inference. Do not
|
||
search the repository or run the suite during normal draft authoring.
|
||
|
||
If the user explicitly asks to run the suite and provides a source checkout:
|
||
|
||
```sh
|
||
PIPELINES_REFERENCE_LIVE=1 \
|
||
PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL \
|
||
PIPELINES_REFERENCE_DIR=/private/approved/artifacts \
|
||
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m
|
||
```
|
||
|
||
Docker credentials require running the correctly compiled test binary in that
|
||
container; do not copy host credentials or publish tests to the normal database.
|
||
`TestReferenceDefinitions` checks definitions without inference.
|
||
`TestReferenceArtifactsReview` rechecks copied run/fixture evidence without
|
||
inference using `PIPELINES_REFERENCE_REVIEW_DIR`; select only cases present in
|
||
that artifact directory. Missing cases skip and must not be reported as passed.
|
||
|
||
## 2026-10-12 run22 success, run23 live CSV fixture and run24 interruption
|
||
|
||
Source/structural and live-fixture evidence stays separate from runtime
|
||
deployment; nothing here is a deployment claim.
|
||
|
||
- **run22 (structural/source checks, success):** the retained run22 verification
|
||
passed its structural/source checks of the scoped template and prompt changes.
|
||
- **run23 (live Pi CSV fixture):** a real exact-template Pi CSV run fetched the
|
||
official Python `csv` documentation through the conditional official-source
|
||
webfetch and implemented CSV parsing with the standard library's `csv.reader`
|
||
through an existing shared helper, adding no duplicate preview parser; both the
|
||
implementer and reviewer checks actually passed. Root independently verified the
|
||
full retained native tool results and protected-file hashes against its saved
|
||
baseline, rather than relying on model-written reports.
|
||
- **run24 (interrupted, not a pass):** the run24 implementation completed, but its
|
||
reviewer ignored the scope instruction and ran a prohibited filesystem-root scan
|
||
(`find /`), leaving a process in uninterruptible D-state and stalling Docker
|
||
mount/snapshot reads. Root canceled the run, made a consistent SQLite backup
|
||
(`integrity ok`), recovered Docker with the supported stop/start path, and
|
||
restored the existing ntfy/Pipelines services with the original definitions, run
|
||
history, credentials and settings verified unchanged. run24 is retained as
|
||
**interrupted**, not a success.
|
||
|
||
- **run25 (live Pi source-only review):** the reviewer identified duplicated
|
||
implementation, a hardcoded special case, suppressed errors and test gaps.
|
||
Root inspected retained native calls: only local read/search tools ran, with no
|
||
shell/edit/network calls. Original fixture hashes were unchanged; runtime
|
||
behavior was explicitly marked untested.
|
||
- **run26 (final maintenance review, success):** the scoped plan/implement/review
|
||
workflow passed package tests, vet, build and formatting checks. Root inspected
|
||
the actual retained outputs and changes. No rendered UI change was claimed.
|
||
|
||
The scoped prompt guidance asks stages to keep inspection/verification to relevant
|
||
repository paths and supported commands; it is a working instruction, **not** an OS
|
||
sandbox, and neither fixes Docker VM/disk bugs nor universally enforces tool
|
||
boundaries. At these recorded checkpoints the source changes had not been deployed.
|
||
Catalog deployment requires a separate check of the running server and its actual
|
||
template contents; these observations do not establish publication or deployment.
|
||
Other providers and phone delivery remain **not tested** by this record.
|
||
|
||
## 2026-10-12 final template verification and Docker catalog deployment
|
||
|
||
The final three-stage template ran as live Pi run27 (verification definition7,
|
||
version2) in a fresh disposable CSV fixture. Root inspected full retained native
|
||
results: the planner fetched official Python CSV documentation, the implementer
|
||
used the standard library through the existing shared parser, and implementer and
|
||
reviewer independently ran `python3 checks.py` successfully. Root reran those
|
||
checks, verified the protected fixture hashes and planner-owned review criteria,
|
||
and confirmed the preview reused the shared parser.
|
||
|
||
The Docker server was then refreshed from clean source commit
|
||
`143f237268dbde2efd6c5ce5722a3e1fd7e08c9c`, retaining OpenCode 2.0.26, Pi 1.1.0
|
||
and Node 24.18.1. The running server reported that exact commit and `dirty=false`;
|
||
all five served templates matched current source with the saved Pi/model defaults.
|
||
A consistent pre-deployment database backup passed integrity checks. Every stored
|
||
definition version/source, retained run ID, saved setting, endpoint and provider
|
||
credential/account checked by Root was preserved. This record is a documentation
|
||
follow-up to that deployed build; it changes no runtime code or template source.
|
||
|
||
These bounded fixtures establish the observed behavior on this instance, not a
|
||
guarantee of every future model review, provider, task or platform. They do not
|
||
establish fresh account sign-in or phone notification reception.
|
||
|
||
## Orchestration verification habits
|
||
|
||
A run's status and a stage's summary are **claims**, not evidence. When the agent
|
||
manages runs rather than making the change, its job includes independently checking
|
||
what each run claims.
|
||
|
||
- **Read the retained evidence yourself** — `inspect RUN_ID --json` for the exact
|
||
stage outputs, outcomes, diagnostics, file changes and artifacts — instead of
|
||
trusting the model-written summary of the run.
|
||
- **Run the check.** Where an item claims a build or tests, run the exact command and
|
||
read its output; "tests pass" without a command and its observed result is not a
|
||
check. Never report a check you did not execute.
|
||
- **Read the images.** Open the screenshots the reviewer cites; do not accept a
|
||
description of an image you can open. Confirm a capture is a **real image** (magic
|
||
bytes or the reported content type), not a text file named `.png`
|
||
(`references/debugging.md`).
|
||
- **Compare the built artefact with what was delivered.** Check the md5 (or
|
||
equivalent) of what was built against what is installed or published, so a stale or
|
||
substituted file cannot pass as the artefact (`references/delivery.md`).
|
||
- **State "not tested" explicitly.** A run that `succeeded` means its stages reported
|
||
success; it is not proof that the operator's acceptance criteria were met. Name what
|
||
was not covered.
|
||
- **Report failures as failures.** Never re-run until something passes and then call
|
||
that verified; record the failure and its category.
|