9f88e4ff71
A model stage whose final result was malformed failed the whole run even when
the work and evidence had survived, because the only recovery was an isolated
formatter call on a fixed 30-second deadline that could time out on its own
(run 55, ui_reviewer of pipeline 9 v6).
Add an ordered, deterministic local repair pass (internal/stage/result_repair.go)
that runs before any model call: (a) append missing closing delimiters via the
existing closeJSONContainers; (b) remove one unambiguous trailing comma before a
closing } or ]; (c) insert exactly the single comma/colon the JSON parser reported
at its byte offset. Every candidate must pass losslessSyntaxRepair(original,
candidate) and decodeModelResult before it is accepted, so no field value is
invented, removed, reordered or rewritten. Ambiguous payloads (an unescaped quote
or colon inside a string value, or a stray delimiter with no single completion)
return not-ok and fall through to the bounded model re-emit rather than guessing.
stageResultText still strips surrounding prose and a single code fence first;
model_prompt.go now also strips prose around a single balanced-but-malformed
object span so the deterministic pass sees just the object.
Replace the fixed 30-second formatter deadline with repairDeadline(payloadBytes,
remaining), which grows monotonically with payload size, has a floor, and is
clamped to the attempt's remaining budget minus a safety margin, so the stage
context remains the outer bound and a large result is not guaranteed to time out.
The one-request-per-attempt, 32 KiB bound and the deny-all, retained, new-session
property of RepairFormat are unchanged for OpenCodeClient and PiClient.
Record an explicit recovery status on every terminal path: repaired, partial or
unrecoverable, plus a byte offset. A candidate that is valid JSON but still fails
envelope validation is marked partial and never returned as a completed result;
the unrecoverable path retains the bounded original payload, the precise parse
error (including its byte offset) and surfaces the status in the FINAL RESULT
VALIDATION diagnostic.
safeToReplay is intentionally unchanged. The deterministic local repair and the
fresh deny-all, no-tools, retained formatter session are independent of the
stage's tool set and never re-prompt or replay the admitted agent session, so a
shell-enabled review stage gains malformed-result recovery without any replay of
admitted work. Risk: no change to mutation safety; the only added cost is one
bounded formatter call, already budgeted. Replayability is not extended for shell
stages.
Docs and packaged skill references (docs/model-results.md,
internal/author/skill/references/{format,debugging,verification}.md) describe the
three-tier recovery, the status words and the derived deadline; the safeToReplay
decision and its risk are recorded there alongside the commit.
Tests add mocked, offline coverage for the deterministic and partial paths, a
repairDeadline monotonicity/floor/clamp table, and a shell-stage regression
proving a recoverable malformed result does not fail the run or replay tool work.
546 lines
35 KiB
Markdown
546 lines
35 KiB
Markdown
# Verification, accounting and real-provider evidence
|
||
|
||
Structural validation is not inference, tool execution, correct task completion,
|
||
phone delivery or correct billing. Distinguish observed artifacts/command results
|
||
from model-written reports. State the command/check actually run and say **not
|
||
tested** for anything not executed. Never present mocked results as live-provider
|
||
evidence or a different backend/model as already validated.
|
||
|
||
## Authoring useful verification
|
||
|
||
Preserve explicit output handoffs and evidence fields. Implementer/reviewer stages
|
||
can use appropriate permitted commands; do not accept a preceding stage's success
|
||
summary as independent proof. Actual file bytes, command exit status, retained tool
|
||
parts and server-captured diffs are stronger evidence than prose. A read failure
|
||
can prove a deliberately missing input, but is not a successful read. Tool names
|
||
mentioned in text do not prove tools executed. A reviewer stage must inspect the
|
||
actual files and run its own checks; a source-only review runs no commands and
|
||
must state that runtime behavior was not tested. Structural, mocked or
|
||
authoring-time checks are not deployed or live-provider evidence.
|
||
|
||
Do not add technology-specific checks unless the task/repository requires them.
|
||
Keep runtime verification separate from authoring: ask before running project code,
|
||
making chargeable model calls, editing fixtures, or sending test notifications.
|
||
Do not weaken an acceptance check just to get a passing run.
|
||
`provider-check` must run on the intended execution computer/container with its
|
||
credentials; a client-side check is not proof of a remote server's provider health.
|
||
|
||
## Budgets and backend caveats
|
||
|
||
Defaults remain two retries and one hour per stage attempt; a run-level wall-clock
|
||
deadline is not added. Budgets are optional, positive and independent USD/token
|
||
reported-usage limits shared across attempts, format repairs and repeat visits.
|
||
Accounting failures stop budget-limited stages. In-flight calls can overshoot;
|
||
these are **not provider-side hard caps or a guaranteed bill ceiling**. Reported
|
||
USD may validly be zero; do not fabricate prices/accounting from output length.
|
||
|
||
A live V2 trial found `/generate` returned text while its retained session had no
|
||
messages and zero counters. Current source uses tracked `/prompt` → bounded
|
||
active-session polling → message readback for **budgeted V2 text-only calls, with
|
||
all tools denied**. Completed token-budgeted responses reporting zero tokens fail
|
||
closed. No new definition flags or agent tools are required for this fix.
|
||
Unbudgeted tool-free V2 generation can still have empty backend message history;
|
||
pipeline inputs, results and command audit remain retained. An unbudgeted V2
|
||
audited agent stage makes exactly one extra bounded final session-aggregate read
|
||
to record any reported USD/token usage best-effort: missing or invalid usage stays
|
||
unknown and never turns a valid run into a failure, cumulative sessions/retries
|
||
are keyed per session so they cannot double count, and totals are never inferred
|
||
from final-assistant-only or native-partial data. Budgeted settlement and native
|
||
V1 enforcement are unchanged. Native V1 uses its existing path.
|
||
|
||
The V2 `.../experimental/session/{id}/wait` request is **no longer used**: a dated
|
||
Docker log shows it timing out at the CLI's own request timeout after about six
|
||
minutes (run started 2026-10-10T23:07:16.547Z, `TimeoutError` logged at
|
||
2026-10-10T23:13:17.262Z, server `http.status=499`), which aborted a long
|
||
implementation stage. Instead current source polls `GET /api/session/active`
|
||
about once per second with a context-aware delay and no busy loop, until the
|
||
admitted session is absent (inactive). Missing/null/non-object `data`, or a state
|
||
other than the documented `running` type, fails closed and is never treated as
|
||
inactive. This is **mocked regression coverage** in `internal/stage` (fake CLI),
|
||
not live-provider evidence; V1 native settlement is unchanged.
|
||
|
||
Current source reads the V2 final result with a single bounded descending
|
||
final-assistant readback (`order=desc&limit=1&type=assistant`) after the settled
|
||
poll, for both audited agent stages and budgeted V2 text-only calls, so large
|
||
retained history (including prior tool-read contents) is no longer fetched just
|
||
to select the completed result. A truncated/incomplete readback, an empty data
|
||
list, or a latest message that is neither completed `stop` nor a completed
|
||
structured tool-call fails closed with a sanitized diagnostic that omits
|
||
provider content; retained audit previews are **not** executable final results.
|
||
Session/debug evidence now reuses the same single latest-assistant readback
|
||
instead of a recent-20 all-message fetch, and returns each payload as a labelled
|
||
bounded, credential-redacted envelope with explicit preview/truncation/missing
|
||
diagnostics; a small final assistant result is therefore selected even when large
|
||
prior tool content exists. Pi evidence remains documented JSONL and is never
|
||
parsed as one JSON object. This describes current source behavior, not a guarantee
|
||
that a user's installed server includes the fix. An updated skill/client is not
|
||
deployment evidence; never rebuild/restart services or log into providers without
|
||
specific authorization.
|
||
|
||
Retained model-stage failures now carry a fixed sanitized category. A strictly
|
||
invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a
|
||
byte offset) is labeled `FINAL RESULT VALIDATION`; request, V2 active-session poll
|
||
and final-readback failures are labeled `HARNESS TRANSPORT`. A validation failure
|
||
preserves the admitted session, transcript and evidence and performs no second tool
|
||
execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history
|
||
readback of exactly 262144 bytes of incomplete JSON (`invalid agent messages`), a
|
||
reviewer result with a duplicate nested `performance_review` field, and an
|
||
onboarding visual-review result with `invalid character 'z' after object key:value
|
||
pair at byte 3968` where unsafe lossless repair was rejected. These are **mocked
|
||
`internal/stage` fixtures**: executable source-level evidence, not live-provider
|
||
evidence and not a deployment. OpenCode V2 prose `output_contract` does not enforce
|
||
structured output; V2 results are validated locally and no unsupported V2 format
|
||
field is sent. A review-only fresh execution is a future recovery idea, **not
|
||
implemented**. Onboarding UI visual approval remains unresolved and must not be
|
||
reported complete.
|
||
|
||
## Bounded malformed-result recovery: 2026-10-16
|
||
|
||
Source-level, mocked `internal/stage` regression coverage (not live-provider
|
||
evidence and not a deployment): malformed final-result recovery now tries a
|
||
deterministic local pass first (append missing closers, drop one trailing comma,
|
||
or insert the single comma/colon the parser expected), then at most one deny-all
|
||
formatter request whose deadline is derived from the payload size and the
|
||
attempt's remaining budget rather than a fixed 30-second constant. Every path
|
||
records `repaired`, `partial` or `unrecoverable` in `format_repairs`; a `partial`
|
||
or `unrecoverable` payload is never returned as a completed result, and an
|
||
ambiguous payload (an unescaped quote/colon inside a string value) is never
|
||
guessed at. A shell-enabled stage's malformed but deterministically recoverable
|
||
result now completes without replaying admitted tool work; `safeToReplay` is
|
||
unchanged. The run-55 `ui_reviewer` payload was retrieved read-only with
|
||
`pipelines inspect 55 --json` and reproduced its exact byte offset 7837; it is the
|
||
ambiguous error class, so it is reported unrecoverable rather than repaired. This
|
||
does not update or restart any running server.
|
||
|
||
## Failure-handling source changes: 2026-10-15
|
||
|
||
Source-level, mocked `internal/stage` and `internal/server` regression coverage
|
||
(not live-provider evidence): an explicit read-only tools stage now actually
|
||
retries a post-admission stage deadline or harness transport failure, creating a
|
||
fresh bounded session per attempt, while a stage with `edit`/`shell` or no tools
|
||
block still never replays admitted work; a malformed final result is re-emitted
|
||
up to `retries` only on such a read-only stage with its `format_repairs` evidence
|
||
retained. A timed-out agent stage settles knowable V2 usage and retains one
|
||
bounded, redacted latest-assistant readback plus a `retained_session` pointer in
|
||
its stored transcript. A declared outcome routed to `#failure` with no step
|
||
diagnostic now yields a run diagnostic like `stage <key> returned outcome
|
||
"blocked"`; an existing step diagnostic is still used verbatim.
|
||
|
||
## Dated source-level coverage: 2026-10-11
|
||
|
||
Source-level/mocked regression coverage (not live-provider evidence):
|
||
`resolve-releases` now uses a single bounded
|
||
`GET .../git/matching-refs/tags/v2` response that the GitHub REST docs describe as
|
||
complete with no pagination; offline fixtures cover a 100+ ref array,
|
||
`MAX_REFS`/`MAX_BODY` caps, ignored prerelease/build tags, malformed refs and a
|
||
missing newest-tag digest failing closed with no older fallback, with no
|
||
`tags/list` cross-check. Picker tests cover blink/index preservation,
|
||
ANSI-readable selection without a colliding background, and
|
||
40x16/80x24/120x36 height budgets. The compiled picker PTY check (opt-in via
|
||
`PIPELINES_HARNESS_PTY_DIR`) waits through blink intervals and asserts exact
|
||
saved/arrow-scrolled models. `setup-docker.py` accepts the actual
|
||
`opencode v2.0.26` probe token and compares source identity only on a full
|
||
install; model/login-only reconfiguration compares the verified release image
|
||
identity. Python fixtures bind temporary HOME directories so the suite passes
|
||
under the container default `/home/pipelines` without weakening mount guards.
|
||
The Docker authoring transport is also covered at source level: the shared
|
||
prefix picks `/usr/local/bin/pipelines` for a Docker-linked profile. Loopback is
|
||
recognized only via `net.ParseIP(...).IsLoopback()` and an exact case-insensitive
|
||
`localhost`, so a remote DNS name such as `127.remote.example` stays remote with
|
||
no probe. A single bounded `docker compose port` probe parses complete
|
||
`host:port` bindings, and scoped source/target aliases are embedded only when a
|
||
binding covers the requested loopback address and port; malformed or non-matching
|
||
bindings and a loopback `wss` endpoint fail closed, and `client.Dial` rewrites
|
||
only an exact source match. `provider-check` dispatches through a Docker-linked
|
||
profile's existing service with `-T`, the original cwd/model/timeout, exact
|
||
stdout/exit propagation and `--json` passed only in JSON mode (fixture tests, no
|
||
real provider), while explicit native profiles keep the host generator. The
|
||
picker uses a true `x/term` TTY check so `/dev/null` and pipes fail fast, with a
|
||
short context deadline so a regression fails rather than hangs. Real Docker
|
||
rebuild/reconfigure against live registries, a real
|
||
40x16/80x24/120x36 terminal capture, and real model inference remain **not tested**
|
||
in this change.
|
||
|
||
### 2026-10-11 Pi sign-in handoff (source + Root PTY observation)
|
||
|
||
Root separately exercised current-source `pipelines harness setup pi --login`
|
||
under a real Docker PTY with copied harness preferences. Root observed that
|
||
Pi 1.1.0 defaults to fullscreen and clears the Pipelines login instructions, so
|
||
the user sees a coding interface without the `/login` guidance, and that
|
||
`/login` first shows "Select authentication method" (account / API key / Radius)
|
||
rather than a provider picker. The same handler run with a temporary executable
|
||
wrapper passing `pi --tui-mode regular` retained the Pipelines header and
|
||
instructions above Pi's interface. Installed `pi --help` documents the supported
|
||
`--tui-mode regular` flag and the fullscreen default; `pi auth --help` confirms
|
||
there is no standalone login (only print/check, never run). These are source/root
|
||
observations with copied preferences: **no provider was selected, no key was
|
||
entered and no inference occurred**, and this source is **not deployed** and is
|
||
not live-provider evidence. A first reviewer capture at 80x30 found the longer
|
||
handoff plus Pi's own banner scrolled the Pipelines header off the top, so the
|
||
copy was shortened to a title plus three short lines (existing-credential reuse
|
||
folded into the `/login` line, with the account/API key/Radius method choice and
|
||
`ctrl+c`/`ctrl+d` exit keys kept) and the redundant trailing blank removed.
|
||
Independent PTY re-capture at 80x30/100x36 after this shortening is **not
|
||
tested** in this change. The matching exact login-argv and copy assertions are
|
||
covered by fixture tests in `internal/cli`/`internal/harness`.
|
||
|
||
### 2026-10-11 server CLI fixture isolation (source-level + Root-reported run)
|
||
|
||
Root-reported runtime evidence supplied for this change (not re-executed by the
|
||
authoring stages): the deterministic seeded-profile regression
|
||
`go test ./tmp/preview-verification -run TestServerCLIFixtureDoesNotInheritUserDockerProfile -count=1`
|
||
seeds a synthetic Docker-linked harness profile in a child
|
||
`go test ./internal/server -run '^TestCLIHeadlessRunInspectionCancellationAndRPC$'`.
|
||
It previously failed because the child inherited the caller's real
|
||
`PIPELINES_HARNESS_CONFIG`, so the CLI `debug` prefix failed closed
|
||
("refusing to redirect") against the test's ephemeral httptest endpoint. Root's
|
||
four independent preview tests for the bounded transcript helpers pass; Root's
|
||
full host race run produced no race warning and only this fixture failure, and
|
||
`go vet ./...` passed. Current source therefore isolates the server CLI fixture
|
||
with the same missing-temp-file
|
||
`t.Setenv("PIPELINES_HARNESS_CONFIG", filepath.Join(t.TempDir(), "harnesses.json"))`
|
||
pattern already used by `internal/cli`'s setup, so the host executable stays
|
||
authoritative and no Docker probe runs for an ephemeral test endpoint. The
|
||
production fail-closed redirect guard in `internal/harness/prefix.go` and its
|
||
`refusing to redirect` / loopback-wss regressions are unchanged; the failing
|
||
verification was fixed rather than removed.
|
||
|
||
This is a current-source and Root-reported observation. At the time of that
|
||
observation, independent re-execution of the full `go build`/`go vet`/`go test`
|
||
matrix (including Docker-only race results) and the final installed-host Docker
|
||
verification were **pending**; they are now recorded in the 2026-10-11 FINAL
|
||
INSTALLED section below, so this phase-specific wording is superseded by that dated
|
||
install. The packaged reference describes source behavior; a source fix or updated
|
||
skill is not deployment evidence.
|
||
|
||
### 2026-10-11 retained run11 reviewer re-capture (source-level, not a final install)
|
||
|
||
Independently reading the retained run11 verification at
|
||
`tmp/maintenance-run11-verification.json` records the source-level review
|
||
sequence: an earlier reviewer PTY capture rejected the longer handoff because at
|
||
80x30 it scrolled the Pipelines header off the top; the implementer then
|
||
shortened `AuthHandoff` to a title plus three short lines; and the re-run
|
||
`reviewer` and `ui_reviewer` stages approved from real current-source Pipelines
|
||
PTY captures at 80x30 and 100x36 that showed the header and handoff visible above
|
||
installed Pi 1.1.0's regular interface. A labelled controlled fixture that
|
||
stripped `--tui-mode regular` reproduced the pre-fix fullscreen clear, and the
|
||
review stages report no provider selected, no key entered and no inference. This
|
||
is source-level reviewer evidence, read from retained output; it was not a
|
||
deployed/installed state at that time, and the final full Docker install
|
||
verification was then **pending** — it is now recorded in the 2026-10-11 FINAL
|
||
INSTALLED section below, so this phase-specific wording is superseded by that dated
|
||
install.
|
||
|
||
Root-reported actual runs (source-level observations supplied by Root for this
|
||
change and not re-verified here): run7 showed both configured providers' exact
|
||
handoff markers; runs6/9/10 exercised live maintenance with final-assistant
|
||
polling/readback succeeding. These are Root observations, not mocked-regression
|
||
coverage and not a completed installation; no final deployed/installed state is
|
||
claimed.
|
||
|
||
### 2026-10-11 run13 interruption, Docker race limitation and state separation
|
||
|
||
Four states are kept explicitly separate and must not be conflated:
|
||
|
||
1. **Current source** (this checkout, not deployed): bounded 32 KiB
|
||
(`transcriptBound = 32<<10`) redacted transcript previews produce a valid JSON
|
||
envelope with redaction applied before truncation and a rune-safe 8 KiB preview;
|
||
session/debug evidence is scoped `latest_bounded_evidence` with the single
|
||
bounded latest-assistant readback and at most the 3 most recent session-bearing
|
||
steps; an unbudgeted V2 audited agent stage records reported usage best-effort
|
||
through exactly one 2s-bounded final session-aggregate read, leaving
|
||
missing/invalid usage unknown; and the isolated server CLI fixture uses the
|
||
missing-temp-file `PIPELINES_HARNESS_CONFIG` pattern while the production
|
||
fail-closed redirect guard in `internal/harness/prefix.go` is unchanged. Pi JSONL
|
||
is never parsed as one JSON object; native V1 keeps its existing path.
|
||
2. **Host verification evidence independently recorded by Root**: after the fixture
|
||
fix, Root ran the full `go test -race -count=1 ./...` and `go vet ./...`; after
|
||
three harness-neutral evidence labels and Step wording changes Root re-ran the
|
||
affected `tui`/`author`/`cli` race and vet checks, the Python suite (80) and the
|
||
opt-in compiled picker PTY check (4). These are Root-recorded host results, not
|
||
re-executed by the authoring stages.
|
||
3. **Deployed old image**: the running Docker image predates these source changes
|
||
and does not include them.
|
||
4. **Final install (completed 2026-10-11)**: the final install and installed
|
||
authoring/provider verification were **pending** when this separation was
|
||
written and are now recorded in the 2026-10-11 FINAL INSTALLED section below; an
|
||
updated skill/client or source checkout is still not deployment evidence on its
|
||
own.
|
||
|
||
Docker race limitation (confirmed): the Docker runtime is built with
|
||
`CGO_ENABLED=0` and has **no C compiler** (the Alpine runtime stage installs no
|
||
GCC/musl-dev), so `go test -race` cannot run there. In-Docker reviewers must use
|
||
ordinary `go build`/`go test`/`go vet` and rely on independently executed host
|
||
`-race` runs. Race checks must not be attempted in Docker and no compiler may be
|
||
installed or searched for.
|
||
|
||
Run13 interruption (dated evidence, not a fix): the previous reviewer ran `find /`
|
||
looking for GCC; the tool timeout left a process in disk sleep, Docker could not
|
||
stop the container, and an approved Desktop restart recovered run13 as interrupted
|
||
(`executing=false`, every step `agent_active=false`) with **no automatic replay**.
|
||
This is an environment/interruption observation only: no general cancellation fix
|
||
is inferred and process termination is unchanged. Docker startup recovery already
|
||
sets running steps to `interrupted` with `agent_active=0`.
|
||
|
||
### 2026-10-11 short-lived CLI inherited-pipe cancellation (current source)
|
||
|
||
Current source (not deployed) adds one shared, dependency-free constructor in
|
||
`internal/opencode`: `CommandContext(ctx, executable, args...)` sets the single
|
||
finite `os/exec.Cmd.WaitDelay` constant `ShortLivedWaitDelay = 250ms` and is the
|
||
only place that value is defined. It is reused by the `internal/stage`
|
||
`OpenCodeClient.call()` API/agent path (including `Interrupt`), `opencode.Detect()`
|
||
`--help` and the native `Connection.Messages()` export fallback. It bounds
|
||
**draining of inherited output pipes only**: after the child exits os/exec waits at
|
||
most the grace for descendants holding the inherited stdout/stderr pipes, then
|
||
closes them and reports `exec.ErrWaitDelay`; it does not terminate or confirm
|
||
cleanup of descendants.
|
||
|
||
`context.Canceled`/`context.DeadlineExceeded` identities, `exec.ErrWaitDelay` on
|
||
the sanitized errors, the ≤64 KiB request / ≤1 MiB response / 8 KiB stderr bounds,
|
||
`boundedBuffer`/`boundedOutput` drain-with-overflow behavior, redaction, strict
|
||
result validation and cleanup-unconfirmed active ownership (`ErrStageCleanup`, no
|
||
replay) are unchanged; no new module dependency is added and a parent that exits
|
||
successfully while a descendant still holds partial stdout is never treated as
|
||
success.
|
||
|
||
Dated synthetic reproduction (`tmp/interrupt-verification`, synthetic
|
||
real-subprocess diagnostics, **not provider evidence**): with Root waiting for the
|
||
descendant itself to start before cancelling public
|
||
`OpenCodeClient{Backend:"v2"}.Interrupt`, two Docker/Linux returns took
|
||
3996/3995ms despite `context.Canceled` (`descendant-started.jsonl`); the stronger
|
||
host reproduction took 4003/4000ms (`reproduction.log`). Added source-level
|
||
regressions use real `os/exec` fixtures with start-marker barriers, bounded
|
||
watchdogs and naturally terminating children for cancel-after-descendant,
|
||
deadline identity, descendant-held pipes → `ErrWaitDelay`, ordinary
|
||
output/exit/redaction, failed-interrupt ownership/no replay, plus `Detect`/export
|
||
wait-delay and cancellation cases. This is **current source, not deployed**:
|
||
descendant termination, confirmed remote interruption and any run13 Docker disk
|
||
sleep fix are **not claimed**, and Docker race remains unavailable (no C
|
||
compiler).
|
||
|
||
### 2026-10-11 FINAL INSTALLED verification (friend-ready preview)
|
||
|
||
This is the dated receipt for the installed runtime, recorded by Root from
|
||
`tmp/friend-final-verification/summary.json`; it supersedes the earlier
|
||
"pending final install" statements above, which stay as phase-specific history.
|
||
A normal interactive `./install --model opencode-go/deepseek-v4.1-flash
|
||
--harness-model opencode-go/deepseek-v4.1-flash` exited **0**, resolving the current
|
||
releases live and building fresh dirty source into actual image
|
||
`sha256:1810ab50f3ea3991e7f3548c79250e65a114bb1821eac52efd872b91cf7c3136`
|
||
(OpenCode 2.0.26 / Pi 1.1.0 / Node 24.18.1) with real tool-free provider
|
||
validation, ntfy authenticated publish, server + host health, home/workspace entry,
|
||
preserved authoritative endpoint and `ok` pre-install database/home backup
|
||
integrity. Post-install preservation all passed (original definitions 1–4, history
|
||
1–16, host/container default Pi and both models/profiles, logical endpoint,
|
||
service URLs/timezone/ports/models, Pi credential hash, OpenCode connected accounts,
|
||
ntfy token); neither original-repo commit/push nor credential/data cleanup ran.
|
||
Installed host-CLI authoring with **both** Docker harnesses (separate retained
|
||
drafts, each changing only the worker prompt) performed real `ping` → `pong` and
|
||
`validate` → `validated` through the generated Linux `/usr/local/bin/pipelines` and
|
||
scoped transport aliases preserving the logical `8081` endpoint and draft metadata,
|
||
independently inspected from native OpenCode tool outputs and retained Pi JSONL
|
||
`toolResults`, with no draft submit/run/publication. Docker-proxy submit on
|
||
verification-only definition 5 published v2, a second stale v1 draft was rejected as
|
||
conflict, and **Run17** succeeded in 7 seconds with the exact
|
||
`pipelines-docker-verified` markers in OpenCode then Pi, both inactive and exactly
|
||
one completed marker read verified from retained native history (not the
|
||
latest-assistant preview); best-effort unbudgeted usage was OpenCode 4011
|
||
tokens/$0.00029769599999999996 and Pi 1071 tokens/$0.000142002, and
|
||
`/api/session/active` was empty after exit. Installed-UI captures
|
||
(`installed-pi-*`, `pi-signin-*`, `installed-pi-models-*`) cover the actual run17
|
||
80x30/120x36 neutral-Pi/green-both-stage renders, the Pi auth handoff/method menu
|
||
viewed/canceled/exit0 with no sign-in, and the model wizard using copied
|
||
preferences only while production preferences stayed preserved. Root's independent
|
||
checks passed: full host `go test -race -count=1 ./...` and `go vet ./...`,
|
||
Python 80 tests (1 optional PTY skipped), Node policy, isolated developer lifecycle
|
||
fixture (no inference/deploy) and author package tests; earlier independent
|
||
reviewers ran ordinary Docker build/vet/tests and focused subprocess regressions
|
||
(no Docker race, no GCC search, no global `find`). **UNVERIFIED** here: fresh
|
||
OAuth/provider login completion, phone delivery, other real providers, the
|
||
D-disk-sleep fix and active lease expiry; authentication menus and
|
||
existing-credential inference are not fresh sign-in proof. The stronger synthetic
|
||
PUBLIC `Interrupt` child-start barrier (host 252/252 ms, Docker 250/251 ms, context
|
||
canceled true, bounded true vs ~4 s before) is not proof of descendant termination,
|
||
remote cleanup or a Docker VM disk fix. Archive extraction, build-identity and
|
||
compile verification remain pending until Root executes them.
|
||
|
||
### 2026-10-11 generated release receipt ignore (source-hygiene regression)
|
||
|
||
Current source adds two root-anchored `.gitignore` rules for the installer's
|
||
machine-local generated artifacts: `/.pipelines-releases.json` (the receipt path
|
||
from `setup-docker.py` `release_receipt_path()`) and `/.pipelines-releases-*` (the
|
||
`tempfile.mkstemp` `prefix=".pipelines-releases-"` in `record_release_receipt()`);
|
||
no installer/runtime code changed and the pre-existing receipt and other files are
|
||
retained. A new `scripts/test_build_metadata.py` case is a controlled, isolated Git
|
||
fixture using the repository's actual `.gitignore`: a clean tracked source plus a
|
||
representative generated receipt/`.pipelines-releases-*` temporary file leave `git
|
||
status --porcelain` empty and `build_metadata.docker_args` at
|
||
`PIPELINES_DIRTY=false` with no `-dirty` version; editing the tracked source then
|
||
makes the status non-empty and `docker_args` report `PIPELINES_DIRTY=true` with a
|
||
`-dirty` version. This is **offline source-level regression coverage**, not the
|
||
recorded actual new-account install/provider login above and not live-provider or
|
||
deployment evidence; this docs-and-ignore change has no rendered UI.
|
||
|
||
## Dated live reference validation: 2026-10-10
|
||
|
||
Environment: Docker Linux aarch64, OpenCode v2.0.20,
|
||
**opencode-go/deepseek-v4.1-flash**. Isolated databases/fixtures and local HTTP
|
||
notification endpoints; normal published pipelines/repositories were not changed.
|
||
|
||
- Incident extraction → report: exact IDs/severities/counts, unchanged handoff,
|
||
retained session readback and one local success receipt passed (6.63s).
|
||
- File edit → verification: exact final release-file bytes, unchanged fixtures,
|
||
independently executed `cmp`, completed read/edit/shell evidence, retained edit
|
||
diff and no verifier edits passed (14.60s).
|
||
- Missing required input: blocked route ended failed; missing file was not invented;
|
||
retained read attempt and one local failure receipt passed (4.21s).
|
||
- Accounting/one-token budget initially failed its acceptance check because session
|
||
zeros let the run succeed. After the tracked-prompt fix, a focused live rerun
|
||
passed: **952 reported tokens, $0.00016440**, budget-exceeded failure, stopped
|
||
owned session (2.43s). This is not an independently reconciled provider bill.
|
||
|
||
The earlier three workflows were rechecked from retained artifacts with current
|
||
assertions, without more inference. There was **no fresh full-suite live run of
|
||
all four workflows after the fix**. Do not generalize these results to all five
|
||
starter templates, other models/providers, or compatible native schema/tool usage.
|
||
Real ntfy authentication/storage, phone reception, provider outages/timeouts,
|
||
long-stream interruption and promptness of cleanup were **not tested** by this trial.
|
||
The trial ran rebuilt test binaries; it did not deploy the fix to the ordinary server.
|
||
|
||
## Dated live reference validation: 2026-10-11 (run7)
|
||
|
||
One existing real run7 (2026-10-11) is the retained live evidence dated for this
|
||
change: it executed both configured harnesses, OpenCode 2.0.26 and Pi 1.1.0, with
|
||
`opencode-go/deepseek-v4.1-flash` stages on the Docker server, and both steps
|
||
succeeded (6s; the retained Pi evidence panel reports 1054/10000 tokens). The
|
||
captures are retained at `tmp/friend-ui-verification/source-pi-80-*` and
|
||
`source-pi-120-*`, and a current-source TUI inspection of that run recorded actual
|
||
80x30 and 120x36 renderings, including the run graph and the bounded evidence
|
||
panel.
|
||
|
||
This is a single real run, **not** a fresh full reference suite. Other providers,
|
||
real outages/long-stream interruption, fresh OAuth and phone delivery remain
|
||
**not tested**; the new-source installer and installed-authoring verification were
|
||
separately completed by the dated 2026-10-11 FINAL INSTALLED section above. The
|
||
[2026-10-10 OpenCode v2.0.20 trial](#dated-live-reference-validation-2026-10-10)
|
||
above remains dated history.
|
||
|
||
## Reference suite when explicitly requested
|
||
|
||
The source checkout's `TestReferenceWorkflowsLive` uses six model stage attempts
|
||
plus possible compatibility/format-repair calls. Retries are disabled and stages
|
||
have two-minute timeouts. It uses real existing credentials and retains private
|
||
`runs.db`, pinned definitions, fixture directories, run/session JSON and report.
|
||
The accounting case has a one-token/$0.25 budget; other cases have no spending
|
||
limit. Test notifications use a local endpoint, not the configured phone topic.
|
||
|
||
Live inference requires explicit `PIPELINES_REFERENCE_LIVE=1`, a chosen
|
||
`PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL`, and an existing private
|
||
`PIPELINES_REFERENCE_DIR`. Get consent for chargeable inference and confirm which
|
||
execution computer/account to use. Ordinary tests skip live inference. Do not
|
||
search the repository or run the suite during normal draft authoring.
|
||
|
||
If the user explicitly asks to run the suite and provides a source checkout:
|
||
|
||
```sh
|
||
PIPELINES_REFERENCE_LIVE=1 \
|
||
PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL \
|
||
PIPELINES_REFERENCE_DIR=/private/approved/artifacts \
|
||
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m
|
||
```
|
||
|
||
Docker credentials require running the correctly compiled test binary in that
|
||
container; do not copy host credentials or publish tests to the normal database.
|
||
`TestReferenceDefinitions` checks definitions without inference.
|
||
`TestReferenceArtifactsReview` rechecks copied run/fixture evidence without
|
||
inference using `PIPELINES_REFERENCE_REVIEW_DIR`; select only cases present in
|
||
that artifact directory. Missing cases skip and must not be reported as passed.
|
||
|
||
## 2026-10-12 run22 success, run23 live CSV fixture and run24 interruption
|
||
|
||
Source/structural and live-fixture evidence stays separate from runtime
|
||
deployment; nothing here is a deployment claim.
|
||
|
||
- **run22 (structural/source checks, success):** the retained run22 verification
|
||
passed its structural/source checks of the scoped template and prompt changes.
|
||
- **run23 (live Pi CSV fixture):** a real exact-template Pi CSV run fetched the
|
||
official Python `csv` documentation through the conditional official-source
|
||
webfetch and implemented CSV parsing with the standard library's `csv.reader`
|
||
through an existing shared helper, adding no duplicate preview parser; both the
|
||
implementer and reviewer checks actually passed. Root independently verified the
|
||
full retained native tool results and protected-file hashes against its saved
|
||
baseline, rather than relying on model-written reports.
|
||
- **run24 (interrupted, not a pass):** the run24 implementation completed, but its
|
||
reviewer ignored the scope instruction and ran a prohibited filesystem-root scan
|
||
(`find /`), leaving a process in uninterruptible D-state and stalling Docker
|
||
mount/snapshot reads. Root canceled the run, made a consistent SQLite backup
|
||
(`integrity ok`), recovered Docker with the supported stop/start path, and
|
||
restored the existing ntfy/Pipelines services with the original definitions, run
|
||
history, credentials and settings verified unchanged. run24 is retained as
|
||
**interrupted**, not a success.
|
||
|
||
- **run25 (live Pi source-only review):** the reviewer identified duplicated
|
||
implementation, a hardcoded special case, suppressed errors and test gaps.
|
||
Root inspected retained native calls: only local read/search tools ran, with no
|
||
shell/edit/network calls. Original fixture hashes were unchanged; runtime
|
||
behavior was explicitly marked untested.
|
||
- **run26 (final maintenance review, success):** the scoped plan/implement/review
|
||
workflow passed package tests, vet, build and formatting checks. Root inspected
|
||
the actual retained outputs and changes. No rendered UI change was claimed.
|
||
|
||
The scoped prompt guidance asks stages to keep inspection/verification to relevant
|
||
repository paths and supported commands; it is a working instruction, **not** an OS
|
||
sandbox, and neither fixes Docker VM/disk bugs nor universally enforces tool
|
||
boundaries. At these recorded checkpoints the source changes had not been deployed.
|
||
Catalog deployment requires a separate check of the running server and its actual
|
||
template contents; these observations do not establish publication or deployment.
|
||
Other providers and phone delivery remain **not tested** by this record.
|
||
|
||
## 2026-10-12 final template verification and Docker catalog deployment
|
||
|
||
The final three-stage template ran as live Pi run27 (verification definition7,
|
||
version2) in a fresh disposable CSV fixture. Root inspected full retained native
|
||
results: the planner fetched official Python CSV documentation, the implementer
|
||
used the standard library through the existing shared parser, and implementer and
|
||
reviewer independently ran `python3 checks.py` successfully. Root reran those
|
||
checks, verified the protected fixture hashes and planner-owned review criteria,
|
||
and confirmed the preview reused the shared parser.
|
||
|
||
The Docker server was then refreshed from clean source commit
|
||
`143f237268dbde2efd6c5ce5722a3e1fd7e08c9c`, retaining OpenCode 2.0.26, Pi 1.1.0
|
||
and Node 24.18.1. The running server reported that exact commit and `dirty=false`;
|
||
all five served templates matched current source with the saved Pi/model defaults.
|
||
A consistent pre-deployment database backup passed integrity checks. Every stored
|
||
definition version/source, retained run ID, saved setting, endpoint and provider
|
||
credential/account checked by Root was preserved. This record is a documentation
|
||
follow-up to that deployed build; it changes no runtime code or template source.
|
||
|
||
These bounded fixtures establish the observed behavior on this instance, not a
|
||
guarantee of every future model review, provider, task or platform. They do not
|
||
establish fresh account sign-in or phone notification reception.
|
||
|
||
## Orchestration verification habits
|
||
|
||
A run's status and a stage's summary are **claims**, not evidence. When the agent
|
||
manages runs rather than making the change, its job includes independently checking
|
||
what each run claims.
|
||
|
||
- **Read the retained evidence yourself** — `inspect RUN_ID --json` for the exact
|
||
stage outputs, outcomes, diagnostics, file changes and artifacts — instead of
|
||
trusting the model-written summary of the run.
|
||
- **Run the check.** Where an item claims a build or tests, run the exact command and
|
||
read its output; "tests pass" without a command and its observed result is not a
|
||
check. Never report a check you did not execute.
|
||
- **Read the images.** Open the screenshots the reviewer cites; do not accept a
|
||
description of an image you can open. Confirm a capture is a **real image** (magic
|
||
bytes or the reported content type), not a text file named `.png`
|
||
(`references/debugging.md`).
|
||
- **Compare the built artefact with what was delivered.** Check the md5 (or
|
||
equivalent) of what was built against what is installed or published, so a stale or
|
||
substituted file cannot pass as the artefact (`references/delivery.md`).
|
||
- **State "not tested" explicitly.** A run that `succeeded` means its stages reported
|
||
success; it is not proof that the operator's acceptance criteria were met. Name what
|
||
was not covered.
|
||
- **Report failures as failures.** Never re-run until something passes and then call
|
||
that verified; record the failure and its category.
|