Files
pipelines/internal/author/skill/references/verification.md
T
alan 05ae530442 Make Pipelines Docker-only with OpenCode and Pi
Install current verified harness releases inside Docker, preserve persistent accounts and settings, and provide separate interactive and pipeline model defaults with terminal onboarding and fuzzy pickers.

Add Docker-aware authoring transport, Pi stage execution, bounded workflow controls, reliable final-result readback and usage settlement, and finite short-lived CLI pipe draining. Keep dated installation evidence and packaged authoring guidance consistent.

Verified with a real Docker installation, both harness stages and interactive authors, the built-in plan/implement/review starter, independent host race/vet checks, installer tests, and packaged skill tests. Generated release receipts stay ignored in clean checkouts.
2026-10-11 23:00:25 +01:00

27 KiB
Raw Blame History

Verification, accounting and real-provider evidence

Structural validation is not inference, tool execution, correct task completion, phone delivery or correct billing. Distinguish observed artifacts/command results from model-written reports. State the command/check actually run and say not tested for anything not executed. Never present mocked results as live-provider evidence or a different backend/model as already validated.

Authoring useful verification

Preserve explicit output handoffs and evidence fields. Implementer/reviewer stages can use appropriate permitted commands; do not accept a preceding stage's success summary as independent proof. Actual file bytes, command exit status, retained tool parts and server-captured diffs are stronger evidence than prose. A read failure can prove a deliberately missing input, but is not a successful read. Tool names mentioned in text do not prove tools executed.

Do not add technology-specific checks unless the task/repository requires them. Keep runtime verification separate from authoring: ask before running project code, making chargeable model calls, editing fixtures, or sending test notifications. Do not weaken an acceptance check just to get a passing run. provider-check must run on the intended execution computer/container with its credentials; a client-side check is not proof of a remote server's provider health.

Budgets and backend caveats

Defaults remain two retries and one hour per stage attempt; a run-level wall-clock deadline is not added. Budgets are optional, positive and independent USD/token reported-usage limits shared across attempts, format repairs and repeat visits. Accounting failures stop budget-limited stages. In-flight calls can overshoot; these are not provider-side hard caps or a guaranteed bill ceiling. Reported USD may validly be zero; do not fabricate prices/accounting from output length.

A live V2 trial found /generate returned text while its retained session had no messages and zero counters. Current source uses tracked /prompt → bounded active-session polling → message readback for budgeted V2 text-only calls, with all tools denied. Completed token-budgeted responses reporting zero tokens fail closed. No new definition flags or agent tools are required for this fix. Unbudgeted tool-free V2 generation can still have empty backend message history; pipeline inputs, results and command audit remain retained. An unbudgeted V2 audited agent stage makes exactly one extra bounded final session-aggregate read to record any reported USD/token usage best-effort: missing or invalid usage stays unknown and never turns a valid run into a failure, cumulative sessions/retries are keyed per session so they cannot double count, and totals are never inferred from final-assistant-only or native-partial data. Budgeted settlement and native V1 enforcement are unchanged. Native V1 uses its existing path.

The V2 .../experimental/session/{id}/wait request is no longer used: a dated Docker log shows it timing out at the CLI's own request timeout after about six minutes (run started 2026-10-10T23:07:16.547Z, TimeoutError logged at 2026-10-10T23:13:17.262Z, server http.status=499), which aborted a long implementation stage. Instead current source polls GET /api/session/active about once per second with a context-aware delay and no busy loop, until the admitted session is absent (inactive). Missing/null/non-object data, or a state other than the documented running type, fails closed and is never treated as inactive. This is mocked regression coverage in internal/stage (fake CLI), not live-provider evidence; V1 native settlement is unchanged.

Current source reads the V2 final result with a single bounded descending final-assistant readback (order=desc&limit=1&type=assistant) after the settled poll, for both audited agent stages and budgeted V2 text-only calls, so large retained history (including prior tool-read contents) is no longer fetched just to select the completed result. A truncated/incomplete readback, an empty data list, or a latest message that is neither completed stop nor a completed structured tool-call fails closed with a sanitized diagnostic that omits provider content; retained audit previews are not executable final results. Session/debug evidence now reuses the same single latest-assistant readback instead of a recent-20 all-message fetch, and returns each payload as a labelled bounded, credential-redacted envelope with explicit preview/truncation/missing diagnostics; a small final assistant result is therefore selected even when large prior tool content exists. Pi evidence remains documented JSONL and is never parsed as one JSON object. This describes current source behavior, not a guarantee that a user's installed server includes the fix. An updated skill/client is not deployment evidence; never rebuild/restart services or log into providers without specific authorization.

Retained model-stage failures now carry a fixed sanitized category. A strictly invalid final result (a duplicate nested field, or malformed/ambiguous JSON with a byte offset) is labeled FINAL RESULT VALIDATION; request, V2 active-session poll and final-readback failures are labeled HARNESS TRANSPORT. A validation failure preserves the admitted session, transcript and evidence and performs no second tool execution. Confirmed ordinary-endpoint failures on 2026-10-11: an all-history readback of exactly 262144 bytes of incomplete JSON (invalid agent messages), a reviewer result with a duplicate nested performance_review field, and an onboarding visual-review result with invalid character 'z' after object key:value pair at byte 3968 where unsafe lossless repair was rejected. These are mocked internal/stage fixtures: executable source-level evidence, not live-provider evidence and not a deployment. OpenCode V2 prose output_contract does not enforce structured output; V2 results are validated locally and no unsupported V2 format field is sent. A review-only fresh execution is a future recovery idea, not implemented. Onboarding UI visual approval remains unresolved and must not be reported complete.

Dated source-level coverage: 2026-10-11

Source-level/mocked regression coverage (not live-provider evidence): resolve-releases now uses a single bounded GET .../git/matching-refs/tags/v2 response that the GitHub REST docs describe as complete with no pagination; offline fixtures cover a 100+ ref array, MAX_REFS/MAX_BODY caps, ignored prerelease/build tags, malformed refs and a missing newest-tag digest failing closed with no older fallback, with no tags/list cross-check. Picker tests cover blink/index preservation, ANSI-readable selection without a colliding background, and 40x16/80x24/120x36 height budgets. The compiled picker PTY check (opt-in via PIPELINES_HARNESS_PTY_DIR) waits through blink intervals and asserts exact saved/arrow-scrolled models. setup-docker.py accepts the actual opencode v2.0.26 probe token and compares source identity only on a full install; model/login-only reconfiguration compares the verified release image identity. Python fixtures bind temporary HOME directories so the suite passes under the container default /home/pipelines without weakening mount guards. The Docker authoring transport is also covered at source level: the shared prefix picks /usr/local/bin/pipelines for a Docker-linked profile. Loopback is recognized only via net.ParseIP(...).IsLoopback() and an exact case-insensitive localhost, so a remote DNS name such as 127.remote.example stays remote with no probe. A single bounded docker compose port probe parses complete host:port bindings, and scoped source/target aliases are embedded only when a binding covers the requested loopback address and port; malformed or non-matching bindings and a loopback wss endpoint fail closed, and client.Dial rewrites only an exact source match. provider-check dispatches through a Docker-linked profile's existing service with -T, the original cwd/model/timeout, exact stdout/exit propagation and --json passed only in JSON mode (fixture tests, no real provider), while explicit native profiles keep the host generator. The picker uses a true x/term TTY check so /dev/null and pipes fail fast, with a short context deadline so a regression fails rather than hangs. Real Docker rebuild/reconfigure against live registries, a real 40x16/80x24/120x36 terminal capture, and real model inference remain not tested in this change.

2026-10-11 Pi sign-in handoff (source + Root PTY observation)

Root separately exercised current-source pipelines harness setup pi --login under a real Docker PTY with copied harness preferences. Root observed that Pi 1.1.0 defaults to fullscreen and clears the Pipelines login instructions, so the user sees a coding interface without the /login guidance, and that /login first shows "Select authentication method" (account / API key / Radius) rather than a provider picker. The same handler run with a temporary executable wrapper passing pi --tui-mode regular retained the Pipelines header and instructions above Pi's interface. Installed pi --help documents the supported --tui-mode regular flag and the fullscreen default; pi auth --help confirms there is no standalone login (only print/check, never run). These are source/root observations with copied preferences: no provider was selected, no key was entered and no inference occurred, and this source is not deployed and is not live-provider evidence. A first reviewer capture at 80x30 found the longer handoff plus Pi's own banner scrolled the Pipelines header off the top, so the copy was shortened to a title plus three short lines (existing-credential reuse folded into the /login line, with the account/API key/Radius method choice and ctrl+c/ctrl+d exit keys kept) and the redundant trailing blank removed. Independent PTY re-capture at 80x30/100x36 after this shortening is not tested in this change. The matching exact login-argv and copy assertions are covered by fixture tests in internal/cli/internal/harness.

2026-10-11 server CLI fixture isolation (source-level + Root-reported run)

Root-reported runtime evidence supplied for this change (not re-executed by the authoring stages): the deterministic seeded-profile regression go test ./tmp/preview-verification -run TestServerCLIFixtureDoesNotInheritUserDockerProfile -count=1 seeds a synthetic Docker-linked harness profile in a child go test ./internal/server -run '^TestCLIHeadlessRunInspectionCancellationAndRPC$'. It previously failed because the child inherited the caller's real PIPELINES_HARNESS_CONFIG, so the CLI debug prefix failed closed ("refusing to redirect") against the test's ephemeral httptest endpoint. Root's four independent preview tests for the bounded transcript helpers pass; Root's full host race run produced no race warning and only this fixture failure, and go vet ./... passed. Current source therefore isolates the server CLI fixture with the same missing-temp-file t.Setenv("PIPELINES_HARNESS_CONFIG", filepath.Join(t.TempDir(), "harnesses.json")) pattern already used by internal/cli's setup, so the host executable stays authoritative and no Docker probe runs for an ephemeral test endpoint. The production fail-closed redirect guard in internal/harness/prefix.go and its refusing to redirect / loopback-wss regressions are unchanged; the failing verification was fixed rather than removed.

This is a current-source and Root-reported observation. At the time of that observation, independent re-execution of the full go build/go vet/go test matrix (including Docker-only race results) and the final installed-host Docker verification were pending; they are now recorded in the 2026-10-11 FINAL INSTALLED section below, so this phase-specific wording is superseded by that dated install. The packaged reference describes source behavior; a source fix or updated skill is not deployment evidence.

2026-10-11 retained run11 reviewer re-capture (source-level, not a final install)

Independently reading the retained run11 verification at tmp/maintenance-run11-verification.json records the source-level review sequence: an earlier reviewer PTY capture rejected the longer handoff because at 80x30 it scrolled the Pipelines header off the top; the implementer then shortened AuthHandoff to a title plus three short lines; and the re-run reviewer and ui_reviewer stages approved from real current-source Pipelines PTY captures at 80x30 and 100x36 that showed the header and handoff visible above installed Pi 1.1.0's regular interface. A labelled controlled fixture that stripped --tui-mode regular reproduced the pre-fix fullscreen clear, and the review stages report no provider selected, no key entered and no inference. This is source-level reviewer evidence, read from retained output; it was not a deployed/installed state at that time, and the final full Docker install verification was then pending — it is now recorded in the 2026-10-11 FINAL INSTALLED section below, so this phase-specific wording is superseded by that dated install.

Root-reported actual runs (source-level observations supplied by Root for this change and not re-verified here): run7 showed both configured providers' exact handoff markers; runs6/9/10 exercised live maintenance with final-assistant polling/readback succeeding. These are Root observations, not mocked-regression coverage and not a completed installation; no final deployed/installed state is claimed.

2026-10-11 run13 interruption, Docker race limitation and state separation

Four states are kept explicitly separate and must not be conflated:

  1. Current source (this checkout, not deployed): bounded 32 KiB (transcriptBound = 32<<10) redacted transcript previews produce a valid JSON envelope with redaction applied before truncation and a rune-safe 8 KiB preview; session/debug evidence is scoped latest_bounded_evidence with the single bounded latest-assistant readback and at most the 3 most recent session-bearing steps; an unbudgeted V2 audited agent stage records reported usage best-effort through exactly one 2s-bounded final session-aggregate read, leaving missing/invalid usage unknown; and the isolated server CLI fixture uses the missing-temp-file PIPELINES_HARNESS_CONFIG pattern while the production fail-closed redirect guard in internal/harness/prefix.go is unchanged. Pi JSONL is never parsed as one JSON object; native V1 keeps its existing path.
  2. Host verification evidence independently recorded by Root: after the fixture fix, Root ran the full go test -race -count=1 ./... and go vet ./...; after three harness-neutral evidence labels and Step wording changes Root re-ran the affected tui/author/cli race and vet checks, the Python suite (80) and the opt-in compiled picker PTY check (4). These are Root-recorded host results, not re-executed by the authoring stages.
  3. Deployed old image: the running Docker image predates these source changes and does not include them.
  4. Final install (completed 2026-10-11): the final install and installed authoring/provider verification were pending when this separation was written and are now recorded in the 2026-10-11 FINAL INSTALLED section below; an updated skill/client or source checkout is still not deployment evidence on its own.

Docker race limitation (confirmed): the Docker runtime is built with CGO_ENABLED=0 and has no C compiler (the Alpine runtime stage installs no GCC/musl-dev), so go test -race cannot run there. In-Docker reviewers must use ordinary go build/go test/go vet and rely on independently executed host -race runs. Race checks must not be attempted in Docker and no compiler may be installed or searched for.

Run13 interruption (dated evidence, not a fix): the previous reviewer ran find / looking for GCC; the tool timeout left a process in disk sleep, Docker could not stop the container, and an approved Desktop restart recovered run13 as interrupted (executing=false, every step agent_active=false) with no automatic replay. This is an environment/interruption observation only: no general cancellation fix is inferred and process termination is unchanged. Docker startup recovery already sets running steps to interrupted with agent_active=0.

2026-10-11 short-lived CLI inherited-pipe cancellation (current source)

Current source (not deployed) adds one shared, dependency-free constructor in internal/opencode: CommandContext(ctx, executable, args...) sets the single finite os/exec.Cmd.WaitDelay constant ShortLivedWaitDelay = 250ms and is the only place that value is defined. It is reused by the internal/stage OpenCodeClient.call() API/agent path (including Interrupt), opencode.Detect() --help and the native Connection.Messages() export fallback. It bounds draining of inherited output pipes only: after the child exits os/exec waits at most the grace for descendants holding the inherited stdout/stderr pipes, then closes them and reports exec.ErrWaitDelay; it does not terminate or confirm cleanup of descendants.

context.Canceled/context.DeadlineExceeded identities, exec.ErrWaitDelay on the sanitized errors, the ≤64 KiB request / ≤1 MiB response / 8 KiB stderr bounds, boundedBuffer/boundedOutput drain-with-overflow behavior, redaction, strict result validation and cleanup-unconfirmed active ownership (ErrStageCleanup, no replay) are unchanged; no new module dependency is added and a parent that exits successfully while a descendant still holds partial stdout is never treated as success.

Dated synthetic reproduction (tmp/interrupt-verification, synthetic real-subprocess diagnostics, not provider evidence): with Root waiting for the descendant itself to start before cancelling public OpenCodeClient{Backend:"v2"}.Interrupt, two Docker/Linux returns took 3996/3995ms despite context.Canceled (descendant-started.jsonl); the stronger host reproduction took 4003/4000ms (reproduction.log). Added source-level regressions use real os/exec fixtures with start-marker barriers, bounded watchdogs and naturally terminating children for cancel-after-descendant, deadline identity, descendant-held pipes → ErrWaitDelay, ordinary output/exit/redaction, failed-interrupt ownership/no replay, plus Detect/export wait-delay and cancellation cases. This is current source, not deployed: descendant termination, confirmed remote interruption and any run13 Docker disk sleep fix are not claimed, and Docker race remains unavailable (no C compiler).

2026-10-11 FINAL INSTALLED verification (friend-ready preview)

This is the dated receipt for the installed runtime, recorded by Root from tmp/friend-final-verification/summary.json; it supersedes the earlier "pending final install" statements above, which stay as phase-specific history. A normal interactive ./install --model opencode-go/deepseek-v4.1-flash --harness-model opencode-go/deepseek-v4.1-flash exited 0, resolving the current releases live and building fresh dirty source into actual image sha256:1810ab50f3ea3991e7f3548c79250e65a114bb1821eac52efd872b91cf7c3136 (OpenCode 2.0.26 / Pi 1.1.0 / Node 24.18.1) with real tool-free provider validation, ntfy authenticated publish, server + host health, home/workspace entry, preserved authoritative endpoint and ok pre-install database/home backup integrity. Post-install preservation all passed (original definitions 1–4, history 1–16, host/container default Pi and both models/profiles, logical endpoint, service URLs/timezone/ports/models, Pi credential hash, OpenCode connected accounts, ntfy token); neither original-repo commit/push nor credential/data cleanup ran. Installed host-CLI authoring with both Docker harnesses (separate retained drafts, each changing only the worker prompt) performed real ping → pong and validate → validated through the generated Linux /usr/local/bin/pipelines and scoped transport aliases preserving the logical 8081 endpoint and draft metadata, independently inspected from native OpenCode tool outputs and retained Pi JSONL toolResults, with no draft submit/run/publication. Docker-proxy submit on verification-only definition 5 published v2, a second stale v1 draft was rejected as conflict, and Run17 succeeded in 7 seconds with the exact pipelines-docker-verified markers in OpenCode then Pi, both inactive and exactly one completed marker read verified from retained native history (not the latest-assistant preview); best-effort unbudgeted usage was OpenCode 4011 tokens/$0.00029769599999999996 and Pi 1071 tokens/$0.000142002, and /api/session/active was empty after exit. Installed-UI captures (installed-pi-*, pi-signin-*, installed-pi-models-*) cover the actual run17 80x30/120x36 neutral-Pi/green-both-stage renders, the Pi auth handoff/method menu viewed/canceled/exit0 with no sign-in, and the model wizard using copied preferences only while production preferences stayed preserved. Root's independent checks passed: full host go test -race -count=1 ./... and go vet ./..., Python 80 tests (1 optional PTY skipped), Node policy, isolated developer lifecycle fixture (no inference/deploy) and author package tests; earlier independent reviewers ran ordinary Docker build/vet/tests and focused subprocess regressions (no Docker race, no GCC search, no global find). UNVERIFIED here: fresh OAuth/provider login completion, phone delivery, other real providers, the D-disk-sleep fix and active lease expiry; authentication menus and existing-credential inference are not fresh sign-in proof. The stronger synthetic PUBLIC Interrupt child-start barrier (host 252/252 ms, Docker 250/251 ms, context canceled true, bounded true vs ~4 s before) is not proof of descendant termination, remote cleanup or a Docker VM disk fix. Archive extraction, build-identity and compile verification remain pending until Root executes them.

2026-10-11 generated release receipt ignore (source-hygiene regression)

Current source adds two root-anchored .gitignore rules for the installer's machine-local generated artifacts: /.pipelines-releases.json (the receipt path from setup-docker.py release_receipt_path()) and /.pipelines-releases-* (the tempfile.mkstemp prefix=".pipelines-releases-" in record_release_receipt()); no installer/runtime code changed and the pre-existing receipt and other files are retained. A new scripts/test_build_metadata.py case is a controlled, isolated Git fixture using the repository's actual .gitignore: a clean tracked source plus a representative generated receipt/.pipelines-releases-* temporary file leave git status --porcelain empty and build_metadata.docker_args at PIPELINES_DIRTY=false with no -dirty version; editing the tracked source then makes the status non-empty and docker_args report PIPELINES_DIRTY=true with a -dirty version. This is offline source-level regression coverage, not the recorded actual new-account install/provider login above and not live-provider or deployment evidence; this docs-and-ignore change has no rendered UI.

Dated live reference validation: 2026-10-10

Environment: Docker Linux aarch64, OpenCode v2.0.20, opencode-go/deepseek-v4.1-flash. Isolated databases/fixtures and local HTTP notification endpoints; normal published pipelines/repositories were not changed.

  • Incident extraction → report: exact IDs/severities/counts, unchanged handoff, retained session readback and one local success receipt passed (6.63s).
  • File edit → verification: exact final release-file bytes, unchanged fixtures, independently executed cmp, completed read/edit/shell evidence, retained edit diff and no verifier edits passed (14.60s).
  • Missing required input: blocked route ended failed; missing file was not invented; retained read attempt and one local failure receipt passed (4.21s).
  • Accounting/one-token budget initially failed its acceptance check because session zeros let the run succeed. After the tracked-prompt fix, a focused live rerun passed: 952 reported tokens, $0.00016440, budget-exceeded failure, stopped owned session (2.43s). This is not an independently reconciled provider bill.

The earlier three workflows were rechecked from retained artifacts with current assertions, without more inference. There was no fresh full-suite live run of all four workflows after the fix. Do not generalize these results to all five starter templates, other models/providers, or compatible native schema/tool usage. Real ntfy authentication/storage, phone reception, provider outages/timeouts, long-stream interruption and promptness of cleanup were not tested by this trial. The trial ran rebuilt test binaries; it did not deploy the fix to the ordinary server.

Dated live reference validation: 2026-10-11 (run7)

One existing real run7 (2026-10-11) is the retained live evidence dated for this change: it executed both configured harnesses, OpenCode 2.0.26 and Pi 1.1.0, with opencode-go/deepseek-v4.1-flash stages on the Docker server, and both steps succeeded (6s; the retained Pi evidence panel reports 1054/10000 tokens). The captures are retained at tmp/friend-ui-verification/source-pi-80-* and source-pi-120-*, and a current-source TUI inspection of that run recorded actual 80x30 and 120x36 renderings, including the run graph and the bounded evidence panel.

This is a single real run, not a fresh full reference suite. Other providers, real outages/long-stream interruption, fresh OAuth and phone delivery remain not tested; the new-source installer and installed-authoring verification were separately completed by the dated 2026-10-11 FINAL INSTALLED section above. The 2026-10-10 OpenCode v2.0.20 trial above remains dated history.

Reference suite when explicitly requested

The source checkout's TestReferenceWorkflowsLive uses six model stage attempts plus possible compatibility/format-repair calls. Retries are disabled and stages have two-minute timeouts. It uses real existing credentials and retains private runs.db, pinned definitions, fixture directories, run/session JSON and report. The accounting case has a one-token/$0.25 budget; other cases have no spending limit. Test notifications use a local endpoint, not the configured phone topic.

Live inference requires explicit PIPELINES_REFERENCE_LIVE=1, a chosen PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL, and an existing private PIPELINES_REFERENCE_DIR. Get consent for chargeable inference and confirm which execution computer/account to use. Ordinary tests skip live inference. Do not search the repository or run the suite during normal draft authoring.

If the user explicitly asks to run the suite and provides a source checkout:

PIPELINES_REFERENCE_LIVE=1 \
PIPELINES_REFERENCE_MODEL=PROVIDER/MODEL \
PIPELINES_REFERENCE_DIR=/private/approved/artifacts \
go test ./internal/server -run '^TestReferenceWorkflowsLive$' -count=1 -v -timeout 15m

Docker credentials require running the correctly compiled test binary in that container; do not copy host credentials or publish tests to the normal database. TestReferenceDefinitions checks definitions without inference. TestReferenceArtifactsReview rechecks copied run/fixture evidence without inference using PIPELINES_REFERENCE_REVIEW_DIR; select only cases present in that artifact directory. Missing cases skip and must not be reported as passed.