Files
pipelines/docs/installation-verification.md
T

29 KiB
Raw Blame History

Verification scope

Regression commands:

go test -race -count=1 ./...
go vet ./...
python3 -m unittest discover -s scripts -p 'test_*.py'

Optional no-inference stable OpenCode checks:

PIPELINES_TEST_OPENCODE_V1=/path/to/opencode \
  go test ./internal/opencode ./internal/author -run TestStable -count=1 -v

The stable checks use isolated home/config/data/cache directories; they never copy login credentials or invoke a model. They exercise session creation, explicit model selection, stored scoped permissions, abort on the same owned service, session deletion, and discovery of the installed global authoring skill. Authoring now references that skill directly instead of preloading instructions.

Native generation/assistant completion parsing is regression-tested against a controlled HTTP server. Full live inference, permission enforcement by real model tool calls, cancellation during inference, Docker provider authentication, privileged system-wide copying, and OS service-manager registration are not tested.

The full Docker installer was exercised in an isolated Compose project with real ntfy and Pipelines containers: image build, account/ACL/token bootstrap, Compose health checks, authenticated notification publishing from the Pipelines container, CLI status, reinstall preservation of .env/token, and an empty pipeline database. Provider-account inspection/login completion was mocked in that installer test. A separate real-container PTY invocation of opencode auth login displayed its searchable provider choices and was canceled without credentials. Provider login, model inference and phone delivery remain not tested. LAN-default checks cover physical-interface selection on Linux/macOS, rejection of loopback/public/link-local/Tailscale fallback addresses, prompt-free configuration, timezone fallback, and clean Ctrl+C exit with configuration retained. Actual host LAN detection and .env generation were executed without networking input. The isolated Docker ntfy endpoint was also reached through the detected LAN IP: health succeeded, anonymous publishing was denied, and authenticated publishing succeeded. These host-side checks do not prove reception on another device. Docker bindings now default to 0.0.0.0 for both services. Reinstall tests check normalization of old bind settings without replacing URLs/timezone/ports/tokens; docker compose config --format json confirmed both host bindings are 0.0.0.0.

Consequential-action regressions check run/retry/cancellation and CLI publication/ draft deletion confirmation, default-No behavior, JSON stdout cleanliness, and --yes for explicit automation. The compiled CLI/TUI PTY check verified that q does not exit, :quit exits, canceled retry prompts create no run, and an explicit y starts a new retry with the original input/directory and pinned version.

Autostart regression tests exercise both service-definition formats, scoped environment capture, escaping, private files, refusal to overwrite unmanaged files/symlinks, preview without side effects, and lifecycle calls using a mock OS manager. The installed macOS CLI's preview was validated with plutil -lint. GOOS=linux GOARCH=amd64 go build ./cmd/... also passed. Actual launchd/systemd registration, logout/login restart, and Linux lingering remain not tested.

An isolated installed-CLI check started the server in one directory and invoked pipelines run from two other directories. With a mock OpenCode executable, both runs completed and recorded/executed in the client invocation directory. The autostart preview required no repository configuration and passed plutil -lint. CLI and dashboard regression tests also check invocation-directory selection.

The installation tests were also executed as a Linux binary in Debian with systemd installed. The native systemd-analyze verify regression passed for the generated unit, including a directory containing spaces, quotes, and %/$. This reproduced and fixed the invalid quoted WorkingDirectory setting. A container syntax check does not prove live user-service startup or login restart.

Authoring regression tests exercise idempotent skill installation, preservation of user modifications, one direct OpenCode launch with a short skill/context reference, and an unnamed new draft without CLI questions. TUI integration checks browse a CLI-style retained draft, validate through the server, require explicit submission confirmation, preserve create-only guards, and allow portable drafts to be imported on a different server without inheriting the origin's version number.

An isolated installed-binary PTY check also opened a CLI-created draft in the TUI, reopened the same file through a mock OpenCode executable, and explicitly submitted it while retaining the draft. Plain unnamed CLI creation invoked the mock exactly once. Real model-driven authoring/questions and native skill loading during a model turn remain not tested; global skill discovery was tested without inference.

An installed-binary integration check executed a Bash/QML fixture through a mock OpenCode backend: it changed README/QML, created a new text file, and actually ran bash -n install.sh. Both JSON and PTY supplied-task runs started without approval flags, file lists, or typed confirmation, in the invocation directory rather than the server directory. Permission regression tests cover language-neutral shell, project file types/new files, optional narrower scopes, and outside-edit denials. Real model/tool-permission enforcement remains not tested. Shell uses the host user’s authority; these policies are not an OS sandbox.

A compiled-binary PTY integration check exercised the fuzzy editor picker in standalone pipelines-tui, choosing both OpenCode and nvim with an intentionally invalid EDITOR override. It checked confirmed TUI draft deletion, CLI deletion without touching run history, the shared run graph, and D launching a fresh debugging conversation containing authoritative CLI queries. Mock OpenCode/nvim processes recorded the actual arguments and directory. Real model-driven run investigation and live OpenCode UI attachment remain not tested. Focused tests cover elapsed duration, live graph updates via run responses, shared-renderer equivalence, loop visits, viewport resizing, and retry rejection during execution cleanup even after a terminal status is persisted.

Result-recovery regression tests actually execute a mock audited agent that edits a file once, returns malformed JSON, and receives a separate deny-all formatting session. They check exact parser feedback, no agent/tool replay, preserved approval/evidence, rejection of changed verdicts or omitted evidence, and optional file/command permission rules. A native stable no-inference live contract test accepted a json_schema message with retryCount=0 and noReply. Real model-driven schema generation, formatter behavior and tool-permission enforcement remain not tested.

Status/dashboard checks execute server and fake ntfy HTTP health endpoints, CLI JSON/plain status and statistics, all-history SQL aggregation with more than 50 runs, interrupted/unused pipeline counts, zero-filled daily activity, responsive dashboard columns, and status popup navigation from a run without losing its selection. Compiled-binary PTY integration also exercises pipelines status, pipelines stats, :status, dashboard activity and Home's all-runs r shortcut. Candidate URL reachability and phone reception are not tested and are not reported as verified by the UI.

Editor-return checks query actual published history before showing confirmation; tests cover a submitted draft followed by a newer publication, unsent edits, failed/stale verification, and immutable history. Compiled CLI/TUI PTY integration uses a mock OpenCode editor that submits with the real CLI, then verifies automatic reload to version 2 without another submission prompt. Another mock editor only saves changes and still requires confirmation. A real local V2 no-inference create/wait/delete contract succeeded; the reported remote planner /wait failure could not be reproduced here. Later dated evidence (2026-10-10) did confirm the CLI-side V2 .../wait request timeout after about six minutes (HTTP 499) on a long stage; current source therefore settles V2 sessions with bounded /api/session/active polling instead. Safe CLI failure classification/exit-code reporting was executed with a failing subprocess fixture.

New-pipeline picker checks cover both editor choices, fuzzy/Tab navigation, cancel-without-creating, retained source/server/create-only metadata and template source preservation. Compiled CLI/TUI PTY integration launches a new nvim draft (confirmation required), then a new mock OpenCode author that names and publishes through the real CLI (reloaded automatically, no extra submit prompt).

Soft-theme regression checks execute light/dark true-color rendering, ANSI-color fallback, borderless Home/popups, stage-color identity on tinted selections, continuous card backgrounds around nested colored labels, narrow-terminal dimensions, and local spinner animation across pending views without extra RPCs. Actual sample-data renders can be exported with PIPELINES_TUI_PREVIEW_DIR=PATH go test -run TestSoftThemeRenderPreviews ./internal/tui for Home, New and pipeline views in both themes. Compiled CLI/TUI PTY integration passed with the new styling.

2026-10-11 picker, resolver and installer coverage

These items describe current source and its offline/structural regression tests unless they explicitly name an executed live result; the independent live run for this change is still pending.

  • resolve-releases performs one bounded GET /repos/{owner}/{repo}/git/matching-refs/tags/v2 request, which the GitHub REST docs describe as a complete array with no pagination parameters. Offline fixtures cover a 100+ ref response, the explicit MAX_REFS/MAX_BODY caps, ignored prerelease/build tags, malformed refs and a missing newest-tag digest failing closed with no older-tag fallback. The tags/list cross-check and all pagination (per_page/page/Link) claims are removed.
  • internal/harness picker tests check that text cursor blinks and unrelated messages preserve the selected index while a real filter change resets it, that reduced ANSI output keeps the selected row readable (caret/bold, no colliding background) while TrueColor keeps the adaptive background, and that 40x16/80x24/120x36 layouts stay within the measured height with the step/heading and selected row visible, including a long no-match query.
  • The compiled picker PTY check is opt-in (PIPELINES_HARNESS_PTY_DIR); it now waits through blink intervals and asserts the exact saved and arrow-scrolled model. It was not executed as part of this documentation change.
  • setup-docker.py accepts OpenCode's actual opencode v2.0.26 probe token and compares source identity only for a full install; model/login-only reconfiguration compares the verified release image identity, so a changed source no longer forces a reconfigure rebuild while a dirty install still always invokes the Docker build. The image now includes python3.
  • Python fixtures bind temporary HOME/USERPROFILE directories, so the installer/configure suites pass under the container default /home/pipelines without weakening host_workspaces guards (which still reject /, /data, /workspace, /home/pipelines and home == repository).
  • Still not tested here: a real Docker rebuild/reconfigure against live registries, a real 40x16/80x24/120x36 terminal capture of the compiled picker, phone delivery, and real model inference.

2026-10-11 retained-run fixture isolation and final-validation envelope

Root-reported runtime evidence supplied for this change (not re-executed by the authoring stages): the existing seeded-profile regression go test ./tmp/preview-verification -run TestServerCLIFixtureDoesNotInheritUserDockerProfile -count=1 failed because the child internal/server integration test inherited the caller's real Docker-linked PIPELINES_HARNESS_CONFIG, so the CLI debug prefix failed closed ("refusing to redirect") against the test's ephemeral httptest endpoint. Root's four preview tests for the bounded transcript helpers pass, Root's full host race run showed no race warning and only this fixture failure, and go vet ./... passed. Current source isolates the server CLI fixture with the existing missing-temp-file t.Setenv("PIPELINES_HARNESS_CONFIG", filepath.Join(t.TempDir(), "harnesses.json")) pattern, leaving the production fail-closed redirect guard in internal/harness/prefix.go and its regressions unchanged and keeping the real-profile isolation regression meaningful rather than removed.

Current-source behavior, not a deployment: independent re-execution of the full go build/go vet/go test matrix (Docker-only race results stay Docker-only) and the final installed-host Docker install verification are still pending; no installed/running server state is claimed here.

2026-10-11 run13 interruption, Docker race limitation and state separation

Four states are kept explicitly separate and must not be conflated:

  1. Current source (this checkout, not deployed): the retained maintenance changes are present here — internal/stage/diagnostics.go BoundTranscript/transcriptBound = 32<<10 turns an oversized transcript into a valid, bounded JSON envelope with credential redaction applied before any head is taken and a rune-safe 8 KiB preview; internal/stage/evidence.go scopes session evidence as latest_bounded_evidence using the single bounded finalAssistantReadback (never the recent-20 all-message fetch); internal/server/evidence.go reads at most the 3 most recent session-bearing steps; settleV2Usage in internal/stage/agent.go issues exactly one 2s-bounded session-aggregate read after an unbudgeted V2 agent stage and leaves missing/invalid usage unknown; and the isolated server CLI fixture uses the missing-temp-file t.Setenv("PIPELINES_HARNESS_CONFIG", …) pattern with the production fail-closed redirect guard in internal/harness/prefix.go unchanged. Pi JSONL is never parsed as one JSON object and native V1 keeps its existing path.
  2. Host verification evidence independently recorded by Root: after the fixture fix, Root ran the full go test -race -count=1 ./... (no race warning, only the fixture failure before the fix) and go vet ./...; after the three harness-neutral evidence labels and Step wording changes Root re-ran the affected tui/author/cli race and vet checks, the Python suite (80) and the opt-in compiled picker PTY check (4). These are Root-recorded host results; this authoring change did not re-execute them.
  3. Deployed old image: the running Docker image predates these source changes, so the deployed server does not include them.
  4. Pending final install: the final install and installed authoring/provider verification remain pending. An updated skill/client or source checkout is not deployment evidence.

Docker race limitation (confirmed): the Docker runtime is built with CGO_ENABLED=0 (Dockerfile) and has no C compiler (the Alpine runtime stage installs no GCC/musl-dev), so go test -race cannot run there. In-Docker reviewers must use ordinary go build/go test/go vet; race coverage comes from the independently executed host -race runs above. Race checks must not be attempted in Docker and no compiler may be installed or searched for.

Run13 interruption (dated evidence, not a fix): the previous reviewer ran find / looking for GCC; the tool timeout left a process in disk sleep, Docker could not stop the container, and an approved Desktop restart recovered run13 as interrupted (executing=false, every step agent_active=false) with no automatic replay. Docker startup recovery already sets running steps to interrupted with agent_active=0. This is an environment/interruption observation only: no general cancellation fix is inferred and process termination is unchanged.

2026-10-11 short-lived CLI inherited-pipe cancellation (current source)

Current source (this checkout, not deployed) adds one shared, dependency-free constructor in internal/opencode: CommandContext(ctx, executable, args...) sets the single finite os/exec.Cmd.WaitDelay constant ShortLivedWaitDelay = 250ms and is the only place that value is defined. It is reused by all three short-lived, context-controlled CLI call sites — the internal/stage OpenCodeClient.call() API/agent path (including Interrupt), opencode.Detect()'s --help probe and the native Connection.Messages() export fallback. It bounds draining of inherited output pipes only: after the child exits, os/exec waits at most the grace for descendants that still hold the inherited stdout/stderr pipes, then closes them and reports exec.ErrWaitDelay. It does not terminate, signal or confirm cleanup of descendants.

Existing identities and bounds are preserved: context.Canceled and context.DeadlineExceeded are returned unchanged when the context is done, exec.ErrWaitDelay survives on the sanitized openCodeError cause and on the Detect/Messages errors, request (≤64 KiB), response (≤1 MiB), 8 KiB stderr and boundedBuffer/boundedOutput drain-with-overflow behavior are unchanged, error text stays free of provider bodies/prompts/credentials/CLI logs, and a failed interrupt still leaves the session active and joins ErrStageCleanup with no replay. A successful parent exit whose partial stdout is still held open by a descendant is never parsed as a successful result. No new module dependency is added.

Dated synthetic reproduction (tmp/interrupt-verification, synthetic real-subprocess diagnostics, not provider evidence): Root waited for the descendant itself to start before cancelling public OpenCodeClient{Backend:"v2"}.Interrupt. Two Docker/Linux returns took 3996ms/3995ms despite context.Canceled (tmp/interrupt-verification/descendant-started.jsonl); the stronger host reproduction took 4003ms/4000ms (tmp/interrupt-verification/reproduction.log). These observed the previous behavior and motivated the shared finite grace.

Added regression coverage (source-level/mocked; internal/stage and internal/opencode, using real os/exec fixtures with start-marker barriers, bounded watchdogs and naturally terminating owned children): cancel after a descendant starts; deadline error identity; successful parent exit with descendant-held pipes → exec.ErrWaitDelay with no success text; ordinary help/API/export output, non-zero exit and redaction; and a failed interrupt that retains cleanup-unconfirmed active ownership and performs no replay.

Tested vs not deployed: this describes current source. The running Docker image predates these changes, so the fix is not deployed. Descendant termination, confirmed remote interruption and any fix for the run13 Docker disk sleep are explicitly not claimed. Docker race remains unavailable (the runtime has no C compiler), so in-Docker reviewers use ordinary go build/go test/go vet and host -race runs stay host-only.

2026-10-11 FINAL INSTALLED verification (friend-ready preview)

This is the dated receipt for the friend-ready preview runtime, recorded by Root from tmp/friend-final-verification (authoritative summary.json plus the installed PNG captures). It supersedes the phase-specific "pending final install" statements in the earlier dated sections above; those observations stay as history and are not deleted.

A normal interactive ./install with both GoDeepseek flags —

./install --model opencode-go/deepseek-v4.1-flash \
  --harness-model opencode-go/deepseek-v4.1-flash

— exited 0. The installer resolved the current OpenCode/Pi releases live, built from the fresh dirty source, and produced actual image sha256:1810ab50f3ea3991e7f3548c79250e65a114bb1821eac52efd872b91cf7c3136 with OpenCode 2.0.26 / Pi 1.1.0 / Node 24.18.1. The install made a real tool-free provider validation and a real ntfy authenticated publish, then confirmed server + host health, home/workspace entry and the saved authoritative endpoint preserved. Pre-install database/home backup integrity was ok.

Post-install preservation checks all passed: original definitions 1–4, run history 1–16, host/container default Pi and both models/profiles, the logical endpoint, service URLs/timezone/ports/models, the Pi credential hash, the OpenCode connected accounts and the ntfy token. Neither the original repository commit/push nor any credential/data cleanup was performed.

Installed host-CLI authoring was actually used with both Docker harnesses: separate retained drafts, each changing only the worker prompt, performed a real shell CLI ping → pong and validate → validated through the generated Linux /usr/local/bin/pipelines plus scoped transport aliases that preserved the original logical 8081 endpoint and draft metadata. Root independently inspected the native OpenCode completed shell tool outputs and the Pi retained JSONL toolResults and validated both drafts; there was no submit/run/publication of the authoring drafts.

Docker-proxy submit on verification-only definition 5 → published v2; a second stale v1 draft was rejected as conflict, and the original definitions stayed untouched. Run17 (definition 5 v2) succeeded in 7 seconds: OpenCode then Pi reported the exact pipelines-docker-verified markers, both went inactive, and OpenCode's exactly one completed read of the marker file was verified from the retained native history, not the latest-assistant preview. Unbudgeted reported usage was recorded best-effort and known here: OpenCode 4011 tokens / $0.00029769599999999996 and Pi 1071 tokens / $0.000142002; unknown usage remains allowed by design. GET /api/session/active was empty after the sessions exited.

Installed-UI captures under tmp/friend-final-verification (installed-pi-*, pi-signin-*, installed-pi-models-*): actual run17 at 80x30 and 120x36 showed neutral Pi evidence and the green both-stage graph; the actual installed Pi auth handoff/method menu at 80x30 and 100x36 was viewed, canceled and exited 0 with no sign-in; the actual installed Pi model wizard at 80x30 viewed the saved current model, filtered, confirmed the first with Enter and the second with Esc (exit 1, expected cancel) using copied preferences only, while the actual production preferences were preserved.

Root's independent checks: full host go test -race -count=1 ./... and go vet ./... exit 0; Python suite 80 tests, exit 0, 1 optional PTY; Node policy exit 0; the isolated developer lifecycle fixture exit 0 with no inference/deploy; author package tests exit 0. Earlier independent reviewer runs performed the ordinary Docker full build/vet/tests and focused subprocess regressions (ordinary build/vet/test only — no Docker race, no GCC search, no global find).

Explicitly UNVERIFIED for this install: fresh OAuth/provider login completion, phone delivery, other real providers, the D-disk-sleep fix and active lease expiry. Authentication menus and existing-credential inference are not fresh sign-in proof. The shared finite WaitDelay stronger synthetic PUBLIC Interrupt child-start barrier (host 252/252 ms, Docker 250/251 ms, context canceled true, bounded true vs ~4 s before) is a synthetic diagnostic and is not proof of descendant termination, remote cleanup or a Docker VM disk fix. Archive extraction, build-identity and compile verification remain pending until Root executes it and are not claimed here.

2026-10-11 generated release receipt ignore (source-hygiene regression)

Current source adds two root-anchored .gitignore rules for the installer's machine-local generated artifacts: /.pipelines-releases.json (the receipt path from setup-docker.py release_receipt_path()) and /.pipelines-releases-* (the tempfile.mkstemp prefix=".pipelines-releases-" in record_release_receipt()). No installer/runtime code changed, and the pre-existing receipt and every other file are retained.

Controlled isolated Git fixture (offline, credential-free, no network, Docker, provider or model call): a new scripts/test_build_metadata.py case copies the repository's actual .gitignore into a temp dir, git inits it, commits one tracked source file, then writes a representative .pipelines-releases.json and a .pipelines-releases-abc123 temporary file. git status --porcelain --untracked-files=normal is empty and the existing build_metadata.docker_args helper returns PIPELINES_DIRTY=false with a version lacking -dirty; after editing the tracked source the status is non-empty and docker_args returns PIPELINES_DIRTY=true with a -dirty version. This shows the generated receipt no longer dirties a clean checkout while dirty source still rebuilds.

This controlled fixture is not the actual new-account install/provider login recorded in the 2026-10-11 FINAL INSTALLED section above, which this change does not re-run. It is docs-and-ignore only, with no rendered UI and no live install or provider evidence.

2026-10-12 run22 success, run23 live CSV fixture and run24 interruption

Source/structural and live-fixture evidence is kept separate from runtime deployment; nothing in this section is a deployment claim.

  • run22 (structural/source checks, success): the retained run22 verification completed its structural/source checks of the scoped template and prompt changes and passed.

  • run23 (live Pi CSV fixture): a real exact-template Pi CSV run fetched the official Python csv documentation through the stage's conditional official-source webfetch and implemented CSV parsing with the standard library's csv.reader through an existing shared helper, adding no duplicate preview parser; both the implementer and reviewer checks actually passed. Root independently checked full retained native tool results and protected-file hashes against its saved baseline, rather than relying on model-written reports.

  • run24 (interrupted, not a pass): the run24 implementation completed, but its reviewer ignored the scope instruction and ran a prohibited filesystem-root scan (find /), leaving a process in uninterruptible D-state and stalling Docker mount/snapshot reads. Root canceled the run, made a consistent SQLite backup (integrity ok), recovered Docker with the supported stop/start path, and restored the existing ntfy/Pipelines services. The original definitions, run history, credentials and settings were verified unchanged. run24 remains retained as interrupted, not a success.

  • run25 (live Pi source-only review): the reviewer identified duplicated implementation, a hardcoded special case, suppressed errors and test gaps. Root inspected retained native calls: only local read/search tools ran, with no shell/edit/network calls. Original fixture hashes were unchanged; runtime behavior was explicitly marked untested.

  • run26 (final maintenance review, success): the scoped plan/implement/review workflow passed package tests, vet, build and formatting checks. Root inspected the actual retained outputs and changes. No rendered UI change was claimed.

The scoped prompt guidance asks stages to keep inspection/verification to relevant repository paths and supported commands. That is a working instruction, not an OS sandbox: it neither fixes Docker VM/disk bugs nor universally enforces tool boundaries. At these recorded checkpoints the source changes had not been deployed. Catalog deployment requires a separate check of the running server and its actual template contents; these observations do not establish publication or deployment.

2026-10-12 final template verification and Docker catalog deployment

The final three-stage template ran as live Pi run27 (verification definition7, version2) in a fresh disposable CSV fixture. Root inspected full retained native results: the planner fetched official Python CSV documentation, the implementer used the standard library through the existing shared parser, and implementer and reviewer independently ran python3 checks.py successfully. Root reran those checks, verified the protected fixture hashes and planner-owned review criteria, and confirmed the preview reused the shared parser.

The Docker server was then refreshed from clean source commit 143f237268dbde2efd6c5ce5722a3e1fd7e08c9c, retaining OpenCode 2.0.26, Pi 1.1.0 and Node 24.18.1. The running server reported that exact commit and dirty=false; all five served templates matched current source with the saved Pi/model defaults. A consistent pre-deployment database backup passed integrity checks. Every stored definition version/source, retained run ID, saved setting, endpoint and provider credential/account checked by Root was preserved. This record is a documentation follow-up to that deployed build; it changes no runtime code or template source.

These bounded fixtures establish the observed behavior on this instance, not a guarantee of every future model review, provider, task or platform. They do not establish fresh account sign-in or phone notification reception.