Files
pipelines/docs/installation-verification.md
T

8.7 KiB
Raw Blame History

Verification scope

Regression commands:

go test -race -count=1 ./...
go vet ./...
python3 -m unittest discover -s scripts -p 'test_*.py'

Optional no-inference stable OpenCode checks:

PIPELINES_TEST_OPENCODE_V1=/path/to/opencode \
  go test ./internal/opencode ./internal/author -run TestStable -count=1 -v

The stable checks use isolated home/config/data/cache directories; they never copy login credentials or invoke a model. They exercise session creation, explicit model selection, stored scoped permissions, abort on the same owned service, session deletion, and discovery of the installed global authoring skill. Authoring now references that skill directly instead of preloading instructions.

Native generation/assistant completion parsing is regression-tested against a controlled HTTP server. Full live inference, permission enforcement by real model tool calls, cancellation during inference, Docker provider authentication, privileged system-wide copying, and OS service-manager registration are not tested.

The full Docker installer was exercised in an isolated Compose project with real ntfy and Pipelines containers: image build, account/ACL/token bootstrap, Compose health checks, authenticated notification publishing from the Pipelines container, CLI status, reinstall preservation of .env/token, and an empty pipeline database. Provider-account inspection/login completion was mocked in that installer test. A separate real-container PTY invocation of opencode auth login displayed its searchable provider choices and was canceled without credentials. Provider login, model inference and phone delivery remain not tested. LAN-default checks cover physical-interface selection on Linux/macOS, rejection of loopback/public/link-local/Tailscale fallback addresses, prompt-free configuration, timezone fallback, and clean Ctrl+C exit with configuration retained. Actual host LAN detection and .env generation were executed without networking input. The isolated Docker ntfy endpoint was also reached through the detected LAN IP: health succeeded, anonymous publishing was denied, and authenticated publishing succeeded. These host-side checks do not prove reception on another device.

Consequential-action regressions check run/retry/cancellation and CLI publication/ draft deletion confirmation, default-No behavior, JSON stdout cleanliness, and --yes for explicit automation. The compiled CLI/TUI PTY check verified that q does not exit, :quit exits, canceled retry prompts create no run, and an explicit y starts a new retry with the original input/directory and pinned version.

Autostart regression tests exercise both service-definition formats, scoped environment capture, escaping, private files, refusal to overwrite unmanaged files/symlinks, preview without side effects, and lifecycle calls using a mock OS manager. The installed macOS CLI's preview was validated with plutil -lint. GOOS=linux GOARCH=amd64 go build ./cmd/... also passed. Actual launchd/systemd registration, logout/login restart, and Linux lingering remain not tested.

An isolated installed-CLI check started the server in one directory and invoked pipelines run from two other directories. With a mock OpenCode executable, both runs completed and recorded/executed in the client invocation directory. The autostart preview required no repository configuration and passed plutil -lint. CLI and dashboard regression tests also check invocation-directory selection.

The installation tests were also executed as a Linux binary in Debian with systemd installed. The native systemd-analyze verify regression passed for the generated unit, including a directory containing spaces, quotes, and %/$. This reproduced and fixed the invalid quoted WorkingDirectory setting. A container syntax check does not prove live user-service startup or login restart.

Authoring regression tests exercise idempotent skill installation, preservation of user modifications, one direct OpenCode launch with a short skill/context reference, and an unnamed new draft without CLI questions. TUI integration checks browse a CLI-style retained draft, validate through the server, require explicit submission confirmation, preserve create-only guards, and reject a different server.

An isolated installed-binary PTY check also opened a CLI-created draft in the TUI, reopened the same file through a mock OpenCode executable, and explicitly submitted it while retaining the draft. Plain unnamed CLI creation invoked the mock exactly once. Real model-driven authoring/questions and native skill loading during a model turn remain not tested; global skill discovery was tested without inference.

An installed-binary integration check executed a Bash/QML fixture through a mock OpenCode backend: it changed README/QML, created a new text file, and actually ran bash -n install.sh. Both JSON and PTY supplied-task runs started without approval flags, file lists, or typed confirmation, in the invocation directory rather than the server directory. Permission regression tests cover language-neutral shell, project file types/new files, optional narrower scopes, and outside-edit denials. Real model/tool-permission enforcement remains not tested. Shell uses the host user’s authority; these policies are not an OS sandbox.

A compiled-binary PTY integration check exercised the fuzzy editor picker in standalone pipelines-tui, choosing both OpenCode and nvim with an intentionally invalid EDITOR override. It checked confirmed TUI draft deletion, CLI deletion without touching run history, the shared run graph, and D launching a fresh debugging conversation containing authoritative CLI queries. Mock OpenCode/nvim processes recorded the actual arguments and directory. Real model-driven run investigation and live OpenCode UI attachment remain not tested. Focused tests cover elapsed duration, live graph updates via run responses, shared-renderer equivalence, loop visits, viewport resizing, and retry rejection during execution cleanup even after a terminal status is persisted.

Result-recovery regression tests actually execute a mock audited agent that edits a file once, returns malformed JSON, and receives a separate deny-all formatting session. They check exact parser feedback, no agent/tool replay, preserved approval/evidence, rejection of changed verdicts or omitted evidence, and optional file/command permission rules. A native stable no-inference live contract test accepted a json_schema message with retryCount=0 and noReply. Real model-driven schema generation, formatter behavior and tool-permission enforcement remain not tested.

Status/dashboard checks execute server and fake ntfy HTTP health endpoints, CLI JSON/plain status and statistics, all-history SQL aggregation with more than 50 runs, interrupted/unused pipeline counts, zero-filled daily activity, responsive dashboard columns, and status popup navigation from a run without losing its selection. Compiled-binary PTY integration also exercises pipelines status, pipelines stats, :status, dashboard activity and Home's all-runs r shortcut. Candidate URL reachability and phone reception are not tested and are not reported as verified by the UI.

Editor-return checks query actual published history before showing confirmation; tests cover a submitted draft followed by a newer publication, unsent edits, failed/stale verification, and immutable history. Compiled CLI/TUI PTY integration uses a mock OpenCode editor that submits with the real CLI, then verifies automatic reload to version 2 without another submission prompt. Another mock editor only saves changes and still requires confirmation. A real local V2 no-inference create/wait/delete contract succeeded; the reported remote planner /wait failure could not be reproduced here. Safe CLI failure classification/exit-code reporting was executed with a failing subprocess fixture.

New-pipeline picker checks cover both editor choices, fuzzy/Tab navigation, cancel-without-creating, retained source/server/create-only metadata and template source preservation. Compiled CLI/TUI PTY integration launches a new nvim draft (confirmation required), then a new mock OpenCode author that names and publishes through the real CLI (reloaded automatically, no extra submit prompt).

Soft-theme regression checks execute light/dark true-color rendering, ANSI-color fallback, borderless Home/popups, stage-color identity on tinted selections, continuous card backgrounds around nested colored labels, narrow-terminal dimensions, and local spinner animation across pending views without extra RPCs. Actual sample-data renders can be exported with PIPELINES_TUI_PREVIEW_DIR=PATH go test -run TestSoftThemeRenderPreviews ./internal/tui for Home, New and pipeline views in both themes. Compiled CLI/TUI PTY integration passed with the new styling.