Files
pipelines/docs/installation-verification.md
T

146 lines
9.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Verification scope
Regression commands:
```sh
go test -race -count=1 ./...
go vet ./...
python3 -m unittest discover -s scripts -p 'test_*.py'
```
Optional no-inference stable OpenCode checks:
```sh
PIPELINES_TEST_OPENCODE_V1=/path/to/opencode \
go test ./internal/opencode ./internal/author -run TestStable -count=1 -v
```
The stable checks use isolated home/config/data/cache directories; they never
copy login credentials or invoke a model. They exercise session creation, explicit
model selection, stored scoped permissions, abort on the same owned service,
session deletion, and discovery of the installed global authoring skill. Authoring
now references that skill directly instead of preloading instructions.
Native generation/assistant completion parsing is regression-tested against a
controlled HTTP server. Full live inference, permission enforcement by real model
tool calls, cancellation during inference, Docker provider authentication, privileged
system-wide copying, and OS service-manager registration are **not tested**.
The full Docker installer was exercised in an isolated Compose project with real
ntfy and Pipelines containers: image build, account/ACL/token bootstrap, Compose
health checks, authenticated notification publishing from the Pipelines container,
CLI status, reinstall preservation of `.env`/token, and an empty pipeline database.
Provider-account inspection/login completion was mocked in that installer test.
A separate real-container PTY invocation of `opencode auth login` displayed its
searchable provider choices and was canceled without credentials. Provider login,
model inference and phone delivery remain **not tested**.
LAN-default checks cover physical-interface selection on Linux/macOS, rejection of
loopback/public/link-local/Tailscale fallback addresses, prompt-free configuration,
timezone fallback, and clean Ctrl+C exit with configuration retained. Actual host
LAN detection and `.env` generation were executed without networking input.
The isolated Docker ntfy endpoint was also reached through the detected LAN IP:
health succeeded, anonymous publishing was denied, and authenticated publishing
succeeded. These host-side checks do not prove reception on another device.
Docker bindings now default to `0.0.0.0` for both services. Reinstall tests check
normalization of old bind settings without replacing URLs/timezone/ports/tokens;
`docker compose config --format json` confirmed both host bindings are `0.0.0.0`.
Consequential-action regressions check run/retry/cancellation and CLI publication/
draft deletion confirmation, default-No behavior, JSON stdout cleanliness, and
`--yes` for explicit automation. The compiled CLI/TUI PTY check verified that `q`
does not exit, `:quit` exits, canceled retry prompts create no run, and an explicit
`y` starts a new retry with the original input/directory and pinned version.
Autostart regression tests exercise both service-definition formats, scoped
environment capture, escaping, private files, refusal to overwrite unmanaged
files/symlinks, preview without side effects, and lifecycle calls using a mock OS
manager. The installed macOS CLI's preview was validated with `plutil -lint`.
`GOOS=linux GOARCH=amd64 go build ./cmd/...` also passed. Actual launchd/systemd
registration, logout/login restart, and Linux lingering remain **not tested**.
An isolated installed-CLI check started the server in one directory and invoked
`pipelines run` from two other directories. With a mock OpenCode executable, both
runs completed and recorded/executed in the client invocation directory. The
autostart preview required no repository configuration and passed `plutil -lint`.
CLI and dashboard regression tests also check invocation-directory selection.
The installation tests were also executed as a Linux binary in Debian with
systemd installed. The native `systemd-analyze verify` regression passed for the
generated unit, including a directory containing spaces, quotes, and `%`/`$`.
This reproduced and fixed the invalid quoted `WorkingDirectory` setting. A
container syntax check does not prove live user-service startup or login restart.
Authoring regression tests exercise idempotent skill installation, preservation of
user modifications, one direct OpenCode launch with a short skill/context reference,
and an unnamed new draft without CLI questions. TUI integration checks browse a
CLI-style retained draft, validate through the server, require explicit submission
confirmation, preserve create-only guards, and reject a different server.
An isolated installed-binary PTY check also opened a CLI-created draft in the TUI,
reopened the same file through a mock OpenCode executable, and explicitly submitted
it while retaining the draft. Plain unnamed CLI creation invoked the mock exactly
once. Real model-driven authoring/questions and native skill loading during a
model turn remain **not tested**; global skill discovery was tested without inference.
An installed-binary integration check executed a Bash/QML fixture through a mock
OpenCode backend: it changed README/QML, created a new text file, and actually ran
`bash -n install.sh`. Both JSON and PTY supplied-task runs started without approval
flags, file lists, or typed confirmation, in the invocation directory rather than
the server directory. Permission regression tests cover language-neutral shell,
project file types/new files, optional narrower scopes, and outside-edit denials.
Real model/tool-permission enforcement remains **not tested**. Shell uses the host
user’s authority; these policies are not an OS sandbox.
A compiled-binary PTY integration check exercised the fuzzy editor picker in
standalone `pipelines-tui`, choosing both OpenCode and nvim with an intentionally
invalid EDITOR override. It checked confirmed TUI draft deletion, CLI deletion
without touching run history, the shared run graph, and `D` launching a fresh
debugging conversation containing authoritative CLI queries. Mock OpenCode/nvim
processes recorded the actual arguments and directory. Real model-driven run
investigation and live OpenCode UI attachment remain **not tested**.
Focused tests cover elapsed duration, live graph updates via run responses,
shared-renderer equivalence, loop visits, viewport resizing, and retry rejection
during execution cleanup even after a terminal status is persisted.
Result-recovery regression tests actually execute a mock audited agent that
edits a file once, returns malformed JSON, and receives a separate deny-all
formatting session. They check exact parser feedback, no agent/tool replay,
preserved approval/evidence, rejection of changed verdicts or omitted evidence,
and optional file/command permission rules. A native stable no-inference live
contract test accepted a `json_schema` message with `retryCount=0` and `noReply`.
Real model-driven schema generation, formatter behavior and tool-permission
enforcement remain **not tested**.
Status/dashboard checks execute server and fake ntfy HTTP health endpoints,
CLI JSON/plain status and statistics, all-history SQL aggregation with more than
50 runs, interrupted/unused pipeline counts, zero-filled daily activity, responsive
dashboard columns, and status popup navigation from a run without losing its
selection. Compiled-binary PTY integration also exercises `pipelines status`,
`pipelines stats`, `:status`, dashboard activity and Home's all-runs `r` shortcut.
Candidate URL reachability and phone reception are **not tested** and are not
reported as verified by the UI.
Editor-return checks query actual published history before showing confirmation;
tests cover a submitted draft followed by a newer publication, unsent edits,
failed/stale verification, and immutable history. Compiled CLI/TUI PTY integration
uses a mock OpenCode editor that submits with the real CLI, then verifies automatic
reload to version 2 without another submission prompt. Another mock editor only
saves changes and still requires confirmation. A real local V2 no-inference
create/wait/delete contract succeeded; the reported remote planner `/wait` failure
could not be reproduced here. Safe CLI failure classification/exit-code reporting
was executed with a failing subprocess fixture.
New-pipeline picker checks cover both editor choices, fuzzy/Tab navigation,
cancel-without-creating, retained source/server/create-only metadata and template
source preservation. Compiled CLI/TUI PTY integration launches a new nvim draft
(confirmation required), then a new mock OpenCode author that names and publishes
through the real CLI (reloaded automatically, no extra submit prompt).
Soft-theme regression checks execute light/dark true-color rendering, ANSI-color
fallback, borderless Home/popups, stage-color identity on tinted selections,
continuous card backgrounds around nested colored labels, narrow-terminal
dimensions, and local spinner animation across pending views without extra RPCs.
Actual sample-data renders can be exported with `PIPELINES_TUI_PREVIEW_DIR=PATH
go test -run TestSoftThemeRenderPreviews ./internal/tui` for Home, New and pipeline
views in both themes. Compiled CLI/TUI PTY integration passed with the new styling.