8.2 KiB
Verification scope
Regression commands:
go test -race -count=1 ./...
go vet ./...
python3 -m unittest discover -s scripts -p 'test_*.py'
Optional no-inference stable OpenCode checks:
PIPELINES_TEST_OPENCODE_V1=/path/to/opencode \
go test ./internal/opencode ./internal/author -run TestStable -count=1 -v
The stable checks use isolated home/config/data/cache directories; they never copy login credentials or invoke a model. They exercise session creation, explicit model selection, stored scoped permissions, abort on the same owned service, session deletion, and discovery of the installed global authoring skill. Authoring now references that skill directly instead of preloading instructions.
Native generation/assistant completion parsing is regression-tested against a controlled HTTP server. Full live inference, permission enforcement by real model tool calls, cancellation during inference, Docker provider authentication, privileged system-wide copying, and OS service-manager registration are not tested.
The full Docker installer was exercised in an isolated Compose project with real
ntfy and Pipelines containers: image build, account/ACL/token bootstrap, Compose
health checks, authenticated notification publishing from the Pipelines container,
CLI status, reinstall preservation of .env/token, and an empty pipeline database.
Provider-account inspection/login completion was mocked in that installer test.
A separate real-container PTY invocation of opencode auth login displayed its
searchable provider choices and was canceled without credentials. Provider login,
model inference and phone delivery remain not tested.
Consequential-action regressions check run/retry/cancellation and CLI publication/
draft deletion confirmation, default-No behavior, JSON stdout cleanliness, and
--yes for explicit automation. The compiled CLI/TUI PTY check verified that q
does not exit, :quit exits, canceled retry prompts create no run, and an explicit
y starts a new retry with the original input/directory and pinned version.
Autostart regression tests exercise both service-definition formats, scoped
environment capture, escaping, private files, refusal to overwrite unmanaged
files/symlinks, preview without side effects, and lifecycle calls using a mock OS
manager. The installed macOS CLI's preview was validated with plutil -lint.
GOOS=linux GOARCH=amd64 go build ./cmd/... also passed. Actual launchd/systemd
registration, logout/login restart, and Linux lingering remain not tested.
An isolated installed-CLI check started the server in one directory and invoked
pipelines run from two other directories. With a mock OpenCode executable, both
runs completed and recorded/executed in the client invocation directory. The
autostart preview required no repository configuration and passed plutil -lint.
CLI and dashboard regression tests also check invocation-directory selection.
The installation tests were also executed as a Linux binary in Debian with
systemd installed. The native systemd-analyze verify regression passed for the
generated unit, including a directory containing spaces, quotes, and %/$.
This reproduced and fixed the invalid quoted WorkingDirectory setting. A
container syntax check does not prove live user-service startup or login restart.
Authoring regression tests exercise idempotent skill installation, preservation of user modifications, one direct OpenCode launch with a short skill/context reference, and an unnamed new draft without CLI questions. TUI integration checks browse a CLI-style retained draft, validate through the server, require explicit submission confirmation, preserve create-only guards, and reject a different server.
An isolated installed-binary PTY check also opened a CLI-created draft in the TUI, reopened the same file through a mock OpenCode executable, and explicitly submitted it while retaining the draft. Plain unnamed CLI creation invoked the mock exactly once. Real model-driven authoring/questions and native skill loading during a model turn remain not tested; global skill discovery was tested without inference.
An installed-binary integration check executed a Bash/QML fixture through a mock
OpenCode backend: it changed README/QML, created a new text file, and actually ran
bash -n install.sh. Both JSON and PTY supplied-task runs started without approval
flags, file lists, or typed confirmation, in the invocation directory rather than
the server directory. Permission regression tests cover language-neutral shell,
project file types/new files, optional narrower scopes, and outside-edit denials.
Real model/tool-permission enforcement remains not tested. Shell uses the host
user’s authority; these policies are not an OS sandbox.
A compiled-binary PTY integration check exercised the fuzzy editor picker in
standalone pipelines-tui, choosing both OpenCode and nvim with an intentionally
invalid EDITOR override. It checked confirmed TUI draft deletion, CLI deletion
without touching run history, the shared run graph, and D launching a fresh
debugging conversation containing authoritative CLI queries. Mock OpenCode/nvim
processes recorded the actual arguments and directory. Real model-driven run
investigation and live OpenCode UI attachment remain not tested.
Focused tests cover elapsed duration, live graph updates via run responses,
shared-renderer equivalence, loop visits, viewport resizing, and retry rejection
during execution cleanup even after a terminal status is persisted.
Result-recovery regression tests actually execute a mock audited agent that
edits a file once, returns malformed JSON, and receives a separate deny-all
formatting session. They check exact parser feedback, no agent/tool replay,
preserved approval/evidence, rejection of changed verdicts or omitted evidence,
and optional file/command permission rules. A native stable no-inference live
contract test accepted a json_schema message with retryCount=0 and noReply.
Real model-driven schema generation, formatter behavior and tool-permission
enforcement remain not tested.
Status/dashboard checks execute server and fake ntfy HTTP health endpoints,
CLI JSON/plain status and statistics, all-history SQL aggregation with more than
50 runs, interrupted/unused pipeline counts, zero-filled daily activity, responsive
dashboard columns, and status popup navigation from a run without losing its
selection. Compiled-binary PTY integration also exercises pipelines status,
pipelines stats, :status, dashboard activity and Home's all-runs r shortcut.
Candidate URL reachability and phone reception are not tested and are not
reported as verified by the UI.
Editor-return checks query actual published history before showing confirmation;
tests cover a submitted draft followed by a newer publication, unsent edits,
failed/stale verification, and immutable history. Compiled CLI/TUI PTY integration
uses a mock OpenCode editor that submits with the real CLI, then verifies automatic
reload to version 2 without another submission prompt. Another mock editor only
saves changes and still requires confirmation. A real local V2 no-inference
create/wait/delete contract succeeded; the reported remote planner /wait failure
could not be reproduced here. Safe CLI failure classification/exit-code reporting
was executed with a failing subprocess fixture.
New-pipeline picker checks cover both editor choices, fuzzy/Tab navigation, cancel-without-creating, retained source/server/create-only metadata and template source preservation. Compiled CLI/TUI PTY integration launches a new nvim draft (confirmation required), then a new mock OpenCode author that names and publishes through the real CLI (reloaded automatically, no extra submit prompt).
Soft-theme regression checks execute light/dark true-color rendering, ANSI-color
fallback, borderless Home/popups, stage-color identity on tinted selections,
continuous card backgrounds around nested colored labels, narrow-terminal
dimensions, and local spinner animation across pending views without extra RPCs.
Actual sample-data renders can be exported with PIPELINES_TUI_PREVIEW_DIR=PATH go test -run TestSoftThemeRenderPreviews ./internal/tui for Home, New and pipeline
views in both themes. Compiled CLI/TUI PTY integration passed with the new styling.