113 lines
6.6 KiB
Markdown
113 lines
6.6 KiB
Markdown
# Verification scope
|
||
|
||
Regression commands:
|
||
|
||
```sh
|
||
go test -race -count=1 ./...
|
||
go vet ./...
|
||
python3 -m unittest discover -s scripts -p 'test_*.py'
|
||
```
|
||
|
||
Optional no-inference stable OpenCode checks:
|
||
|
||
```sh
|
||
PIPELINES_TEST_OPENCODE_V1=/path/to/opencode \
|
||
go test ./internal/opencode ./internal/author -run TestStable -count=1 -v
|
||
```
|
||
|
||
The stable checks use isolated home/config/data/cache directories; they never
|
||
copy login credentials or invoke a model. They exercise session creation, explicit
|
||
model selection, stored scoped permissions, abort on the same owned service,
|
||
session deletion, and discovery of the installed global authoring skill. Authoring
|
||
now references that skill directly instead of preloading instructions.
|
||
|
||
Native generation/assistant completion parsing is regression-tested against a
|
||
controlled HTTP server. Full live inference, permission enforcement by real model
|
||
tool calls, cancellation during inference, optional Docker deployment, privileged
|
||
system-wide copying, and OS service-manager registration are **not tested**.
|
||
|
||
Autostart regression tests exercise both service-definition formats, scoped
|
||
environment capture, escaping, private files, refusal to overwrite unmanaged
|
||
files/symlinks, preview without side effects, and lifecycle calls using a mock OS
|
||
manager. The installed macOS CLI's preview was validated with `plutil -lint`.
|
||
`GOOS=linux GOARCH=amd64 go build ./cmd/...` also passed. Actual launchd/systemd
|
||
registration, logout/login restart, and Linux lingering remain **not tested**.
|
||
|
||
An isolated installed-CLI check started the server in one directory and invoked
|
||
`pipelines run` from two other directories. With a mock OpenCode executable, both
|
||
runs completed and recorded/executed in the client invocation directory. The
|
||
autostart preview required no repository configuration and passed `plutil -lint`.
|
||
CLI and dashboard regression tests also check invocation-directory selection.
|
||
|
||
The installation tests were also executed as a Linux binary in Debian with
|
||
systemd installed. The native `systemd-analyze verify` regression passed for the
|
||
generated unit, including a directory containing spaces, quotes, and `%`/`$`.
|
||
This reproduced and fixed the invalid quoted `WorkingDirectory` setting. A
|
||
container syntax check does not prove live user-service startup or login restart.
|
||
|
||
Authoring regression tests exercise idempotent skill installation, preservation of
|
||
user modifications, one direct OpenCode launch with a short skill/context reference,
|
||
and an unnamed new draft without CLI questions. TUI integration checks browse a
|
||
CLI-style retained draft, validate through the server, require explicit submission
|
||
confirmation, preserve create-only guards, and reject a different server.
|
||
|
||
An isolated installed-binary PTY check also opened a CLI-created draft in the TUI,
|
||
reopened the same file through a mock OpenCode executable, and explicitly submitted
|
||
it while retaining the draft. Plain unnamed CLI creation invoked the mock exactly
|
||
once. Real model-driven authoring/questions and native skill loading during a
|
||
model turn remain **not tested**; global skill discovery was tested without inference.
|
||
|
||
An installed-binary integration check executed a Bash/QML fixture through a mock
|
||
OpenCode backend: it changed README/QML, created a new text file, and actually ran
|
||
`bash -n install.sh`. Both JSON and PTY supplied-task runs started without approval
|
||
flags, file lists, or typed confirmation, in the invocation directory rather than
|
||
the server directory. Permission regression tests cover language-neutral shell,
|
||
project file types/new files, optional narrower scopes, and outside-edit denials.
|
||
Real model/tool-permission enforcement remains **not tested**. Shell uses the host
|
||
user’s authority; these policies are not an OS sandbox.
|
||
|
||
A compiled-binary PTY integration check exercised the fuzzy editor picker in
|
||
standalone `pipelines-tui`, choosing both OpenCode and nvim with an intentionally
|
||
invalid EDITOR override. It checked confirmed TUI draft deletion, CLI deletion
|
||
without touching run history, the shared run graph, and `D` launching a fresh
|
||
debugging conversation containing authoritative CLI queries. Mock OpenCode/nvim
|
||
processes recorded the actual arguments and directory. Real model-driven run
|
||
investigation and live OpenCode UI attachment remain **not tested**.
|
||
Focused tests cover elapsed duration, live graph updates via run responses,
|
||
shared-renderer equivalence, loop visits, viewport resizing, and retry rejection
|
||
during execution cleanup even after a terminal status is persisted.
|
||
|
||
Result-recovery regression tests actually execute a mock audited agent that
|
||
edits a file once, returns malformed JSON, and receives a separate deny-all
|
||
formatting session. They check exact parser feedback, no agent/tool replay,
|
||
preserved approval/evidence, rejection of changed verdicts or omitted evidence,
|
||
and optional file/command permission rules. A native stable no-inference live
|
||
contract test accepted a `json_schema` message with `retryCount=0` and `noReply`.
|
||
Real model-driven schema generation, formatter behavior and tool-permission
|
||
enforcement remain **not tested**.
|
||
|
||
Status/dashboard checks execute server and fake ntfy HTTP health endpoints,
|
||
CLI JSON/plain status and statistics, all-history SQL aggregation with more than
|
||
50 runs, interrupted/unused pipeline counts, zero-filled daily activity, responsive
|
||
dashboard columns, and status popup navigation from a run without losing its
|
||
selection. Compiled-binary PTY integration also exercises `pipelines status`,
|
||
`pipelines stats`, `:status`, dashboard activity and Home's all-runs `r` shortcut.
|
||
Candidate URL reachability and phone reception are **not tested** and are not
|
||
reported as verified by the UI.
|
||
|
||
Editor-return checks query actual published history before showing confirmation;
|
||
tests cover a submitted draft followed by a newer publication, unsent edits,
|
||
failed/stale verification, and immutable history. Compiled CLI/TUI PTY integration
|
||
uses a mock OpenCode editor that submits with the real CLI, then verifies automatic
|
||
reload to version 2 without another submission prompt. Another mock editor only
|
||
saves changes and still requires confirmation. A real local V2 no-inference
|
||
create/wait/delete contract succeeded; the reported remote planner `/wait` failure
|
||
could not be reproduced here. Safe CLI failure classification/exit-code reporting
|
||
was executed with a failing subprocess fixture.
|
||
|
||
New-pipeline picker checks cover both editor choices, fuzzy/Tab navigation,
|
||
cancel-without-creating, retained source/server/create-only metadata and template
|
||
source preservation. Compiled CLI/TUI PTY integration launches a new nvim draft
|
||
(confirmation required), then a new mock OpenCode author that names and publishes
|
||
through the real CLI (reloaded automatically, no extra submit prompt).
|