Files
pipelines/internal/author/skill/references/harnesses.md
T
alan fae1005962 Allow launching runs with whole-run and per-stage model overrides
Add `pipelines run --model PROVIDER/MODEL` and repeatable
`--stage-model STAGE=PROVIDER/MODEL`, resolved with precedence
per-stage > whole-run > the model pinned in the pipeline version.

- A shared pure resolver (`pipeline.ParseModelOverride` /
  `pipeline.EffectiveModels`) owns the precedence and validation: every value
  must be a `provider/model` pair (`stage.ValidModel`), every stage key must
  exist in the pinned graph, an unknown key fails naming the valid keys, and a
  per-stage key that names a non-model stage is refused.
- The server validates against the pinned version before admission (no run row
  is ever inserted for an invalid request), persists `model`/`stage_models` on
  the run and `provider`/`model` on each step, applies the effective pair to a
  clone of the stage definition so the stage and its repair/formatter path share
  it, overlays the returned run graph with the effective pair (presentation
  only), and reuses the recorded overrides on retry.
- The run record echoes the overrides it was launched with and each step records
  the provider/model it actually ran with; the TUI Evidence pane renders that
  effective pair. The pinned pipeline version and `show --version N` are never
  modified.
- CLI help, `docs/cli.md` and the packaged authoring skill references document
  the flags and the precedence. A TUI override picker remains a documented
  follow-up; the CLI is the deliverable.

Tests: pure precedence/validation table tests, store round-trip and
request-key-mismatch plus migration, server admission/application/overlay/retry
with a controlled executor, CLI flag/help/malformed-value tests, and a TUI
evidence test.
2026-10-16 23:33:53 +01:00

189 lines
13 KiB
Markdown

# Configured harnesses
Current source supports OpenCode and Pi interactive authoring, debugging and
model stages. nvim remains a file editor/read-only debug packet viewer, not a
stage backend. Only configured harnesses appear in TUI pickers, default first.
CLI `new/edit/debug --harness opencode|pi|nvim` overrides the default.
`pipelines harness` launches the configured default in the invocation directory.
CLI `new/edit/debug` and TUI author/debug build one shared, Docker-aware CLI
prefix. A Docker-linked profile embeds the in-container `/usr/local/bin/pipelines`
(never the host binary: Linux can see but cannot execute a mounted macOS host
executable). Loopback is recognized only via `net.ParseIP(...).IsLoopback()` plus
an exact case-insensitive `localhost`, so a remote DNS name such as
`127.remote.example` stays remote and runs no probe. For a loopback logical
`--url`, a single bounded `docker compose port` probe parses each returned line
as a complete `host:port` binding (IPv4/IPv6/wildcard) and the prefix maps only
when a binding covers the requested loopback address and port; malformed or
non-matching bindings fail closed. A loopback `wss` endpoint is rejected as
having no supported local TLS mapping rather than downgraded to plaintext. On a
match it embeds scoped `PIPELINES_TRANSPORT_SOURCE`/`PIPELINES_TRANSPORT_TARGET`
aliases in the prefix string (not `cmd.Env`, so the pre-existing harness
server's tools inherit them). The client rewrites an endpoint only on an exact
source match; remote endpoints stay remote. Draft metadata (server, base
version, create-only guard) is preserved unchanged.
`provider-check` with a Docker-linked profile runs inside that profile's existing
Compose service with `-T`, the original cwd/model/timeout and exact stdout/exit
propagation; container profiles have no `DockerDirectory` and cannot recurse, and
explicit native profiles keep the host path. The picker uses a true `x/term` TTY
check, so `/dev/null` and pipes fail fast instead of hanging.
`pipelines harness setup opencode|pi --login --default` runs a styled onboarding
wizard (bold title, verified Docker harness versions) and selects separate
harness and new-pipeline models using type-to-filter fuzzy pickers, not numbered
prompts. Each picker itself shows its numbered step with the title and purpose
(no duplicate numbered prose is printed outside it); the interactive harness
model is `Step 1 of 2` and the new-pipeline model is `Step 2 of 2`, matching
fresh Docker setup. Typing filters, arrow keys or Tab navigate, Enter confirms,
and Esc cancels. The saved value is preselected and marked `(current)`; the window
follows the active row on narrow terminals and the heading, selected row and
no-match state stay within the real terminal height.
Automation: `--model PROVIDER/MODEL --pipeline-model PROVIDER/MODEL`.
`harness default [opencode|pi]` switches defaults; `harness list` prints settings.
Setup reads catalogs/login state without inference. Changing defaults never
rewrites published definitions or retained drafts. Canceling or failing either
picker leaves saved preferences byte-identical. Non-terminal automation with a
missing model fails clearly instead of starting a picker.
`./install --opencode` / `./install --pi` reopen login and selection. With an
existing Docker setup, they configure inside its container and link the host
launcher without restarting/deploying services. They resolve and verify the current
OpenCode V2 release from the single documented complete
`GET .../git/matching-refs/tags/v2` response (a bounded complete array with no
pagination parameters; never the moving `latest` tag or the registry's insertion
order) and the current Pi release from npm, resolve the selected tag's immutable
GHCR `sha256` digest, and require a rebuild only when the verified release image
identity no longer matches. A full install always invokes the Docker build for a
dirty source (relying on layer caching); a model/login-only reconfiguration does
not rebuild. Setup writes its generated release receipt to the machine-local root
file `.pipelines-releases.json` (with atomic `.pipelines-releases-*` temporary
files); both are git-ignored, so a fresh clean checkout stays clean and reinstall
still reuses the verified release/image identity. The actual built image's labels
stay authoritative and a dirty source still rebuilds. If discovery, digest
resolution or completeness cannot be established they stop with an error (keeping
the last verified Dockerfile fallback pins); they never guess, fall back to a host
runtime or install a stale pin.
Without a Docker installation, `./install --opencode`/`--pi` stop and require an
existing Docker setup; the native installation reuses the host OpenCode
configuration, writes **host** (non-Docker) profiles for the harnesses it detects
into XDG config home's `pipelines/harnesses.json`, and clears a stale Docker link
so a Docker-era profile is not left Docker-linked. It never installs, upgrades or
downloads a harness. Container and host credentials and accounts remain independent and
are reused on reinstall/reconfigure. `./install` installs natively by default
(`--native` is equivalent); explicit `./install --docker` selects the Docker
server + ntfy installation.
Fresh setup accepts `./install --docker --harness pi` to skip OpenCode
login. Existing setup can configure either harness independently. Pi setup does
not run a generation check; `provider-check --harness pi --model PROVIDER/MODEL`
is a separate confirmed, potentially chargeable check.
Profiles live in XDG config home's `pipelines/harnesses.json`, store preferences
only, and preserve unrelated configuration. A profile launches a native host
harness unless it records a Docker checkout directory, in which case it launches
through that installation's Compose service. `harness setup <opencode|pi> --host`
configures or converts a host profile and clears any Docker link; without
`--host`, reconfiguring a Docker-linked profile re-verifies that the recorded
directory still has `compose.yaml` and fails naming it rather than saving a link
that cannot launch. Setup's summary and `harness list` report the resulting mode
(and, for Docker, the directory). Harness CLI setup/default commands
always configure the local execution user, not a remote `--url` server.
OpenCode interactive launch uses
`mini --model`: verified V2 full CLI lacks a model flag. Pi uses its CLI with the
packaged skill explicitly loaded. Pi has no standalone `auth login` command
(verified `pi auth --help`: only credential check/print): setup prints a
title-plus-three-lines Pi sign-in handoff and opens a fresh, no-context,
extension-free, no-prompt-templates, no-tools session with Pi's documented
`--tui-mode regular` flag, so the header and handoff stay visible above Pi's
interface at 80x30 instead of the default fullscreen clear. The user types
`/login`, chooses an authentication method ("Sign in with an API key" for
provider API credentials; account and Radius remain the user's choice), finishes
provider sign-in, then exits with `ctrl+c`/`ctrl+d` to return to Pipelines model
configuration. Existing sign-ins are reused by exiting when no new sign-in is
needed; completing a sign-in can update the credentials stored in the Docker
volume. No coding/task prompt or scripted keystrokes are injected.
OpenCode V2 model stages settle long sessions by polling `GET /api/session/active`
about once per second under the stage context, instead of the `.../wait` request
whose CLI-side request timeout failed long stages after roughly six minutes
(HTTP 499, dated 2026-10-10). V1 native settlement is unchanged. Final results
and debug session evidence use the same single bounded latest-assistant readback
(`order=desc&limit=1&type=assistant`), so a small final result is visible behind
large prior tool content; retained transcripts over 32 KiB are stored as valid
bounded preview envelopes that preserve stage limits and diagnostics, and exact
outputs remain authoritative. Pi evidence is JSONL and is not parsed as one JSON
object. Short-lived context-controlled CLI calls (`--help` detection, API calls
and the native export fallback) share one finite inherited-pipe drain grace
(`os/exec.Cmd.WaitDelay` ~250ms) so a canceled context or a descendant that still
holds stdout/stderr cannot block for the child's natural lifetime; the
`context.Canceled`/`DeadlineExceeded` and `exec.ErrWaitDelay` identities are
preserved and partial stdout after the grace is never accepted as success.
Stages set `harness=pi` or `harness=opencode` in configs; omission preserves
historical OpenCode behavior. Pi must be configured on the execution server.
A run may override the pinned model at launch: `pipelines run --model
PROVIDER/MODEL` covers every model stage and repeatable `--stage-model
STAGE=PROVIDER/MODEL` overrides one stage, with precedence **per-stage >
whole-run > the model pinned in the pipeline version**. Overrides are validated
against the pinned version before the run starts (an unknown stage key fails
naming the valid keys; a non-model stage cannot be overridden), apply to that run
only, and are recorded on the run so `retry` reproduces them. The effective
provider/model is stamped on the step and exposed by the run's graph, and the
stage's repair/formatter path uses the same resolved model because both read the
stage's effective configs.
The RPC adapter starts new retained sessions per visit with explicit tools and
a Pipelines permission extension. Ambient extensions, skills, context files,
prompt templates, automatic retries/compaction are disabled. Tool gates are not
an OS sandbox: approved shell can have broad side effects. Usage accumulates
across attempts/visits; missing required accounting fails closed. Strict envelope
validation and bounded syntax-only repair never replay tools. A model-driven
repair, when needed, uses a separate deny-all session with retained ownership
and the same cumulative budget. Its output is checked locally for value changes.
Session/RPC audit lives in the execution user's cache under `pipelines/pi-sessions`.
Ownership persists before prompts. Cleanup waits for settlement; expired leases
request abort through the extension. Unconfirmed cleanup blocks continuation/retry.
2026-10-10 verification: `scripts/check-pi-rpc.py` checked installed Pi 0.87.1
readiness, metadata/config RPC and **idle** lease cleanup without inference.
Go RPC fixtures cover result/budget/missing-accounting/timeout/format-repair paths;
Node tests cover permission checks. `PIPELINES_PI_LOCAL=1 go test ./internal/stage
-run TestPiInstalledLocalIntegration` also exercised the **installed Pi** against
scripted loopback responses: actual allowed/denied writes, budget exhaustion,
and active-shell cancellation (including no delayed write after cleanup).
The installed-Pi loopback suite also covers isolated deny-all syntax repair with
retained ownership and cumulative usage. The compiled fuzzy picker passed PTY
selection/cancellation; installed Pi catalog discovery/configuration passed with
an unreachable local endpoint and no prompt.
This is integration evidence, not real-provider/model evidence. Active lease-expiry
recovery, web search delivery and fresh interactive Docker Pi setup remain
**not tested**. Pi catalogs exclude ambient provider extensions for parity with
audited stages; compatible custom endpoints can use Pi's private `models.json`.
An isolated built Docker image also passed non-root, network-disabled policy/RPC
readiness and idle lease cleanup, without credentials/mounts or inference.
Codex/Claude
adapters and AI-assisted server guidelines validation remain future work.
2026-10-11 source-level coverage (source-level regressions; the separate dated
2026-10-11 final install is live evidence): the
shared Docker-aware CLI prefix, exact-match client transport rewriting,
Docker-linked `provider-check` dispatch and the `x/term` TTY check have
source-level regression tests (`internal/harness`, `internal/client`,
`internal/cli`). The prefix tests cover the remote-DNS negative
(`127.remote.example` stays unmapped with no probe), IPv4/IPv6/`localhost`
positives, complete `host:port` binding matching (a different service on the same
port, or a family mismatch, does not match), malformed/empty output failing
clearly, multiple bindings under a single probe, and loopback `wss` rejection.
Terminal Pick regression tests use a short deadline so a TTY regression fails
fast rather than hanging. Docker dispatch tests assert `--json` is passed only in
JSON mode while plain text stays readable. Python fixtures cover arbitrary-digit
OpenCode patches (a parsed `2.0.260` is still rejected against an expected
`2.0.26` by exact equality; 260/100 are illustrative valid/future examples, not
verified releases) and reject both raw and canonical reserved homes. Real
in-container `provider-check`/ping confirmation, a compiled PTY run and a real
Docker reconfigure remain **not tested** in this change. These source-level
regressions are not a claim about the deployed image; the separate dated 2026-10-11
FINAL INSTALLED verification recorded that the runtime was actually installed and
used (OpenCode 2.0.26 / Pi 1.1.0 / Node 24.18.1, image
`sha256:1810ab50f3ea3991e7f3548c79250e65a114bb1821eac52efd872b91cf7c3136`, with
both real stages executed in run17 and passed in Root's full host race/vet runs).
Active lease-expiry recovery, web search delivery and fresh interactive Docker Pi
sign-in remain **not tested**.