Allow launching runs with whole-run and per-stage model overrides

Add `pipelines run --model PROVIDER/MODEL` and repeatable
`--stage-model STAGE=PROVIDER/MODEL`, resolved with precedence
per-stage > whole-run > the model pinned in the pipeline version.

- A shared pure resolver (`pipeline.ParseModelOverride` /
  `pipeline.EffectiveModels`) owns the precedence and validation: every value
  must be a `provider/model` pair (`stage.ValidModel`), every stage key must
  exist in the pinned graph, an unknown key fails naming the valid keys, and a
  per-stage key that names a non-model stage is refused.
- The server validates against the pinned version before admission (no run row
  is ever inserted for an invalid request), persists `model`/`stage_models` on
  the run and `provider`/`model` on each step, applies the effective pair to a
  clone of the stage definition so the stage and its repair/formatter path share
  it, overlays the returned run graph with the effective pair (presentation
  only), and reuses the recorded overrides on retry.
- The run record echoes the overrides it was launched with and each step records
  the provider/model it actually ran with; the TUI Evidence pane renders that
  effective pair. The pinned pipeline version and `show --version N` are never
  modified.
- CLI help, `docs/cli.md` and the packaged authoring skill references document
  the flags and the precedence. A TUI override picker remains a documented
  follow-up; the CLI is the deliverable.

Tests: pure precedence/validation table tests, store round-trip and
request-key-mismatch plus migration, server admission/application/overlay/retry
with a controlled executor, CLI flag/help/malformed-value tests, and a TUI
evidence test.
This commit is contained in:
2026-10-16 23:33:53 +01:00
parent 80232a4ffe
commit fae1005962
19 changed files with 912 additions and 31 deletions
+36 -1
View File
@@ -219,7 +219,7 @@ Submitting activates included schedules, so review them before publishing.
| Browse templates/source | `templates --json`, `template KEY --source` |
| Create or edit a definition | `new`, `edit`, `drafts` |
| Validate and publish | `validate FILE`, `submit FILE` |
| Start a run | `run NAME PROMPT [--title TITLE] --yes --json` |
| Start a run | `run NAME PROMPT [--title TITLE] [--model P/M] [--stage-model S=P/M] --yes --json` |
| Run history/inspection | `runs --json`, `inspect ID --json` |
| Rename a run | `title ID "TITLE" --json` |
| Decide a waiting human stage | `decide ID [STAGE] --option KEY [--feedback TEXT] --json` |
@@ -262,6 +262,41 @@ pipeline are distinguishable. Rename after the fact with `pipelines title
<run-id> "<title>"`; the run's input, steps and execution are unchanged. The
stored prompt is unchanged, and listing/renaming never calls a model.
### Model overrides at launch
A pipeline version pins one model per stage. Without republishing, a run can
choose different models at launch:
```sh
pipelines run "bug fix" "Fix the layout" --json --yes \
--model openai/gpt-5 \
--stage-model review=openai/o4-mini
```
- `--model PROVIDER/MODEL` overrides the model for **every** model stage of that
run.
- `--stage-model STAGE=PROVIDER/MODEL` overrides **one** stage and is repeatable,
so any number of stages can be overridden. A repeated key for the same stage is
last-wins.
- Precedence: a per-stage override beats the whole-run override, which beats the
model pinned in the pipeline version.
Overrides apply to that run only. The pipeline definition and its immutable
version are never modified, `show <name> --version N` still returns the pinned
source and graph, and the run records the `model` and `stage_models` it was
launched with so it stays reproducible; `retry` reuses them. At launch every
value must be a `provider/model` pair of the configured form and every stage key
must exist in the pinned version — an unknown key fails naming the valid keys and
no run is created. A per-stage key that names a non-model stage is refused
because that stage never runs a model; `--model` on a pipeline with no model stage
is accepted and inert. The effective model reaches the stage and its
repair/formatter path from the same resolved definition, and each step's evidence
records the provider/model it actually ran with (`inspect <id> --json` exposes
per-step `provider`/`model`, and the run's graph shows the effective pair). The
CLI is the deliverable; a TUI picker for choosing overrides is a documented
follow-up, not half-built.
On either the runs list or run inspector, press `P` (or use `:pipeline`) to open
the selected run's exact pipeline version. Navigation does not cancel the run.
@@ -118,6 +118,16 @@ preserved and partial stdout after the grace is never accepted as success.
Stages set `harness=pi` or `harness=opencode` in configs; omission preserves
historical OpenCode behavior. Pi must be configured on the execution server.
A run may override the pinned model at launch: `pipelines run --model
PROVIDER/MODEL` covers every model stage and repeatable `--stage-model
STAGE=PROVIDER/MODEL` overrides one stage, with precedence **per-stage >
whole-run > the model pinned in the pipeline version**. Overrides are validated
against the pinned version before the run starts (an unknown stage key fails
naming the valid keys; a non-model stage cannot be overridden), apply to that run
only, and are recorded on the run so `retry` reproduces them. The effective
provider/model is stamped on the step and exposed by the run's graph, and the
stage's repair/formatter path uses the same resolved model because both read the
stage's effective configs.
The RPC adapter starts new retained sessions per visit with explicit tools and
a Pipelines permission extension. Ambient extensions, skills, context files,
prompt templates, automatic retries/compaction are disabled. Tool gates are not
@@ -126,6 +126,13 @@ Adapt the scope and publishing lines per item; keep the rest.
- Launch with the authoritative `CLI_PREFIX` (`pipelines run`). The brief text is the
run's task; pass it whole. Read `references/operations.md` for the exact flags —
`--json`, confirmation (`--yes`) and `--wait`/`--wait-timeout`.
- **Model overrides at launch.** Use `--model PROVIDER/MODEL` to override the model
for every model stage of that run, and repeatable `--stage-model
STAGE=PROVIDER/MODEL` to override individual stages. Precedence is per-stage >
whole-run > the model pinned in the pipeline version. Overrides apply to that run
only (the pinned pipeline is never edited), are validated before the run starts,
and are recorded on the run; `retry` reuses them. See `references/harnesses.md`
for how a stage's model and its repair path both use the effective model.
- Watch to a **terminal state** (`succeeded`, `failed`, `canceled`, `interrupted`).
Poll `CLI_PREFIX inspect RUN_ID --json` (or use `--wait`) until terminal. A reached
client wait limit is not a run failure: the run stays active; keep waiting or
+76 -5
View File
@@ -10,6 +10,7 @@ import (
"os"
"os/exec"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
@@ -43,6 +44,41 @@ type options struct {
json, source, noOpen, yes, allow, createOnly, force, wait bool
version, expected, limit int
timeout, waitTimeout time.Duration
stageModels stageModelFlags
}
// stageModelFlags collects the repeatable --stage-model STAGE=PROVIDER/MODEL
// values. A repeated key is last-wins. It is a flag.Value so the repeatable flag
// needs no dependency (Go stdlib has no slice flag).
type stageModelFlags map[string]string
// String renders the collected overrides for flag's defaults display.
func (s stageModelFlags) String() string {
keys := make([]string, 0, len(s))
for key := range s {
keys = append(keys, key)
}
sort.Strings(keys)
parts := make([]string, 0, len(keys))
for _, key := range keys {
parts = append(parts, key+"="+s[key])
}
return strings.Join(parts, ",")
}
// Set parses one STAGE=PROVIDER/MODEL value and validates the provider/model with
// the shared parser, so a malformed value fails before any run is created.
func (s stageModelFlags) Set(value string) error {
key, model, ok := strings.Cut(value, "=")
key, model = strings.TrimSpace(key), strings.TrimSpace(model)
if !ok || key == "" || model == "" {
return fmt.Errorf("--stage-model requires STAGE=PROVIDER/MODEL")
}
if _, err := pipeline.ParseModelOverride(model); err != nil {
return fmt.Errorf("--stage-model %s: %w", key, err)
}
s[key] = model
return nil
}
func (a App) flags(command string) (*flag.FlagSet, *options) {
@@ -98,6 +134,9 @@ func (a App) flags(command string) (*flag.FlagSet, *options) {
f.StringVar(&o.paths, "write-paths", "", "Optional narrower edit scope; default is all files in the run directory")
f.StringVar(&o.key, "request-key", "", "Idempotency key (default: generated unique key)")
f.IntVar(&o.version, "version", 0, "Version to run (default: latest)")
f.StringVar(&o.model, "model", "", "Override the model for every stage of this run (PROVIDER/MODEL)")
o.stageModels = stageModelFlags{}
f.Var(o.stageModels, "stage-model", "Override one stage's model; repeatable (STAGE=PROVIDER/MODEL); beats --model")
f.BoolVar(&o.wait, "wait", false, "In JSON mode, wait for a terminal run state")
f.DurationVar(&o.waitTimeout, "wait-timeout", 0, "Client limit for --wait after the run is admitted (0 waits until terminal); separate from per-request --timeout")
case "runs":
@@ -278,10 +317,11 @@ func (a App) Run(ctx context.Context, args []string) error {
if len(pos) != 0 {
return fmt.Errorf("provider-check takes no positional arguments")
}
provider, model, ok := strings.Cut(o.model, "/")
if !ok || !stage.ValidModel(provider, model) {
override, err := pipeline.ParseModelOverride(o.model)
if err != nil {
return fmt.Errorf("--model requires provider/model")
}
provider, model := override.Provider, override.Model
if err := a.confirmAction("Validate "+o.model+" with one small tool-free generation (provider charges may apply)?", o.yes); err != nil {
return err
}
@@ -1117,6 +1157,14 @@ func (a App) run(ctx context.Context, pos []string, o *options, call callFunc, r
paths = append(paths, p)
}
}
// Pre-flight form validation of the whole-run override with the same parser
// the server uses; stage-key existence stays server-authoritative. A malformed
// value fails here, before any confirmation or run request.
if o.model != "" {
if _, err := pipeline.ParseModelOverride(o.model); err != nil {
return err
}
}
if o.yes && (o.name == "" || strings.TrimSpace(o.prompt) == "") {
return fmt.Errorf("--yes requires a pipeline and nonempty prompt")
}
@@ -1131,7 +1179,7 @@ func (a App) run(ctx context.Context, pos []string, o *options, call callFunc, r
if o.wait || o.key != "" || o.version > 0 {
return fmt.Errorf("--wait, --request-key and --version require --json")
}
_, err := tea.NewProgram(tui.NewRun(ctx, o.url, tui.RunOptions{Pipeline: o.name, Prompt: o.prompt, Title: o.title, Workspace: workspace, Yes: o.yes, AllowTools: o.allow, WritePaths: paths}), tea.WithContext(ctx), tea.WithAltScreen()).Run()
_, err := tea.NewProgram(tui.NewRun(ctx, o.url, tui.RunOptions{Pipeline: o.name, Prompt: o.prompt, Title: o.title, Workspace: workspace, Yes: o.yes, AllowTools: o.allow, WritePaths: paths, Model: o.model, StageModels: o.stageModels}), tea.WithContext(ctx), tea.WithAltScreen()).Run()
return err
}
if o.name == "" || strings.TrimSpace(o.prompt) == "" {
@@ -1145,20 +1193,43 @@ func (a App) run(ctx context.Context, pos []string, o *options, call callFunc, r
if version == 0 {
version = p.Version
}
if err := a.confirmAction(fmt.Sprintf("Run %s v%d in %s? This may edit files, execute commands and publish notifications.", p.Name, version, workspace), o.yes); err != nil {
confirmation := fmt.Sprintf("Run %s v%d in %s? This may edit files, execute commands and publish notifications.", p.Name, version, workspace) + o.modelSummary()
if err := a.confirmAction(confirmation, o.yes); err != nil {
return err
}
key := o.key
if key == "" {
key = fmt.Sprintf("cli-%d-%d", os.Getpid(), time.Now().UnixNano())
}
r, err := call(protocol.Message{Type: protocol.RunPipeline, PipelineID: p.ID, Version: version, RequestKey: key, Workspace: workspace, Title: o.title, Input: map[string]any{"prompt": o.prompt}, AllowTools: o.allow, WritePaths: paths})
r, err := call(protocol.Message{Type: protocol.RunPipeline, PipelineID: p.ID, Version: version, RequestKey: key, Workspace: workspace, Title: o.title, Input: map[string]any{"prompt": o.prompt}, AllowTools: o.allow, WritePaths: paths, Model: o.model, StageModels: o.stageModels})
if err != nil {
return err
}
return a.finishRun(ctx, r, o, call)
}
// modelSummary describes the launch-time model overrides for the action
// confirmation, so a model mutation is visible before approval. It is empty when
// no override was supplied.
func (o *options) modelSummary() string {
if o.model == "" && len(o.stageModels) == 0 {
return ""
}
parts := []string{}
if o.model != "" {
parts = append(parts, "all stages: "+o.model)
}
keys := make([]string, 0, len(o.stageModels))
for key := range o.stageModels {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
parts = append(parts, key+": "+o.stageModels[key])
}
return " Model overrides — " + strings.Join(parts, "; ") + "."
}
func (a App) finishRun(ctx context.Context, r protocol.Message, o *options, call callFunc) error {
var err error
if o.wait {
+2 -2
View File
@@ -16,7 +16,7 @@ var usage = map[string]string{
"import": "import [FILE.zip] [--all] [--on-conflict ask|keep|both|import] [--yes] [--json]",
"templates": "templates [--json]",
"template": "template <key> [--source | --json]",
"run": "run [name|id] [prompt] [--title TITLE] [--json [--wait [--wait-timeout DURATION]]]",
"run": "run [name|id] [prompt] [--title TITLE] [--model PROVIDER/MODEL] [--stage-model STAGE=PROVIDER/MODEL ...] [--json [--wait [--wait-timeout DURATION]]]",
"runs": "runs [--pipeline NAME] [--json]",
"inspect": "inspect <run-id> [--json]",
"cancel": "cancel <run-id> [--json]",
@@ -44,7 +44,7 @@ var notes = map[string]string{
"edit": "Omit the name to open fuzzy selection. Original immutable source is copied to a private draft. The configured harness edits that draft, never the live definition.",
"validate": "Uses the same server parser as submission. Does not persist or activate schedules. Structural validity does not prove runtime/tool availability.",
"submit": "Confirm publication at the prompt, or use --yes for explicit automation; schedules activate on acceptance. Managed drafts provide the name and base-version guard. --force never bypasses validation.",
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. --model PROVIDER/MODEL overrides the model for every stage of that run; --stage-model STAGE=PROVIDER/MODEL overrides one stage and is repeatable, with a repeated key last-wins. Precedence is per-stage > whole-run > the model pinned in the pipeline version; overrides apply to that run only and never change the pinned definition. An unknown stage key or a malformed provider/model is refused before the run starts, naming the valid stage keys. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
"retry": "Start a new attempt of a failed run with the same initial input, directory, pipeline version, and edit scope. Original history is retained. Already-completed side effects are not undone. --wait/--wait-timeout behave as for run: a reached wait limit leaves the new run active; inspect read-only with `inspect <run-id> --json`, or reattach with --wait and the original --request-key instead of starting a duplicate retry.",
"title": "Rename a run's operator-facing title after the fact without touching its input, steps or execution. The title is collapsed to a single bounded line and recorded as explicitly set. `pipelines runs` and `pipelines inspect` then show it.",
"decide": "Choose an option for a run waiting on a human stage. The run must be `waiting`; the option must be one of that stage's non-error edges. The stage is inferred from the run's current stage unless named. The choice and optional --feedback are recorded as the step's evidence (visible in inspect and the TUI) and the run resumes along the chosen edge. An unknown option is refused with the valid list; a repeated decision for an already-decided visit does not double-apply. Works across a server restart because the wait is persisted.",
+7
View File
@@ -61,6 +61,13 @@ Examples:
pipelines validate /path/to/draft.md --json
pipelines submit /path/to/draft.md
pipelines run "bug fix" "Fix the layout" --json
pipelines run "bug fix" "Fix the layout" --model openai/gpt-5 --stage-model review=openai/o4 --json
Launch-time model overrides: 'pipelines run --model P/M' covers every model stage;
'--stage-model STAGE=P/M' (repeatable; a repeated key is last-wins) overrides one
stage and wins over --model, which wins over the model pinned in the pipeline
version. Overrides apply to that run only, are recorded on the run, and are
validated before the run starts (an unknown stage key names the valid keys).
Draft-first: the configured harness (Pi/OpenCode; host by default, or Docker-linked
when a checkout is linked) edits a private draft; nvim remains a client editor.
+79
View File
@@ -0,0 +1,79 @@
package cli
import (
"bytes"
"context"
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// TestRunModelOverrideFlagsParse proves the repeatable --stage-model flag and the
// whole-run --model flag parse onto options, with a repeated key last-wins.
func TestRunModelOverrideFlagsParse(t *testing.T) {
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
f, o := app.flags("run")
if err := parse(f, []string{"--pipeline", "p", "--prompt", "task", "--model", "acme/whole", "--stage-model", "one=acme/first", "--stage-model", "two=acme/two", "--stage-model", "one=acme/last"}); err != nil {
t.Fatal(err)
}
if o.model != "acme/whole" {
t.Fatalf("--model = %q", o.model)
}
if o.stageModels["one"] != "acme/last" {
t.Fatalf("repeated --stage-model key not last-wins: %+v", o.stageModels)
}
if o.stageModels["two"] != "acme/two" {
t.Fatalf("second --stage-model missing: %+v", o.stageModels)
}
}
// TestStageModelFlagRejectsMalformedValues proves a value that is not
// STAGE=PROVIDER/MODEL fails at flag parse, before any run is created.
func TestStageModelFlagRejectsMalformedValues(t *testing.T) {
for _, value := range []string{"nope", "one", "=acme/x", "one=", "one=not-a-pair", "one=acme/"} {
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
f, _ := app.flags("run")
if err := parse(f, []string{"--stage-model", value}); err == nil {
t.Fatalf("malformed --stage-model %q was accepted", value)
}
}
}
// TestRunRejectsInvalidWholeRunModel proves the whole-run --model is form-checked
// with the shared parser before any server request.
func TestRunRejectsInvalidWholeRunModel(t *testing.T) {
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
called := false
call := func(protocol.Message) (protocol.Message, error) { called = true; return protocol.Message{}, nil }
resolve := func(string) (protocol.PipelineSummary, error) { return protocol.PipelineSummary{}, nil }
o := &options{json: true, yes: true, model: "not-a-pair", prompt: "task"}
err := app.run(context.Background(), []string{"pipe"}, o, call, resolve)
if err == nil || !strings.Contains(err.Error(), "provider/model") {
t.Fatalf("invalid --model error = %v", err)
}
if called {
t.Fatal("an invalid --model still sent a run request")
}
}
// TestHelpDocumentsModelOverrides proves the usage map, notes and flag help
// advertise --model and repeatable --stage-model and state the precedence.
func TestHelpDocumentsModelOverrides(t *testing.T) {
if !strings.Contains(usage["run"], "--model") || !strings.Contains(usage["run"], "--stage-model") {
t.Fatalf("run usage does not document the model overrides: %q", usage["run"])
}
note := notes["run"]
for _, want := range []string{"--model", "--stage-model", "per-stage", "pinned"} {
if !strings.Contains(note, want) {
t.Fatalf("run note does not document %q: %q", want, note)
}
}
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
f, _ := app.flags("run")
f.Usage()
help := app.Out.(*bytes.Buffer).String()
if !strings.Contains(help, "--model") || !strings.Contains(help, "--stage-model") {
t.Fatalf("run --help omits the overrides:\n%s", help)
}
}
+97
View File
@@ -0,0 +1,97 @@
package pipeline
import (
"fmt"
"sort"
"strings"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
)
// ModelOverride is a provider/model pair chosen for a run.
type ModelOverride struct {
Provider string
Model string
}
// String renders the pair as provider/model.
func (m ModelOverride) String() string { return m.Provider + "/" + m.Model }
// ParseModelOverride parses a provider/model value and validates both sides with
// stage.ValidModel. It is the one parser shared by launch-time resolution and the
// CLI's pre-flight form check, so the two cannot drift.
func ParseModelOverride(value string) (ModelOverride, error) {
provider, model, ok := strings.Cut(strings.TrimSpace(value), "/")
provider, model = strings.TrimSpace(provider), strings.TrimSpace(model)
if !ok || !stage.ValidModel(provider, model) {
return ModelOverride{}, fmt.Errorf("model override %q must be provider/model", value)
}
return ModelOverride{Provider: provider, Model: model}, nil
}
// EffectiveModels resolves the provider/model each model stage will run with for
// one run. Precedence: a per-stage override beats the whole-run override, which
// beats the model pinned in the pipeline version. It is pure and validates the
// overrides against the pinned graph: an unknown stage key fails naming the valid
// keys, a value that is not a provider/model fails, and a per-stage key for a
// non-model stage fails because that stage could never honour it. A whole-run
// override on a graph with no model stage is accepted and inert. The helper is
// shared by the server (authoritative, against the pinned version) and the CLI
// (pre-flight form check), so the precedence rule lives once.
func EffectiveModels(graph protocol.Graph, model string, stageModels map[string]string) (map[string]ModelOverride, error) {
keys := make([]string, 0, len(graph.Stages))
byKey := make(map[string]protocol.Stage, len(graph.Stages))
for _, s := range graph.Stages {
keys = append(keys, s.Key)
byKey[s.Key] = s
}
sort.Strings(keys)
var whole *ModelOverride
if strings.TrimSpace(model) != "" {
parsed, err := ParseModelOverride(model)
if err != nil {
return nil, err
}
whole = &parsed
}
perStage := make(map[string]ModelOverride, len(stageModels))
overrideKeys := make([]string, 0, len(stageModels))
for key := range stageModels {
overrideKeys = append(overrideKeys, key)
}
sort.Strings(overrideKeys)
for _, key := range overrideKeys {
declared, exists := byKey[key]
if !exists {
return nil, fmt.Errorf("unknown stage key %q; valid keys: %s", key, strings.Join(keys, ", "))
}
if stage.EffectiveType(declared.Type) != stage.ModelType {
return nil, fmt.Errorf("stage %q cannot honour a model override; it is not a model stage", key)
}
parsed, err := ParseModelOverride(stageModels[key])
if err != nil {
return nil, fmt.Errorf("stage %q: %w", key, err)
}
perStage[key] = parsed
}
effective := make(map[string]ModelOverride, len(graph.Stages))
for _, s := range graph.Stages {
if stage.EffectiveType(s.Type) != stage.ModelType {
continue
}
if override, ok := perStage[s.Key]; ok {
effective[s.Key] = override
continue
}
if whole != nil {
effective[s.Key] = *whole
continue
}
effective[s.Key] = ModelOverride{Provider: s.Configs["provider"], Model: s.Configs["model"]}
}
return effective, nil
}
+105
View File
@@ -0,0 +1,105 @@
package pipeline
import (
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
func overrideGraph() protocol.Graph {
return protocol.Graph{Stages: []protocol.Stage{
{Key: "plan", Type: "model", Configs: map[string]string{"provider": "pinned", "model": "pinned-model"}},
{Key: "note", Type: "ntfy", Configs: map[string]string{}},
{Key: "review", Type: "model", Configs: map[string]string{"provider": "pinned2", "model": "pinned2-model"}},
}}
}
// TestEffectiveModelsPrecedence proves the one precedence rule: a per-stage
// override beats the whole-run override, which beats the model pinned in the
// pipeline version, and with no override the pinned pair is returned unchanged.
func TestEffectiveModelsPrecedence(t *testing.T) {
graph := overrideGraph()
cases := []struct {
name string
model string
stageModels map[string]string
want map[string]ModelOverride
}{
{
name: "no override keeps the pinned pair",
want: map[string]ModelOverride{
"plan": {Provider: "pinned", Model: "pinned-model"},
"review": {Provider: "pinned2", Model: "pinned2-model"},
},
},
{
name: "whole-run beats the pinned pair",
model: "acme/whole",
want: map[string]ModelOverride{
"plan": {Provider: "acme", Model: "whole"},
"review": {Provider: "acme", Model: "whole"},
},
},
{
name: "per-stage beats whole-run and leaves the other pinned",
model: "acme/whole",
stageModels: map[string]string{"plan": "acme/stage"},
want: map[string]ModelOverride{
"plan": {Provider: "acme", Model: "stage"},
"review": {Provider: "acme", Model: "whole"},
},
},
{
name: "per-stage alone leaves the other pinned",
stageModels: map[string]string{"plan": "acme/stage"},
want: map[string]ModelOverride{
"plan": {Provider: "acme", Model: "stage"},
"review": {Provider: "pinned2", Model: "pinned2-model"},
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, err := EffectiveModels(graph, tc.model, tc.stageModels)
if err != nil {
t.Fatal(err)
}
if len(got) != len(tc.want) {
t.Fatalf("effective = %+v, want %+v", got, tc.want)
}
for key, want := range tc.want {
if got[key] != want {
t.Fatalf("stage %s = %+v, want %+v", key, got[key], want)
}
}
})
}
}
// TestEffectiveModelsRejectsBadInput proves an unknown stage key fails naming the
// valid keys, a malformed provider/model fails, and a per-stage key for a
// non-model stage is refused as one that can never honour it.
func TestEffectiveModelsRejectsBadInput(t *testing.T) {
graph := overrideGraph()
if _, err := EffectiveModels(graph, "", map[string]string{"nope": "acme/x"}); err == nil || !strings.Contains(err.Error(), "unknown stage key") || !strings.Contains(err.Error(), "note, plan, review") {
t.Fatalf("unknown key error = %v, want the valid keys named", err)
}
for _, value := range []string{"not-a-pair", "/model", "provider/", "bad provider/model", "provider/bad model"} {
if _, err := EffectiveModels(graph, value, nil); err == nil {
t.Fatalf("whole-run value %q was accepted", value)
}
if _, err := EffectiveModels(graph, "", map[string]string{"plan": value}); err == nil {
t.Fatalf("per-stage value %q was accepted", value)
}
}
if _, err := EffectiveModels(graph, "", map[string]string{"note": "acme/x"}); err == nil || !strings.Contains(err.Error(), "cannot honour") {
t.Fatalf("non-model stage override error = %v, want a cannot-honour refusal", err)
}
// A whole-run override on a graph with no model stage is accepted and inert.
inert := protocol.Graph{Stages: []protocol.Stage{{Key: "note", Type: "ntfy", Configs: map[string]string{}}}}
got, err := EffectiveModels(inert, "acme/whole", nil)
if err != nil || len(got) != 0 {
t.Fatalf("whole-run override on a no-model graph = %+v, %v; want inert", got, err)
}
}
+64 -17
View File
@@ -18,6 +18,7 @@ func (s *Store) initRuns() error {
id INTEGER PRIMARY KEY, pipeline_id INTEGER NOT NULL, version INTEGER NOT NULL,
request_key TEXT NOT NULL UNIQUE, workspace TEXT NOT NULL, allow_tools INTEGER NOT NULL,
title TEXT NOT NULL DEFAULT '', title_source TEXT NOT NULL DEFAULT '', title_model TEXT NOT NULL DEFAULT '',
model TEXT NOT NULL DEFAULT '', stage_models TEXT NOT NULL DEFAULT '{}',
write_paths TEXT NOT NULL, input_json TEXT NOT NULL, output_json TEXT NOT NULL DEFAULT '{}',
status TEXT NOT NULL DEFAULT 'queued', outcome TEXT NOT NULL DEFAULT '', current_stage TEXT NOT NULL DEFAULT '',
diagnostic TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, finished_at TEXT,
@@ -26,6 +27,7 @@ CREATE TABLE IF NOT EXISTS run_steps (
run_id INTEGER NOT NULL REFERENCES pipeline_runs(id), sequence INTEGER NOT NULL, stage_key TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'running', outcome TEXT NOT NULL DEFAULT '', target TEXT NOT NULL DEFAULT '',
input_json TEXT NOT NULL, output_json TEXT NOT NULL DEFAULT '{}', diagnostic TEXT NOT NULL DEFAULT '',
provider TEXT NOT NULL DEFAULT '', model TEXT NOT NULL DEFAULT '',
session_id TEXT NOT NULL DEFAULT '', agent_active INTEGER NOT NULL DEFAULT 0, transcript TEXT NOT NULL DEFAULT '',
started_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, finished_at TEXT, PRIMARY KEY(run_id,sequence));
CREATE INDEX IF NOT EXISTS run_latest ON pipeline_runs(id DESC);
@@ -53,7 +55,28 @@ run_id INTEGER PRIMARY KEY REFERENCES pipeline_runs(id), original_run_id INTEGER
if err != nil {
return err
}
return s.migrateRunTitles()
if err := s.migrateRunTitles(); err != nil {
return err
}
return s.migrateRunModels()
}
// migrateRunModels adds the launch-time model-override columns to a database
// created before per-run overrides existed. Each column is added only when
// missing, so an upgrade is idempotent across repeated Open calls and existing
// rows keep their defaults (no model override, no recorded step model).
func (s *Store) migrateRunModels() error {
for _, column := range [][3]string{
{"pipeline_runs", "model", "TEXT NOT NULL DEFAULT ''"},
{"pipeline_runs", "stage_models", "TEXT NOT NULL DEFAULT '{}'"},
{"run_steps", "provider", "TEXT NOT NULL DEFAULT ''"},
{"run_steps", "model", "TEXT NOT NULL DEFAULT ''"},
} {
if err := s.addColumnIfMissing(column[0], column[1], column[2]); err != nil {
return err
}
}
return nil
}
// migrateRunTitles adds the title columns to a database created before run
@@ -98,17 +121,28 @@ func (s *Store) addColumnIfMissing(table, column, definition string) error {
return err
}
// RunOverrides are the launch-time model overrides recorded for a run:
// Model applies to every model stage and StageModels overrides individual model
// stages by key (per-stage wins over whole-run). They are recorded so the run
// stays reproducible without touching the pinned pipeline version.
type RunOverrides struct {
Model string
StageModels map[string]string
}
// CreateRun inserts a run and derives its title from the prompt. It is the entry
// point for callers with no generated title; the derived fallback is always
// non-empty and never calls a model.
func (s *Store) CreateRun(ctx context.Context, d *protocol.PipelineDetail, key, workspace string, tools bool, paths []string, input map[string]any, retryOf ...int64) (int64, bool, error) {
return s.CreateRunTitled(ctx, d, key, workspace, tools, paths, input, summarise.Title{}, retryOf...)
return s.CreateRunTitled(ctx, d, key, workspace, tools, paths, input, RunOverrides{}, summarise.Title{}, retryOf...)
}
// CreateRunTitled inserts a run with the caller's title. A non-empty title (a
// model generation) is stored verbatim; an empty one falls back to the
// prompt-derived title, so a run never stores an empty title.
func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail, key, workspace string, tools bool, paths []string, input map[string]any, title summarise.Title, retryOf ...int64) (int64, bool, error) {
// CreateRunTitled inserts a run with the caller's title and launch-time model
// overrides. A non-empty title (a model generation) is stored verbatim; an empty
// one falls back to the prompt-derived title, so a run never stores an empty
// title. The overrides are compared in the request-key idempotency check, so a
// replay with different overrides is refused as a different run.
func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail, key, workspace string, tools bool, paths []string, input map[string]any, overrides RunOverrides, title summarise.Title, retryOf ...int64) (int64, bool, error) {
var parent int64
if len(retryOf) > 0 {
parent = retryOf[0]
@@ -125,19 +159,27 @@ func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail,
if err != nil {
return 0, false, err
}
stageModels := overrides.StageModels
if stageModels == nil {
stageModels = map[string]string{}
}
encodedModels, err := json.Marshal(stageModels)
if err != nil {
return 0, false, err
}
s.mu.Lock()
defer s.mu.Unlock()
var id, pid int64
var version int
var oldInput, oldWorkspace, oldPaths string
var oldInput, oldWorkspace, oldPaths, oldModel, oldStageModels string
var oldTools bool
err = s.db.QueryRowContext(ctx, "SELECT id,pipeline_id,version,input_json,workspace,write_paths,allow_tools FROM pipeline_runs WHERE request_key=?", key).Scan(&id, &pid, &version, &oldInput, &oldWorkspace, &oldPaths, &oldTools)
err = s.db.QueryRowContext(ctx, "SELECT id,pipeline_id,version,input_json,workspace,write_paths,allow_tools,model,stage_models FROM pipeline_runs WHERE request_key=?", key).Scan(&id, &pid, &version, &oldInput, &oldWorkspace, &oldPaths, &oldTools, &oldModel, &oldStageModels)
if err == nil {
var oldParent int64
if err := s.db.QueryRowContext(ctx, "SELECT COALESCE((SELECT original_run_id FROM run_retries WHERE run_id=?),0)", id).Scan(&oldParent); err != nil {
return 0, false, err
}
if pid != d.ID || version != d.Version || oldInput != string(encoded) || oldWorkspace != workspace || oldPaths != string(writes) || oldTools != tools || oldParent != parent {
if pid != d.ID || version != d.Version || oldInput != string(encoded) || oldWorkspace != workspace || oldPaths != string(writes) || oldTools != tools || oldParent != parent || oldModel != overrides.Model || oldStageModels != string(encodedModels) {
return 0, false, fmt.Errorf("request key already used for a different run")
}
return id, false, nil
@@ -150,7 +192,7 @@ func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail,
return 0, false, err
}
defer tx.Rollback()
result, err := tx.ExecContext(ctx, "INSERT INTO pipeline_runs(pipeline_id,version,request_key,workspace,allow_tools,write_paths,input_json,title,title_source,title_model) VALUES (?,?,?,?,?,?,?,?,?,?)", d.ID, d.Version, key, workspace, tools, string(writes), string(encoded), title.Text, title.Source, title.Model)
result, err := tx.ExecContext(ctx, "INSERT INTO pipeline_runs(pipeline_id,version,request_key,workspace,allow_tools,write_paths,input_json,title,title_source,title_model,model,stage_models) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)", d.ID, d.Version, key, workspace, tools, string(writes), string(encoded), title.Text, title.Source, title.Model, overrides.Model, string(encodedModels))
if err != nil {
return 0, false, err
}
@@ -202,14 +244,19 @@ func (s *Store) SetRunTitle(ctx context.Context, id int64, title summarise.Title
const runSelect = `SELECT r.id,r.pipeline_id,p.name,r.version,v.content_hash,r.status,r.outcome,r.current_stage,
strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',r.finished_at),''),r.diagnostic,
CASE WHEN r.status='succeeded' THEN r.output_json ELSE COALESCE((SELECT output_json FROM run_steps WHERE run_id=r.id AND status='completed' ORDER BY sequence DESC LIMIT 1),'{}') END,
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title,r.model,r.stage_models
FROM pipeline_runs r JOIN pipelines p ON p.id=r.pipeline_id JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version`
func scanRun(row interface{ Scan(...any) error }, r *protocol.RunSummary) error {
var output string
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title); err != nil {
var output, stageModels string
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title, &r.Model, &stageModels); err != nil {
return err
}
if stageModels != "" && stageModels != "{}" {
if err := json.Unmarshal([]byte(stageModels), &r.StageModels); err != nil {
return err
}
}
var values map[string]any
if err := decodeValues(output, &values); err != nil {
return err
@@ -283,7 +330,7 @@ func (s *Store) GetRun(ctx context.Context, id int64) (*protocol.RunDetail, erro
if err := json.Unmarshal([]byte(paths), &r.WritePaths); err != nil {
return nil, err
}
rows, err := tx.QueryContext(ctx, `SELECT sequence,stage_key,status,outcome,target,input_json,output_json,diagnostic,session_id,agent_active,transcript,
rows, err := tx.QueryContext(ctx, `SELECT sequence,stage_key,status,outcome,target,input_json,output_json,diagnostic,session_id,agent_active,transcript,provider,model,
COALESCE((SELECT receipts_json FROM step_notifications n WHERE n.run_id=run_steps.run_id AND n.sequence=run_steps.sequence),'[]'),
COALESCE((SELECT changes_json FROM step_file_changes f WHERE f.run_id=run_steps.run_id AND f.sequence=run_steps.sequence),'null'),
COALESCE((SELECT artifacts_json FROM step_artifacts a WHERE a.run_id=run_steps.run_id AND a.sequence=run_steps.sequence),'null'),
@@ -297,7 +344,7 @@ strftime('%Y-%m-%dT%H:%M:%SZ',started_at),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ'
var step protocol.RunStep
var notifications string
var fileChanges, artifacts, storedVerdict string
if err := rows.Scan(&step.Sequence, &step.Stage, &step.Status, &step.Outcome, &step.Target, &input, &output, &step.Diagnostic, &step.SessionID, &step.AgentActive, &step.Transcript, &notifications, &fileChanges, &artifacts, &storedVerdict, &step.StartedAt, &step.FinishedAt); err != nil {
if err := rows.Scan(&step.Sequence, &step.Stage, &step.Status, &step.Outcome, &step.Target, &input, &output, &step.Diagnostic, &step.SessionID, &step.AgentActive, &step.Transcript, &step.Provider, &step.Model, &notifications, &fileChanges, &artifacts, &storedVerdict, &step.StartedAt, &step.FinishedAt); err != nil {
rows.Close()
return nil, err
}
@@ -537,8 +584,8 @@ func (s *Store) FinishStep(ctx context.Context, id int64, step protocol.RunStep)
return err
}
defer tx.Rollback()
if err := affected(tx.ExecContext(ctx, `UPDATE run_steps SET status=?,outcome=?,target=?,output_json=?,diagnostic=?,transcript=?,finished_at=CURRENT_TIMESTAMP
WHERE run_id=? AND sequence=? AND status='running'`, step.Status, step.Outcome, step.Target, string(output), step.Diagnostic, step.Transcript, id, step.Sequence)); err != nil {
if err := affected(tx.ExecContext(ctx, `UPDATE run_steps SET status=?,outcome=?,target=?,output_json=?,diagnostic=?,transcript=?,provider=?,model=?,finished_at=CURRENT_TIMESTAMP
WHERE run_id=? AND sequence=? AND status='running'`, step.Status, step.Outcome, step.Target, string(output), step.Diagnostic, step.Transcript, step.Provider, step.Model, id, step.Sequence)); err != nil {
return err
}
if step.FileChanges != nil {
+112 -1
View File
@@ -8,6 +8,7 @@ import (
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/summarise"
)
@@ -58,7 +59,7 @@ func TestRunTitleStorageMigrationAndFallback(t *testing.T) {
definition, _ := store.Get(ctx, items[0].ID, 1)
explicit, _, err := store.CreateRunTitled(ctx, definition, "explicit", "workspace", false, nil,
map[string]any{"prompt": "a prompt that must not become the title"},
map[string]any{"prompt": "a prompt that must not become the title"}, RunOverrides{},
summarise.Title{Text: "Fix the flaky login test", Source: summarise.SourceModel, Model: "openai/gpt"})
if err != nil {
t.Fatal(err)
@@ -160,6 +161,116 @@ func TestSetRunTitleRenamesWithoutTouchingInput(t *testing.T) {
}
}
// TestRunModelOverrideRoundTripAndRequestKey proves the launch-time model
// overrides round-trip through the run record and the per-step provider/model,
// and that a request-key replay with different overrides is refused as a
// different run.
func TestRunModelOverrideRoundTripAndRequestKey(t *testing.T) {
ctx := context.Background()
store, err := Open(filepath.Join(t.TempDir(), "override.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "override", Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
definition, _ := store.Get(ctx, items[0].ID, 1)
overrides := RunOverrides{Model: "acme/whole", StageModels: map[string]string{"one": "acme/strong"}}
id, created, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
map[string]any{"prompt": "task"}, overrides, summarise.Title{})
if err != nil || !created {
t.Fatalf("create override run = %d %v %v", id, created, err)
}
if err := store.BeginStep(ctx, id, 1, "one", map[string]any{"prompt": "task"}); err != nil {
t.Fatal(err)
}
if err := store.FinishStep(ctx, id, protocol.RunStep{Sequence: 1, Stage: "one", Status: "completed", Outcome: "success", Output: map[string]any{}, Provider: "acme", Model: "strong"}); err != nil {
t.Fatal(err)
}
got, err := store.GetRun(ctx, id)
if err != nil {
t.Fatal(err)
}
if got.Model != "acme/whole" || got.StageModels["one"] != "acme/strong" {
t.Fatalf("run record did not echo overrides: %+v", got)
}
if len(got.Steps) != 1 || got.Steps[0].Provider != "acme" || got.Steps[0].Model != "strong" {
t.Fatalf("step did not record the effective model: %+v", got.Steps)
}
list, err := store.ListRuns(ctx)
if err != nil || len(list) != 1 || list[0].Model != "acme/whole" || list[0].StageModels["one"] != "acme/strong" {
t.Fatalf("run list omitted the overrides: %+v %v", list, err)
}
// An idempotent replay with the same overrides returns the same run.
again, created, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
map[string]any{"prompt": "task"}, overrides, summarise.Title{})
if err != nil || created || again != id {
t.Fatalf("same-override replay = %d %v %v, want the stored run", again, created, err)
}
// A different override under the same key is a different run.
if _, _, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
map[string]any{"prompt": "task"}, RunOverrides{StageModels: map[string]string{"one": "acme/other"}}, summarise.Title{}); err == nil || !strings.Contains(err.Error(), "request key already used") {
t.Fatalf("differing-override replay was not refused: %v", err)
}
if _, _, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
map[string]any{"prompt": "task"}, RunOverrides{}, summarise.Title{}); err == nil || !strings.Contains(err.Error(), "request key already used") {
t.Fatalf("no-override replay was not refused: %v", err)
}
}
// TestRunModelOverrideMigrationKeepsHistory proves a database created before the
// override columns existed gains them idempotently on Open and still lists and
// reads its historical rows with empty overrides.
func TestRunModelOverrideMigrationKeepsHistory(t *testing.T) {
ctx := context.Background()
path := filepath.Join(t.TempDir(), "override-migration.db")
store, err := Open(path)
if err != nil {
t.Fatal(err)
}
if _, err := store.Submit(ctx, "legacy", Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
definition, _ := store.Get(ctx, items[0].ID, 1)
id, _, err := store.CreateRun(ctx, definition, "legacy-key", "workspace", false, nil, map[string]any{"prompt": "task"})
if err != nil {
t.Fatal(err)
}
// Simulate a pre-override database by dropping the columns, then reopen.
for _, column := range []string{"model", "stage_models"} {
if _, err := store.db.ExecContext(ctx, "ALTER TABLE pipeline_runs DROP COLUMN "+column); err != nil {
t.Fatal(err)
}
}
for _, column := range []string{"provider", "model"} {
if _, err := store.db.ExecContext(ctx, "ALTER TABLE run_steps DROP COLUMN "+column); err != nil {
t.Fatal(err)
}
}
if err := store.Close(); err != nil {
t.Fatal(err)
}
store, err = Open(path)
if err != nil {
t.Fatal(err)
}
defer store.Close()
runs, err := store.ListRuns(ctx)
if err != nil || len(runs) != 1 {
t.Fatalf("legacy run did not survive the override migration: %+v %v", runs, err)
}
got, err := store.GetRun(ctx, id)
if err != nil {
t.Fatal(err)
}
if got.Model != "" || len(got.StageModels) != 0 {
t.Fatalf("migrated row should keep empty overrides: %+v", got)
}
}
func TestRunRecoveryAndVersionPinning(t *testing.T) {
ctx := context.Background()
path := filepath.Join(t.TempDir(), "audit.db")
+7 -1
View File
@@ -147,7 +147,13 @@ type Message struct {
WritePaths []string `json:"write_paths,omitempty"`
RequestKey string `json:"request_key,omitempty"`
Workspace string `json:"workspace,omitempty"`
Limit int `json:"limit,omitempty"`
// Model and StageModels are the launch-time model overrides: Model applies to
// every model stage of the run and StageModels overrides individual stages by
// key (per-stage wins over whole-run). Both are optional so older peers ignore
// them; the same fields are echoed on the run record.
Model string `json:"model,omitempty"`
StageModels map[string]string `json:"stage_models,omitempty"`
Limit int `json:"limit,omitempty"`
// Chat is the brokered OpenCode chat traffic (route B). See chat.go for the
// per-type payload structs and the wire shapes they mirror.
ChatSessions []ChatSession `json:"chat_sessions,omitempty"`
+11
View File
@@ -118,6 +118,12 @@ type RunSummary struct {
Diagnostic string `json:"diagnostic"`
ResultPreview string `json:"result_preview,omitempty"`
CompletedSteps int `json:"completed_steps"`
// Model and StageModels are the launch-time overrides this run records so it
// stays reproducible: Model covers every model stage and StageModels overrides
// individual stages by key. They are empty for a run that used the pinned
// models and for historical rows.
Model string `json:"model,omitempty"`
StageModels map[string]string `json:"stage_models,omitempty"`
}
// Label is the operator-facing name of a run: the generated Title when the
@@ -147,6 +153,11 @@ type RunStep struct {
SessionID string `json:"session_id"`
AgentActive bool `json:"agent_active"`
Transcript string `json:"transcript"`
// Provider and Model are the effective provider/model this visit actually ran
// with, recorded for every model stage (including failures). They are empty
// for non-model stages and historical steps.
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
// ArtifactsSkipped and ArtifactsTruncated count files excluded from the
// bounded per-step scan (over the file/size cap) and by the total budget.
ArtifactsSkipped int `json:"artifacts_skipped,omitempty"`
+192
View File
@@ -0,0 +1,192 @@
package server
import (
"context"
"errors"
"path/filepath"
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
)
// TestRunnerAppliesModelOverrides proves a run with exactly one overridden stage
// records that step's provider/model as the override and the other model stage as
// the pinned pair, that the same effective values reach the resolved ModelConfig
// (the single source the repair/formatter path reads), and that the run record
// and returned graph echo the overrides while the pinned source is unchanged.
func TestRunnerAppliesModelOverrides(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "overrides.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "overrides", pipeline.Template([]string{"one", "two"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
definition, err := store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
pinned := map[string]stage.ModelConfig{}
for _, s := range definition.Graph.Stages {
pinned[s.Key] = stage.ModelConfig{Provider: s.Configs["provider"], Model: s.Configs["model"]}
}
r, err := newRunner(ctx, store)
if err != nil {
t.Fatal(err)
}
resolved := map[string]stage.ModelConfig{}
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
spec, err := (stage.ModelStage{}).Spec(def)
if err != nil {
return stage.Result{}, err
}
resolved[def.Key] = spec.Config
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
run, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "one-override", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}, StageModels: map[string]string{"one": "acme/strong"}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
if got := resolved["one"]; got.Provider != "acme" || got.Model != "strong" {
t.Fatalf("stage one resolved %+v, want the override", got)
}
if got := resolved["two"]; got.Provider != pinned["two"].Provider || got.Model != pinned["two"].Model {
t.Fatalf("stage two resolved %+v, want the pinned %+v", got, pinned["two"])
}
detail, err := store.GetRun(ctx, run.ID)
if err != nil {
t.Fatal(err)
}
if detail.Model != "" || detail.StageModels["one"] != "acme/strong" {
t.Fatalf("run record did not echo the overrides: %+v", detail)
}
if len(detail.Steps) != 2 {
t.Fatalf("steps = %+v", detail.Steps)
}
if detail.Steps[0].Provider != "acme" || detail.Steps[0].Model != "strong" {
t.Fatalf("overridden step evidence = %+v", detail.Steps[0])
}
if detail.Steps[1].Provider != pinned["two"].Provider || detail.Steps[1].Model != pinned["two"].Model {
t.Fatalf("pinned step evidence = %+v, want %+v", detail.Steps[1], pinned["two"])
}
// The returned graph shows the effective pair for the overridden stage and
// leaves the other at its pinned configs; `show --version N` (the immutable
// source) is untouched.
overlaid, err := r.getRun(run.ID)
if err != nil {
t.Fatal(err)
}
configs := map[string]map[string]string{}
for _, s := range overlaid.Graph.Stages {
configs[s.Key] = s.Configs
}
if configs["one"]["provider"] != "acme" || configs["one"]["model"] != "strong" {
t.Fatalf("overlaid graph stage one = %+v, want the override", configs["one"])
}
if configs["two"]["provider"] != pinned["two"].Provider || configs["two"]["model"] != pinned["two"].Model {
t.Fatalf("overlaid graph stage two = %+v, want the pinned pair", configs["two"])
}
loaded, err := store.Get(ctx, definition.ID, 1)
if err != nil {
t.Fatal(err)
}
if loaded.Source != definition.Source {
t.Fatal("the pinned immutable source was rewritten by an override")
}
}
// TestRunnerRefusesInvalidModelOverridesWithoutRunRow proves an unknown stage key// and a malformed provider/model both fail before admission and create no run row.
func TestRunnerRefusesInvalidModelOverridesWithoutRunRow(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "reject.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "reject", pipeline.Template([]string{"one", "two"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
r, err := newRunner(ctx, store)
if err != nil {
t.Fatal(err)
}
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
}
before, _ := store.ListRuns(ctx)
cases := []struct {
name string
key string
model string
stage map[string]string
want string
}{
{"unknown stage key", "unknown-key", "", map[string]string{"nope": "acme/strong"}, "unknown stage key"},
{"invalid whole-run pair", "invalid-model", "not-a-pair", nil, "provider/model"},
{"invalid stage pair", "invalid-stage", "", map[string]string{"one": "acme/"}, "provider/model"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
_, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: tc.key, Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}, Model: tc.model, StageModels: tc.stage})
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatalf("error = %v, want %q", err, tc.want)
}
if id, err := store.RunByKey(ctx, tc.key); err != nil || id != 0 {
t.Fatalf("rejected request created run %d: %v", id, err)
}
})
}
after, _ := store.ListRuns(ctx)
if len(after) != len(before) {
t.Fatalf("rejected requests created run rows: before %d after %d", len(before), len(after))
}
}
// TestRetryReusesModelOverrides proves a retry of a failed run reuses the
// original run's recorded overrides, so the new attempt is reproducible.
func TestRetryReusesModelOverrides(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
store, err := pipeline.Open(filepath.Join(t.TempDir(), "retry-override.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "retry-override", pipeline.Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
r, err := newRunner(ctx, store)
if err != nil {
t.Fatal(err)
}
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
return stage.Result{}, errors.New("provider failed")
}
first, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "first", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}, StageModels: map[string]string{"one": "acme/strong"}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
if detail, _ := store.GetRun(ctx, first.ID); detail.Status != "failed" {
t.Fatalf("original run did not fail: %+v", detail)
}
retried, err := r.retry(protocol.Message{RunID: first.ID, RequestKey: "retry"})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
if retried.StageModels["one"] != "acme/strong" || retried.Model != "" {
t.Fatalf("retry did not reuse the original overrides: %+v", retried)
}
}
+73 -3
View File
@@ -141,6 +141,13 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
if _, err := pipeline.Parse(pipeline.ExecutionSource(d.Source)); err != nil {
return nil, err
}
// Validate the launch-time model overrides against the pinned graph before
// admission: an unknown stage key or a malformed provider/model must never
// insert a run row. The resolution is O(#overrides + #stages) with no model or
// network call, and the same helper is the authoritative precedence rule.
if _, err := pipeline.EffectiveModels(d.Graph, request.Model, request.StageModels); err != nil {
return nil, err
}
for _, s := range d.Graph.Stages {
if stage.EffectiveType(s.Type) != stage.ModelType {
continue
@@ -229,7 +236,7 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
return nil, err
}
}
id, created, err := r.store.CreateRunTitled(r.ctx, d, request.RequestKey, workspace, request.AllowTools, request.WritePaths, request.Input, runTitle, retryOf...)
id, created, err := r.store.CreateRunTitled(r.ctx, d, request.RequestKey, workspace, request.AllowTools, request.WritePaths, request.Input, pipeline.RunOverrides{Model: request.Model, StageModels: request.StageModels}, runTitle, retryOf...)
if err != nil {
return nil, err
}
@@ -241,6 +248,7 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
detail, err := r.store.GetRun(r.ctx, id)
if detail != nil {
_, detail.Executing = r.active[id]
r.overlayEffectiveModels(detail)
}
return detail, err
}
@@ -285,10 +293,41 @@ func (r *runner) getRun(id int64) (*protocol.RunDetail, error) {
d.Executing = true
}
}
r.overlayEffectiveModels(d)
}
return d, err
}
// overlayEffectiveModels replaces each model stage's provider/model in a returned
// run's graph with the effective pair the run actually uses, so the app and TUI
// stage-definition surfaces show the override with no client change. It is
// presentation only: the persisted graph_json and the pinned source returned by
// `show` are never rewritten. A run with no overrides is left untouched.
func (r *runner) overlayEffectiveModels(d *protocol.RunDetail) {
if d == nil || (d.Model == "" && len(d.StageModels) == 0) {
return
}
effective, err := pipeline.EffectiveModels(d.Graph, d.Model, d.StageModels)
if err != nil {
return
}
for i := range d.Graph.Stages {
s := &d.Graph.Stages[i]
if stage.EffectiveType(s.Type) != stage.ModelType {
continue
}
override, ok := effective[s.Key]
if !ok {
continue
}
if s.Configs == nil {
s.Configs = map[string]string{}
}
s.Configs["provider"] = override.Provider
s.Configs["model"] = override.Model
}
}
func (r *runner) listRuns() ([]protocol.RunSummary, error) {
r.mu.Lock()
defer r.mu.Unlock()
@@ -324,7 +363,7 @@ func (r *runner) retry(request protocol.Message) (*protocol.RunDetail, error) {
if original.Status != "failed" || original.FinishedAt == "" {
return nil, fmt.Errorf("only failed runs can be retried (run %d is %s)", original.ID, original.Status)
}
return r.start(protocol.Message{PipelineID: original.PipelineID, Version: original.Version, RequestKey: request.RequestKey, Workspace: original.Workspace, Input: original.Input, AllowTools: original.AllowTools, WritePaths: original.WritePaths}, original.ID)
return r.start(protocol.Message{PipelineID: original.PipelineID, Version: original.Version, RequestKey: request.RequestKey, Workspace: original.Workspace, Input: original.Input, AllowTools: original.AllowTools, WritePaths: original.WritePaths, Model: original.Model, StageModels: original.StageModels}, original.ID)
}
func (r *runner) cancel(id int64) error {
@@ -427,7 +466,7 @@ func (r *runner) resume(ctx context.Context, id int64, d *protocol.RunDetail, wo
return
}
state := rebuildRunState(d, step)
r.runLoop(ctx, id, d.Graph, d.Name, workspace, protocol.Message{WritePaths: d.WritePaths, AllowTools: d.AllowTools}, state)
r.runLoop(ctx, id, d.Graph, d.Name, workspace, protocol.Message{WritePaths: d.WritePaths, AllowTools: d.AllowTools, Model: d.Model, StageModels: d.StageModels}, state)
}
// runState is the loop's whole mutable state, extracted so a fresh run and a
@@ -486,6 +525,20 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
finish("failed", "error", "stage returned an undeclared outcome")
return
}
// Resolve the effective provider/model for every model stage once, before the
// loop, so applying an override is an in-memory O(1) map lookup per visit and
// never re-parses the graph or re-resolves the whole definition. A run with no
// overrides skips the resolution entirely. Launch validation already ran this
// helper, so a failure here only guards a resumed run whose overrides drifted.
var effectiveModels map[string]pipeline.ModelOverride
if request.Model != "" || len(request.StageModels) > 0 {
resolved, err := pipeline.EffectiveModels(graph, request.Model, request.StageModels)
if err != nil {
finish("failed", "error", err.Error())
return
}
effectiveModels = resolved
}
for sequence := state.nextSequence; sequence <= 30; sequence++ {
state.nextSequence = sequence + 1
if ctx.Err() != nil {
@@ -508,6 +561,16 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
finish("failed", "error", "missing stage")
return
}
// StageDefinition clones Configs, so setting the effective pair on the
// clone is safe and the pinned graph is never mutated. Because
// ModelStage.Spec reads def.Configs and passes spec.Config into the repair
// path, the same effective model governs the stage and its repair.
if stage.EffectiveType(def.Type) == stage.ModelType {
if override, ok := effectiveModels[key]; ok {
def.Configs["provider"] = override.Provider
def.Configs["model"] = override.Model
}
}
isHuman := stage.EffectiveType(def.Type) == stage.HumanType
state.visits[key]++
maxVisits := 5
@@ -672,6 +735,13 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
}
artifacts, artifactsSkipped, artifactsTruncated := r.scanArtifacts(id, sequence, artifactsDir)
step := protocol.RunStep{Sequence: sequence, Stage: key, Status: "completed", Outcome: result.Outcome, Output: result.Output, Transcript: operations.Transcript(client.Transcript), SessionID: stageSession, FileChanges: changes, Artifacts: artifacts, ArtifactsSkipped: artifactsSkipped, ArtifactsTruncated: artifactsTruncated}
// Stamp the effective provider/model this visit actually ran with for every
// model stage, including failures, so the app/TUI evidence reflects any
// override. Non-model stages keep empty values.
if stage.EffectiveType(def.Type) == stage.ModelType {
step.Provider = modelConfig.Provider
step.Model = modelConfig.Model
}
if budget := budgets[key]; budget != nil {
step.Transcript = stage.BudgetTranscript(step.Transcript, budget, modelConfig.Timeout, modelConfig.Retries)
}
+4
View File
@@ -13,6 +13,10 @@ type RunOptions struct {
Pipeline, Prompt, Title, Workspace string
Yes, AllowTools bool
WritePaths []string
// Model and StageModels are the CLI's launch-time model overrides, passed
// through to the run request without adding a TUI picker (that is a follow-up).
Model string
StageModels map[string]string
}
func NewRun(ctx context.Context, url string, options RunOptions) Model {
+5
View File
@@ -359,6 +359,11 @@ func evidenceLines(step protocol.RunStep) []string {
if step.SessionID == "" {
lines = []string{"Recorded result evidence"}
}
// The effective provider/model this visit ran with, for a model stage; it
// reflects any launch-time override rather than the pinned model.
if step.Provider != "" || step.Model != "" {
lines = append(lines, "Ran with model: "+step.Provider+"/"+step.Model)
}
if step.Transcript == "" {
return append(lines, "No captured tool evidence yet.")
}
+24
View File
@@ -0,0 +1,24 @@
package tui
import (
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// TestEvidenceLinesShowEffectiveModel proves the Evidence pane renders the
// provider/model a step actually ran with, so a launch-time override is visible
// rather than the pinned model.
func TestEvidenceLinesShowEffectiveModel(t *testing.T) {
step := protocol.RunStep{Sequence: 1, Stage: "one", Status: "completed", Outcome: "success", Provider: "acme", Model: "strong"}
text := strings.Join(evidenceLines(step), "\n")
if !strings.Contains(text, "acme/strong") {
t.Fatalf("evidence pane omitted the effective model:\n%s", text)
}
// A non-model step records no provider/model and shows no model line.
plain := strings.Join(evidenceLines(protocol.RunStep{Sequence: 2, Stage: "note", Status: "completed", Outcome: "success"}), "\n")
if strings.Contains(plain, "Ran with model:") {
t.Fatalf("non-model step claimed a model:\n%s", plain)
}
}
+1 -1
View File
@@ -147,7 +147,7 @@ func (m *Model) startSelectedRun() tea.Cmd {
m.runCollapsed = nil
m.runDetail = nil
m.runTop = 0
request := protocol.Message{Type: protocol.RunPipeline, PipelineID: m.detail.ID, Version: m.detail.Version, RequestKey: fmt.Sprintf("tui-%d", time.Now().UnixNano()), Title: m.cliOptions.Title, Input: initialRunInput(m.runTask), AllowTools: m.runHasMutationTools(), WritePaths: m.runPaths, Workspace: m.runWorkspace}
request := protocol.Message{Type: protocol.RunPipeline, PipelineID: m.detail.ID, Version: m.detail.Version, RequestKey: fmt.Sprintf("tui-%d", time.Now().UnixNano()), Title: m.cliOptions.Title, Input: initialRunInput(m.runTask), AllowTools: m.runHasMutationTools(), WritePaths: m.runPaths, Workspace: m.runWorkspace, Model: m.cliOptions.Model, StageModels: m.cliOptions.StageModels}
updated, cmd := m.runRequest(request)
*m = updated.(Model)
return cmd