Allow launching runs with whole-run and per-stage model overrides
Add `pipelines run --model PROVIDER/MODEL` and repeatable `--stage-model STAGE=PROVIDER/MODEL`, resolved with precedence per-stage > whole-run > the model pinned in the pipeline version. - A shared pure resolver (`pipeline.ParseModelOverride` / `pipeline.EffectiveModels`) owns the precedence and validation: every value must be a `provider/model` pair (`stage.ValidModel`), every stage key must exist in the pinned graph, an unknown key fails naming the valid keys, and a per-stage key that names a non-model stage is refused. - The server validates against the pinned version before admission (no run row is ever inserted for an invalid request), persists `model`/`stage_models` on the run and `provider`/`model` on each step, applies the effective pair to a clone of the stage definition so the stage and its repair/formatter path share it, overlays the returned run graph with the effective pair (presentation only), and reuses the recorded overrides on retry. - The run record echoes the overrides it was launched with and each step records the provider/model it actually ran with; the TUI Evidence pane renders that effective pair. The pinned pipeline version and `show --version N` are never modified. - CLI help, `docs/cli.md` and the packaged authoring skill references document the flags and the precedence. A TUI override picker remains a documented follow-up; the CLI is the deliverable. Tests: pure precedence/validation table tests, store round-trip and request-key-mismatch plus migration, server admission/application/overlay/retry with a controlled executor, CLI flag/help/malformed-value tests, and a TUI evidence test.
This commit is contained in:
+36
-1
@@ -219,7 +219,7 @@ Submitting activates included schedules, so review them before publishing.
|
||||
| Browse templates/source | `templates --json`, `template KEY --source` |
|
||||
| Create or edit a definition | `new`, `edit`, `drafts` |
|
||||
| Validate and publish | `validate FILE`, `submit FILE` |
|
||||
| Start a run | `run NAME PROMPT [--title TITLE] --yes --json` |
|
||||
| Start a run | `run NAME PROMPT [--title TITLE] [--model P/M] [--stage-model S=P/M] --yes --json` |
|
||||
| Run history/inspection | `runs --json`, `inspect ID --json` |
|
||||
| Rename a run | `title ID "TITLE" --json` |
|
||||
| Decide a waiting human stage | `decide ID [STAGE] --option KEY [--feedback TEXT] --json` |
|
||||
@@ -262,6 +262,41 @@ pipeline are distinguishable. Rename after the fact with `pipelines title
|
||||
<run-id> "<title>"`; the run's input, steps and execution are unchanged. The
|
||||
stored prompt is unchanged, and listing/renaming never calls a model.
|
||||
|
||||
### Model overrides at launch
|
||||
|
||||
A pipeline version pins one model per stage. Without republishing, a run can
|
||||
choose different models at launch:
|
||||
|
||||
```sh
|
||||
pipelines run "bug fix" "Fix the layout" --json --yes \
|
||||
--model openai/gpt-5 \
|
||||
--stage-model review=openai/o4-mini
|
||||
```
|
||||
|
||||
- `--model PROVIDER/MODEL` overrides the model for **every** model stage of that
|
||||
run.
|
||||
- `--stage-model STAGE=PROVIDER/MODEL` overrides **one** stage and is repeatable,
|
||||
so any number of stages can be overridden. A repeated key for the same stage is
|
||||
last-wins.
|
||||
- Precedence: a per-stage override beats the whole-run override, which beats the
|
||||
model pinned in the pipeline version.
|
||||
|
||||
Overrides apply to that run only. The pipeline definition and its immutable
|
||||
version are never modified, `show <name> --version N` still returns the pinned
|
||||
source and graph, and the run records the `model` and `stage_models` it was
|
||||
launched with so it stays reproducible; `retry` reuses them. At launch every
|
||||
value must be a `provider/model` pair of the configured form and every stage key
|
||||
must exist in the pinned version — an unknown key fails naming the valid keys and
|
||||
no run is created. A per-stage key that names a non-model stage is refused
|
||||
because that stage never runs a model; `--model` on a pipeline with no model stage
|
||||
is accepted and inert. The effective model reaches the stage and its
|
||||
repair/formatter path from the same resolved definition, and each step's evidence
|
||||
records the provider/model it actually ran with (`inspect <id> --json` exposes
|
||||
per-step `provider`/`model`, and the run's graph shows the effective pair). The
|
||||
CLI is the deliverable; a TUI picker for choosing overrides is a documented
|
||||
follow-up, not half-built.
|
||||
|
||||
|
||||
On either the runs list or run inspector, press `P` (or use `:pipeline`) to open
|
||||
the selected run's exact pipeline version. Navigation does not cancel the run.
|
||||
|
||||
|
||||
@@ -118,6 +118,16 @@ preserved and partial stdout after the grace is never accepted as success.
|
||||
|
||||
Stages set `harness=pi` or `harness=opencode` in configs; omission preserves
|
||||
historical OpenCode behavior. Pi must be configured on the execution server.
|
||||
A run may override the pinned model at launch: `pipelines run --model
|
||||
PROVIDER/MODEL` covers every model stage and repeatable `--stage-model
|
||||
STAGE=PROVIDER/MODEL` overrides one stage, with precedence **per-stage >
|
||||
whole-run > the model pinned in the pipeline version**. Overrides are validated
|
||||
against the pinned version before the run starts (an unknown stage key fails
|
||||
naming the valid keys; a non-model stage cannot be overridden), apply to that run
|
||||
only, and are recorded on the run so `retry` reproduces them. The effective
|
||||
provider/model is stamped on the step and exposed by the run's graph, and the
|
||||
stage's repair/formatter path uses the same resolved model because both read the
|
||||
stage's effective configs.
|
||||
The RPC adapter starts new retained sessions per visit with explicit tools and
|
||||
a Pipelines permission extension. Ambient extensions, skills, context files,
|
||||
prompt templates, automatic retries/compaction are disabled. Tool gates are not
|
||||
|
||||
@@ -126,6 +126,13 @@ Adapt the scope and publishing lines per item; keep the rest.
|
||||
- Launch with the authoritative `CLI_PREFIX` (`pipelines run`). The brief text is the
|
||||
run's task; pass it whole. Read `references/operations.md` for the exact flags —
|
||||
`--json`, confirmation (`--yes`) and `--wait`/`--wait-timeout`.
|
||||
- **Model overrides at launch.** Use `--model PROVIDER/MODEL` to override the model
|
||||
for every model stage of that run, and repeatable `--stage-model
|
||||
STAGE=PROVIDER/MODEL` to override individual stages. Precedence is per-stage >
|
||||
whole-run > the model pinned in the pipeline version. Overrides apply to that run
|
||||
only (the pinned pipeline is never edited), are validated before the run starts,
|
||||
and are recorded on the run; `retry` reuses them. See `references/harnesses.md`
|
||||
for how a stage's model and its repair path both use the effective model.
|
||||
- Watch to a **terminal state** (`succeeded`, `failed`, `canceled`, `interrupted`).
|
||||
Poll `CLI_PREFIX inspect RUN_ID --json` (or use `--wait`) until terminal. A reached
|
||||
client wait limit is not a run failure: the run stays active; keep waiting or
|
||||
|
||||
+76
-5
@@ -10,6 +10,7 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -43,6 +44,41 @@ type options struct {
|
||||
json, source, noOpen, yes, allow, createOnly, force, wait bool
|
||||
version, expected, limit int
|
||||
timeout, waitTimeout time.Duration
|
||||
stageModels stageModelFlags
|
||||
}
|
||||
|
||||
// stageModelFlags collects the repeatable --stage-model STAGE=PROVIDER/MODEL
|
||||
// values. A repeated key is last-wins. It is a flag.Value so the repeatable flag
|
||||
// needs no dependency (Go stdlib has no slice flag).
|
||||
type stageModelFlags map[string]string
|
||||
|
||||
// String renders the collected overrides for flag's defaults display.
|
||||
func (s stageModelFlags) String() string {
|
||||
keys := make([]string, 0, len(s))
|
||||
for key := range s {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
parts := make([]string, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
parts = append(parts, key+"="+s[key])
|
||||
}
|
||||
return strings.Join(parts, ",")
|
||||
}
|
||||
|
||||
// Set parses one STAGE=PROVIDER/MODEL value and validates the provider/model with
|
||||
// the shared parser, so a malformed value fails before any run is created.
|
||||
func (s stageModelFlags) Set(value string) error {
|
||||
key, model, ok := strings.Cut(value, "=")
|
||||
key, model = strings.TrimSpace(key), strings.TrimSpace(model)
|
||||
if !ok || key == "" || model == "" {
|
||||
return fmt.Errorf("--stage-model requires STAGE=PROVIDER/MODEL")
|
||||
}
|
||||
if _, err := pipeline.ParseModelOverride(model); err != nil {
|
||||
return fmt.Errorf("--stage-model %s: %w", key, err)
|
||||
}
|
||||
s[key] = model
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a App) flags(command string) (*flag.FlagSet, *options) {
|
||||
@@ -98,6 +134,9 @@ func (a App) flags(command string) (*flag.FlagSet, *options) {
|
||||
f.StringVar(&o.paths, "write-paths", "", "Optional narrower edit scope; default is all files in the run directory")
|
||||
f.StringVar(&o.key, "request-key", "", "Idempotency key (default: generated unique key)")
|
||||
f.IntVar(&o.version, "version", 0, "Version to run (default: latest)")
|
||||
f.StringVar(&o.model, "model", "", "Override the model for every stage of this run (PROVIDER/MODEL)")
|
||||
o.stageModels = stageModelFlags{}
|
||||
f.Var(o.stageModels, "stage-model", "Override one stage's model; repeatable (STAGE=PROVIDER/MODEL); beats --model")
|
||||
f.BoolVar(&o.wait, "wait", false, "In JSON mode, wait for a terminal run state")
|
||||
f.DurationVar(&o.waitTimeout, "wait-timeout", 0, "Client limit for --wait after the run is admitted (0 waits until terminal); separate from per-request --timeout")
|
||||
case "runs":
|
||||
@@ -278,10 +317,11 @@ func (a App) Run(ctx context.Context, args []string) error {
|
||||
if len(pos) != 0 {
|
||||
return fmt.Errorf("provider-check takes no positional arguments")
|
||||
}
|
||||
provider, model, ok := strings.Cut(o.model, "/")
|
||||
if !ok || !stage.ValidModel(provider, model) {
|
||||
override, err := pipeline.ParseModelOverride(o.model)
|
||||
if err != nil {
|
||||
return fmt.Errorf("--model requires provider/model")
|
||||
}
|
||||
provider, model := override.Provider, override.Model
|
||||
if err := a.confirmAction("Validate "+o.model+" with one small tool-free generation (provider charges may apply)?", o.yes); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -1117,6 +1157,14 @@ func (a App) run(ctx context.Context, pos []string, o *options, call callFunc, r
|
||||
paths = append(paths, p)
|
||||
}
|
||||
}
|
||||
// Pre-flight form validation of the whole-run override with the same parser
|
||||
// the server uses; stage-key existence stays server-authoritative. A malformed
|
||||
// value fails here, before any confirmation or run request.
|
||||
if o.model != "" {
|
||||
if _, err := pipeline.ParseModelOverride(o.model); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if o.yes && (o.name == "" || strings.TrimSpace(o.prompt) == "") {
|
||||
return fmt.Errorf("--yes requires a pipeline and nonempty prompt")
|
||||
}
|
||||
@@ -1131,7 +1179,7 @@ func (a App) run(ctx context.Context, pos []string, o *options, call callFunc, r
|
||||
if o.wait || o.key != "" || o.version > 0 {
|
||||
return fmt.Errorf("--wait, --request-key and --version require --json")
|
||||
}
|
||||
_, err := tea.NewProgram(tui.NewRun(ctx, o.url, tui.RunOptions{Pipeline: o.name, Prompt: o.prompt, Title: o.title, Workspace: workspace, Yes: o.yes, AllowTools: o.allow, WritePaths: paths}), tea.WithContext(ctx), tea.WithAltScreen()).Run()
|
||||
_, err := tea.NewProgram(tui.NewRun(ctx, o.url, tui.RunOptions{Pipeline: o.name, Prompt: o.prompt, Title: o.title, Workspace: workspace, Yes: o.yes, AllowTools: o.allow, WritePaths: paths, Model: o.model, StageModels: o.stageModels}), tea.WithContext(ctx), tea.WithAltScreen()).Run()
|
||||
return err
|
||||
}
|
||||
if o.name == "" || strings.TrimSpace(o.prompt) == "" {
|
||||
@@ -1145,20 +1193,43 @@ func (a App) run(ctx context.Context, pos []string, o *options, call callFunc, r
|
||||
if version == 0 {
|
||||
version = p.Version
|
||||
}
|
||||
if err := a.confirmAction(fmt.Sprintf("Run %s v%d in %s? This may edit files, execute commands and publish notifications.", p.Name, version, workspace), o.yes); err != nil {
|
||||
confirmation := fmt.Sprintf("Run %s v%d in %s? This may edit files, execute commands and publish notifications.", p.Name, version, workspace) + o.modelSummary()
|
||||
if err := a.confirmAction(confirmation, o.yes); err != nil {
|
||||
return err
|
||||
}
|
||||
key := o.key
|
||||
if key == "" {
|
||||
key = fmt.Sprintf("cli-%d-%d", os.Getpid(), time.Now().UnixNano())
|
||||
}
|
||||
r, err := call(protocol.Message{Type: protocol.RunPipeline, PipelineID: p.ID, Version: version, RequestKey: key, Workspace: workspace, Title: o.title, Input: map[string]any{"prompt": o.prompt}, AllowTools: o.allow, WritePaths: paths})
|
||||
r, err := call(protocol.Message{Type: protocol.RunPipeline, PipelineID: p.ID, Version: version, RequestKey: key, Workspace: workspace, Title: o.title, Input: map[string]any{"prompt": o.prompt}, AllowTools: o.allow, WritePaths: paths, Model: o.model, StageModels: o.stageModels})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return a.finishRun(ctx, r, o, call)
|
||||
}
|
||||
|
||||
// modelSummary describes the launch-time model overrides for the action
|
||||
// confirmation, so a model mutation is visible before approval. It is empty when
|
||||
// no override was supplied.
|
||||
func (o *options) modelSummary() string {
|
||||
if o.model == "" && len(o.stageModels) == 0 {
|
||||
return ""
|
||||
}
|
||||
parts := []string{}
|
||||
if o.model != "" {
|
||||
parts = append(parts, "all stages: "+o.model)
|
||||
}
|
||||
keys := make([]string, 0, len(o.stageModels))
|
||||
for key := range o.stageModels {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
parts = append(parts, key+": "+o.stageModels[key])
|
||||
}
|
||||
return " Model overrides — " + strings.Join(parts, "; ") + "."
|
||||
}
|
||||
|
||||
func (a App) finishRun(ctx context.Context, r protocol.Message, o *options, call callFunc) error {
|
||||
var err error
|
||||
if o.wait {
|
||||
|
||||
@@ -16,7 +16,7 @@ var usage = map[string]string{
|
||||
"import": "import [FILE.zip] [--all] [--on-conflict ask|keep|both|import] [--yes] [--json]",
|
||||
"templates": "templates [--json]",
|
||||
"template": "template <key> [--source | --json]",
|
||||
"run": "run [name|id] [prompt] [--title TITLE] [--json [--wait [--wait-timeout DURATION]]]",
|
||||
"run": "run [name|id] [prompt] [--title TITLE] [--model PROVIDER/MODEL] [--stage-model STAGE=PROVIDER/MODEL ...] [--json [--wait [--wait-timeout DURATION]]]",
|
||||
"runs": "runs [--pipeline NAME] [--json]",
|
||||
"inspect": "inspect <run-id> [--json]",
|
||||
"cancel": "cancel <run-id> [--json]",
|
||||
@@ -44,7 +44,7 @@ var notes = map[string]string{
|
||||
"edit": "Omit the name to open fuzzy selection. Original immutable source is copied to a private draft. The configured harness edits that draft, never the live definition.",
|
||||
"validate": "Uses the same server parser as submission. Does not persist or activate schedules. Structural validity does not prove runtime/tool availability.",
|
||||
"submit": "Confirm publication at the prompt, or use --yes for explicit automation; schedules activate on acceptance. Managed drafts provide the name and base-version guard. --force never bypasses validation.",
|
||||
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
|
||||
"run": "Runs in your current directory after action confirmation (--yes skips it). Select a pipeline and compose a multiline task with Ctrl+S. Stage tools are enabled automatically; no file list or typed RUN confirmation. Cancellation does not undo edits. --title sets an explicit run title; without it the title is derived from the prompt. --model PROVIDER/MODEL overrides the model for every stage of that run; --stage-model STAGE=PROVIDER/MODEL overrides one stage and is repeatable, with a repeated key last-wins. Precedence is per-stage > whole-run > the model pinned in the pipeline version; overrides apply to that run only and never change the pinned definition. An unknown stage key or a malformed provider/model is refused before the run starts, naming the valid stage keys. In JSON mode --wait polls until terminal; --wait-timeout bounds only the client wait after admission and is independent of per-request --timeout. A reached wait limit leaves the run active: inspect read-only with `inspect <run-id> --json`, or reattach with --wait reusing the original --request-key (never an automatic replay or cancel).",
|
||||
"retry": "Start a new attempt of a failed run with the same initial input, directory, pipeline version, and edit scope. Original history is retained. Already-completed side effects are not undone. --wait/--wait-timeout behave as for run: a reached wait limit leaves the new run active; inspect read-only with `inspect <run-id> --json`, or reattach with --wait and the original --request-key instead of starting a duplicate retry.",
|
||||
"title": "Rename a run's operator-facing title after the fact without touching its input, steps or execution. The title is collapsed to a single bounded line and recorded as explicitly set. `pipelines runs` and `pipelines inspect` then show it.",
|
||||
"decide": "Choose an option for a run waiting on a human stage. The run must be `waiting`; the option must be one of that stage's non-error edges. The stage is inferred from the run's current stage unless named. The choice and optional --feedback are recorded as the step's evidence (visible in inspect and the TUI) and the run resumes along the chosen edge. An unknown option is refused with the valid list; a repeated decision for an already-decided visit does not double-apply. Works across a server restart because the wait is persisted.",
|
||||
|
||||
@@ -61,6 +61,13 @@ Examples:
|
||||
pipelines validate /path/to/draft.md --json
|
||||
pipelines submit /path/to/draft.md
|
||||
pipelines run "bug fix" "Fix the layout" --json
|
||||
pipelines run "bug fix" "Fix the layout" --model openai/gpt-5 --stage-model review=openai/o4 --json
|
||||
|
||||
Launch-time model overrides: 'pipelines run --model P/M' covers every model stage;
|
||||
'--stage-model STAGE=P/M' (repeatable; a repeated key is last-wins) overrides one
|
||||
stage and wins over --model, which wins over the model pinned in the pipeline
|
||||
version. Overrides apply to that run only, are recorded on the run, and are
|
||||
validated before the run starts (an unknown stage key names the valid keys).
|
||||
|
||||
Draft-first: the configured harness (Pi/OpenCode; host by default, or Docker-linked
|
||||
when a checkout is linked) edits a private draft; nvim remains a client editor.
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
// TestRunModelOverrideFlagsParse proves the repeatable --stage-model flag and the
|
||||
// whole-run --model flag parse onto options, with a repeated key last-wins.
|
||||
func TestRunModelOverrideFlagsParse(t *testing.T) {
|
||||
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
|
||||
f, o := app.flags("run")
|
||||
if err := parse(f, []string{"--pipeline", "p", "--prompt", "task", "--model", "acme/whole", "--stage-model", "one=acme/first", "--stage-model", "two=acme/two", "--stage-model", "one=acme/last"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if o.model != "acme/whole" {
|
||||
t.Fatalf("--model = %q", o.model)
|
||||
}
|
||||
if o.stageModels["one"] != "acme/last" {
|
||||
t.Fatalf("repeated --stage-model key not last-wins: %+v", o.stageModels)
|
||||
}
|
||||
if o.stageModels["two"] != "acme/two" {
|
||||
t.Fatalf("second --stage-model missing: %+v", o.stageModels)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStageModelFlagRejectsMalformedValues proves a value that is not
|
||||
// STAGE=PROVIDER/MODEL fails at flag parse, before any run is created.
|
||||
func TestStageModelFlagRejectsMalformedValues(t *testing.T) {
|
||||
for _, value := range []string{"nope", "one", "=acme/x", "one=", "one=not-a-pair", "one=acme/"} {
|
||||
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
|
||||
f, _ := app.flags("run")
|
||||
if err := parse(f, []string{"--stage-model", value}); err == nil {
|
||||
t.Fatalf("malformed --stage-model %q was accepted", value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunRejectsInvalidWholeRunModel proves the whole-run --model is form-checked
|
||||
// with the shared parser before any server request.
|
||||
func TestRunRejectsInvalidWholeRunModel(t *testing.T) {
|
||||
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
|
||||
called := false
|
||||
call := func(protocol.Message) (protocol.Message, error) { called = true; return protocol.Message{}, nil }
|
||||
resolve := func(string) (protocol.PipelineSummary, error) { return protocol.PipelineSummary{}, nil }
|
||||
o := &options{json: true, yes: true, model: "not-a-pair", prompt: "task"}
|
||||
err := app.run(context.Background(), []string{"pipe"}, o, call, resolve)
|
||||
if err == nil || !strings.Contains(err.Error(), "provider/model") {
|
||||
t.Fatalf("invalid --model error = %v", err)
|
||||
}
|
||||
if called {
|
||||
t.Fatal("an invalid --model still sent a run request")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHelpDocumentsModelOverrides proves the usage map, notes and flag help
|
||||
// advertise --model and repeatable --stage-model and state the precedence.
|
||||
func TestHelpDocumentsModelOverrides(t *testing.T) {
|
||||
if !strings.Contains(usage["run"], "--model") || !strings.Contains(usage["run"], "--stage-model") {
|
||||
t.Fatalf("run usage does not document the model overrides: %q", usage["run"])
|
||||
}
|
||||
note := notes["run"]
|
||||
for _, want := range []string{"--model", "--stage-model", "per-stage", "pinned"} {
|
||||
if !strings.Contains(note, want) {
|
||||
t.Fatalf("run note does not document %q: %q", want, note)
|
||||
}
|
||||
}
|
||||
app := App{Out: &bytes.Buffer{}, Err: &bytes.Buffer{}}
|
||||
f, _ := app.flags("run")
|
||||
f.Usage()
|
||||
help := app.Out.(*bytes.Buffer).String()
|
||||
if !strings.Contains(help, "--model") || !strings.Contains(help, "--stage-model") {
|
||||
t.Fatalf("run --help omits the overrides:\n%s", help)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/stage"
|
||||
)
|
||||
|
||||
// ModelOverride is a provider/model pair chosen for a run.
|
||||
type ModelOverride struct {
|
||||
Provider string
|
||||
Model string
|
||||
}
|
||||
|
||||
// String renders the pair as provider/model.
|
||||
func (m ModelOverride) String() string { return m.Provider + "/" + m.Model }
|
||||
|
||||
// ParseModelOverride parses a provider/model value and validates both sides with
|
||||
// stage.ValidModel. It is the one parser shared by launch-time resolution and the
|
||||
// CLI's pre-flight form check, so the two cannot drift.
|
||||
func ParseModelOverride(value string) (ModelOverride, error) {
|
||||
provider, model, ok := strings.Cut(strings.TrimSpace(value), "/")
|
||||
provider, model = strings.TrimSpace(provider), strings.TrimSpace(model)
|
||||
if !ok || !stage.ValidModel(provider, model) {
|
||||
return ModelOverride{}, fmt.Errorf("model override %q must be provider/model", value)
|
||||
}
|
||||
return ModelOverride{Provider: provider, Model: model}, nil
|
||||
}
|
||||
|
||||
// EffectiveModels resolves the provider/model each model stage will run with for
|
||||
// one run. Precedence: a per-stage override beats the whole-run override, which
|
||||
// beats the model pinned in the pipeline version. It is pure and validates the
|
||||
// overrides against the pinned graph: an unknown stage key fails naming the valid
|
||||
// keys, a value that is not a provider/model fails, and a per-stage key for a
|
||||
// non-model stage fails because that stage could never honour it. A whole-run
|
||||
// override on a graph with no model stage is accepted and inert. The helper is
|
||||
// shared by the server (authoritative, against the pinned version) and the CLI
|
||||
// (pre-flight form check), so the precedence rule lives once.
|
||||
func EffectiveModels(graph protocol.Graph, model string, stageModels map[string]string) (map[string]ModelOverride, error) {
|
||||
keys := make([]string, 0, len(graph.Stages))
|
||||
byKey := make(map[string]protocol.Stage, len(graph.Stages))
|
||||
for _, s := range graph.Stages {
|
||||
keys = append(keys, s.Key)
|
||||
byKey[s.Key] = s
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
var whole *ModelOverride
|
||||
if strings.TrimSpace(model) != "" {
|
||||
parsed, err := ParseModelOverride(model)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
whole = &parsed
|
||||
}
|
||||
|
||||
perStage := make(map[string]ModelOverride, len(stageModels))
|
||||
overrideKeys := make([]string, 0, len(stageModels))
|
||||
for key := range stageModels {
|
||||
overrideKeys = append(overrideKeys, key)
|
||||
}
|
||||
sort.Strings(overrideKeys)
|
||||
for _, key := range overrideKeys {
|
||||
declared, exists := byKey[key]
|
||||
if !exists {
|
||||
return nil, fmt.Errorf("unknown stage key %q; valid keys: %s", key, strings.Join(keys, ", "))
|
||||
}
|
||||
if stage.EffectiveType(declared.Type) != stage.ModelType {
|
||||
return nil, fmt.Errorf("stage %q cannot honour a model override; it is not a model stage", key)
|
||||
}
|
||||
parsed, err := ParseModelOverride(stageModels[key])
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("stage %q: %w", key, err)
|
||||
}
|
||||
perStage[key] = parsed
|
||||
}
|
||||
|
||||
effective := make(map[string]ModelOverride, len(graph.Stages))
|
||||
for _, s := range graph.Stages {
|
||||
if stage.EffectiveType(s.Type) != stage.ModelType {
|
||||
continue
|
||||
}
|
||||
if override, ok := perStage[s.Key]; ok {
|
||||
effective[s.Key] = override
|
||||
continue
|
||||
}
|
||||
if whole != nil {
|
||||
effective[s.Key] = *whole
|
||||
continue
|
||||
}
|
||||
effective[s.Key] = ModelOverride{Provider: s.Configs["provider"], Model: s.Configs["model"]}
|
||||
}
|
||||
return effective, nil
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
func overrideGraph() protocol.Graph {
|
||||
return protocol.Graph{Stages: []protocol.Stage{
|
||||
{Key: "plan", Type: "model", Configs: map[string]string{"provider": "pinned", "model": "pinned-model"}},
|
||||
{Key: "note", Type: "ntfy", Configs: map[string]string{}},
|
||||
{Key: "review", Type: "model", Configs: map[string]string{"provider": "pinned2", "model": "pinned2-model"}},
|
||||
}}
|
||||
}
|
||||
|
||||
// TestEffectiveModelsPrecedence proves the one precedence rule: a per-stage
|
||||
// override beats the whole-run override, which beats the model pinned in the
|
||||
// pipeline version, and with no override the pinned pair is returned unchanged.
|
||||
func TestEffectiveModelsPrecedence(t *testing.T) {
|
||||
graph := overrideGraph()
|
||||
cases := []struct {
|
||||
name string
|
||||
model string
|
||||
stageModels map[string]string
|
||||
want map[string]ModelOverride
|
||||
}{
|
||||
{
|
||||
name: "no override keeps the pinned pair",
|
||||
want: map[string]ModelOverride{
|
||||
"plan": {Provider: "pinned", Model: "pinned-model"},
|
||||
"review": {Provider: "pinned2", Model: "pinned2-model"},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "whole-run beats the pinned pair",
|
||||
model: "acme/whole",
|
||||
want: map[string]ModelOverride{
|
||||
"plan": {Provider: "acme", Model: "whole"},
|
||||
"review": {Provider: "acme", Model: "whole"},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "per-stage beats whole-run and leaves the other pinned",
|
||||
model: "acme/whole",
|
||||
stageModels: map[string]string{"plan": "acme/stage"},
|
||||
want: map[string]ModelOverride{
|
||||
"plan": {Provider: "acme", Model: "stage"},
|
||||
"review": {Provider: "acme", Model: "whole"},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "per-stage alone leaves the other pinned",
|
||||
stageModels: map[string]string{"plan": "acme/stage"},
|
||||
want: map[string]ModelOverride{
|
||||
"plan": {Provider: "acme", Model: "stage"},
|
||||
"review": {Provider: "pinned2", Model: "pinned2-model"},
|
||||
},
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got, err := EffectiveModels(graph, tc.model, tc.stageModels)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != len(tc.want) {
|
||||
t.Fatalf("effective = %+v, want %+v", got, tc.want)
|
||||
}
|
||||
for key, want := range tc.want {
|
||||
if got[key] != want {
|
||||
t.Fatalf("stage %s = %+v, want %+v", key, got[key], want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestEffectiveModelsRejectsBadInput proves an unknown stage key fails naming the
|
||||
// valid keys, a malformed provider/model fails, and a per-stage key for a
|
||||
// non-model stage is refused as one that can never honour it.
|
||||
func TestEffectiveModelsRejectsBadInput(t *testing.T) {
|
||||
graph := overrideGraph()
|
||||
if _, err := EffectiveModels(graph, "", map[string]string{"nope": "acme/x"}); err == nil || !strings.Contains(err.Error(), "unknown stage key") || !strings.Contains(err.Error(), "note, plan, review") {
|
||||
t.Fatalf("unknown key error = %v, want the valid keys named", err)
|
||||
}
|
||||
for _, value := range []string{"not-a-pair", "/model", "provider/", "bad provider/model", "provider/bad model"} {
|
||||
if _, err := EffectiveModels(graph, value, nil); err == nil {
|
||||
t.Fatalf("whole-run value %q was accepted", value)
|
||||
}
|
||||
if _, err := EffectiveModels(graph, "", map[string]string{"plan": value}); err == nil {
|
||||
t.Fatalf("per-stage value %q was accepted", value)
|
||||
}
|
||||
}
|
||||
if _, err := EffectiveModels(graph, "", map[string]string{"note": "acme/x"}); err == nil || !strings.Contains(err.Error(), "cannot honour") {
|
||||
t.Fatalf("non-model stage override error = %v, want a cannot-honour refusal", err)
|
||||
}
|
||||
// A whole-run override on a graph with no model stage is accepted and inert.
|
||||
inert := protocol.Graph{Stages: []protocol.Stage{{Key: "note", Type: "ntfy", Configs: map[string]string{}}}}
|
||||
got, err := EffectiveModels(inert, "acme/whole", nil)
|
||||
if err != nil || len(got) != 0 {
|
||||
t.Fatalf("whole-run override on a no-model graph = %+v, %v; want inert", got, err)
|
||||
}
|
||||
}
|
||||
+64
-17
@@ -18,6 +18,7 @@ func (s *Store) initRuns() error {
|
||||
id INTEGER PRIMARY KEY, pipeline_id INTEGER NOT NULL, version INTEGER NOT NULL,
|
||||
request_key TEXT NOT NULL UNIQUE, workspace TEXT NOT NULL, allow_tools INTEGER NOT NULL,
|
||||
title TEXT NOT NULL DEFAULT '', title_source TEXT NOT NULL DEFAULT '', title_model TEXT NOT NULL DEFAULT '',
|
||||
model TEXT NOT NULL DEFAULT '', stage_models TEXT NOT NULL DEFAULT '{}',
|
||||
write_paths TEXT NOT NULL, input_json TEXT NOT NULL, output_json TEXT NOT NULL DEFAULT '{}',
|
||||
status TEXT NOT NULL DEFAULT 'queued', outcome TEXT NOT NULL DEFAULT '', current_stage TEXT NOT NULL DEFAULT '',
|
||||
diagnostic TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, finished_at TEXT,
|
||||
@@ -26,6 +27,7 @@ CREATE TABLE IF NOT EXISTS run_steps (
|
||||
run_id INTEGER NOT NULL REFERENCES pipeline_runs(id), sequence INTEGER NOT NULL, stage_key TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'running', outcome TEXT NOT NULL DEFAULT '', target TEXT NOT NULL DEFAULT '',
|
||||
input_json TEXT NOT NULL, output_json TEXT NOT NULL DEFAULT '{}', diagnostic TEXT NOT NULL DEFAULT '',
|
||||
provider TEXT NOT NULL DEFAULT '', model TEXT NOT NULL DEFAULT '',
|
||||
session_id TEXT NOT NULL DEFAULT '', agent_active INTEGER NOT NULL DEFAULT 0, transcript TEXT NOT NULL DEFAULT '',
|
||||
started_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, finished_at TEXT, PRIMARY KEY(run_id,sequence));
|
||||
CREATE INDEX IF NOT EXISTS run_latest ON pipeline_runs(id DESC);
|
||||
@@ -53,7 +55,28 @@ run_id INTEGER PRIMARY KEY REFERENCES pipeline_runs(id), original_run_id INTEGER
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.migrateRunTitles()
|
||||
if err := s.migrateRunTitles(); err != nil {
|
||||
return err
|
||||
}
|
||||
return s.migrateRunModels()
|
||||
}
|
||||
|
||||
// migrateRunModels adds the launch-time model-override columns to a database
|
||||
// created before per-run overrides existed. Each column is added only when
|
||||
// missing, so an upgrade is idempotent across repeated Open calls and existing
|
||||
// rows keep their defaults (no model override, no recorded step model).
|
||||
func (s *Store) migrateRunModels() error {
|
||||
for _, column := range [][3]string{
|
||||
{"pipeline_runs", "model", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"pipeline_runs", "stage_models", "TEXT NOT NULL DEFAULT '{}'"},
|
||||
{"run_steps", "provider", "TEXT NOT NULL DEFAULT ''"},
|
||||
{"run_steps", "model", "TEXT NOT NULL DEFAULT ''"},
|
||||
} {
|
||||
if err := s.addColumnIfMissing(column[0], column[1], column[2]); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// migrateRunTitles adds the title columns to a database created before run
|
||||
@@ -98,17 +121,28 @@ func (s *Store) addColumnIfMissing(table, column, definition string) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// RunOverrides are the launch-time model overrides recorded for a run:
|
||||
// Model applies to every model stage and StageModels overrides individual model
|
||||
// stages by key (per-stage wins over whole-run). They are recorded so the run
|
||||
// stays reproducible without touching the pinned pipeline version.
|
||||
type RunOverrides struct {
|
||||
Model string
|
||||
StageModels map[string]string
|
||||
}
|
||||
|
||||
// CreateRun inserts a run and derives its title from the prompt. It is the entry
|
||||
// point for callers with no generated title; the derived fallback is always
|
||||
// non-empty and never calls a model.
|
||||
func (s *Store) CreateRun(ctx context.Context, d *protocol.PipelineDetail, key, workspace string, tools bool, paths []string, input map[string]any, retryOf ...int64) (int64, bool, error) {
|
||||
return s.CreateRunTitled(ctx, d, key, workspace, tools, paths, input, summarise.Title{}, retryOf...)
|
||||
return s.CreateRunTitled(ctx, d, key, workspace, tools, paths, input, RunOverrides{}, summarise.Title{}, retryOf...)
|
||||
}
|
||||
|
||||
// CreateRunTitled inserts a run with the caller's title. A non-empty title (a
|
||||
// model generation) is stored verbatim; an empty one falls back to the
|
||||
// prompt-derived title, so a run never stores an empty title.
|
||||
func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail, key, workspace string, tools bool, paths []string, input map[string]any, title summarise.Title, retryOf ...int64) (int64, bool, error) {
|
||||
// CreateRunTitled inserts a run with the caller's title and launch-time model
|
||||
// overrides. A non-empty title (a model generation) is stored verbatim; an empty
|
||||
// one falls back to the prompt-derived title, so a run never stores an empty
|
||||
// title. The overrides are compared in the request-key idempotency check, so a
|
||||
// replay with different overrides is refused as a different run.
|
||||
func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail, key, workspace string, tools bool, paths []string, input map[string]any, overrides RunOverrides, title summarise.Title, retryOf ...int64) (int64, bool, error) {
|
||||
var parent int64
|
||||
if len(retryOf) > 0 {
|
||||
parent = retryOf[0]
|
||||
@@ -125,19 +159,27 @@ func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail,
|
||||
if err != nil {
|
||||
return 0, false, err
|
||||
}
|
||||
stageModels := overrides.StageModels
|
||||
if stageModels == nil {
|
||||
stageModels = map[string]string{}
|
||||
}
|
||||
encodedModels, err := json.Marshal(stageModels)
|
||||
if err != nil {
|
||||
return 0, false, err
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var id, pid int64
|
||||
var version int
|
||||
var oldInput, oldWorkspace, oldPaths string
|
||||
var oldInput, oldWorkspace, oldPaths, oldModel, oldStageModels string
|
||||
var oldTools bool
|
||||
err = s.db.QueryRowContext(ctx, "SELECT id,pipeline_id,version,input_json,workspace,write_paths,allow_tools FROM pipeline_runs WHERE request_key=?", key).Scan(&id, &pid, &version, &oldInput, &oldWorkspace, &oldPaths, &oldTools)
|
||||
err = s.db.QueryRowContext(ctx, "SELECT id,pipeline_id,version,input_json,workspace,write_paths,allow_tools,model,stage_models FROM pipeline_runs WHERE request_key=?", key).Scan(&id, &pid, &version, &oldInput, &oldWorkspace, &oldPaths, &oldTools, &oldModel, &oldStageModels)
|
||||
if err == nil {
|
||||
var oldParent int64
|
||||
if err := s.db.QueryRowContext(ctx, "SELECT COALESCE((SELECT original_run_id FROM run_retries WHERE run_id=?),0)", id).Scan(&oldParent); err != nil {
|
||||
return 0, false, err
|
||||
}
|
||||
if pid != d.ID || version != d.Version || oldInput != string(encoded) || oldWorkspace != workspace || oldPaths != string(writes) || oldTools != tools || oldParent != parent {
|
||||
if pid != d.ID || version != d.Version || oldInput != string(encoded) || oldWorkspace != workspace || oldPaths != string(writes) || oldTools != tools || oldParent != parent || oldModel != overrides.Model || oldStageModels != string(encodedModels) {
|
||||
return 0, false, fmt.Errorf("request key already used for a different run")
|
||||
}
|
||||
return id, false, nil
|
||||
@@ -150,7 +192,7 @@ func (s *Store) CreateRunTitled(ctx context.Context, d *protocol.PipelineDetail,
|
||||
return 0, false, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
result, err := tx.ExecContext(ctx, "INSERT INTO pipeline_runs(pipeline_id,version,request_key,workspace,allow_tools,write_paths,input_json,title,title_source,title_model) VALUES (?,?,?,?,?,?,?,?,?,?)", d.ID, d.Version, key, workspace, tools, string(writes), string(encoded), title.Text, title.Source, title.Model)
|
||||
result, err := tx.ExecContext(ctx, "INSERT INTO pipeline_runs(pipeline_id,version,request_key,workspace,allow_tools,write_paths,input_json,title,title_source,title_model,model,stage_models) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)", d.ID, d.Version, key, workspace, tools, string(writes), string(encoded), title.Text, title.Source, title.Model, overrides.Model, string(encodedModels))
|
||||
if err != nil {
|
||||
return 0, false, err
|
||||
}
|
||||
@@ -202,14 +244,19 @@ func (s *Store) SetRunTitle(ctx context.Context, id int64, title summarise.Title
|
||||
const runSelect = `SELECT r.id,r.pipeline_id,p.name,r.version,v.content_hash,r.status,r.outcome,r.current_stage,
|
||||
strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',r.finished_at),''),r.diagnostic,
|
||||
CASE WHEN r.status='succeeded' THEN r.output_json ELSE COALESCE((SELECT output_json FROM run_steps WHERE run_id=r.id AND status='completed' ORDER BY sequence DESC LIMIT 1),'{}') END,
|
||||
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title
|
||||
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed'),r.workspace,r.title,r.model,r.stage_models
|
||||
FROM pipeline_runs r JOIN pipelines p ON p.id=r.pipeline_id JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version`
|
||||
|
||||
func scanRun(row interface{ Scan(...any) error }, r *protocol.RunSummary) error {
|
||||
var output string
|
||||
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title); err != nil {
|
||||
var output, stageModels string
|
||||
if err := row.Scan(&r.ID, &r.PipelineID, &r.Name, &r.Version, &r.ContentHash, &r.Status, &r.Outcome, &r.CurrentStage, &r.CreatedAt, &r.FinishedAt, &r.Diagnostic, &output, &r.CompletedSteps, &r.Workspace, &r.Title, &r.Model, &stageModels); err != nil {
|
||||
return err
|
||||
}
|
||||
if stageModels != "" && stageModels != "{}" {
|
||||
if err := json.Unmarshal([]byte(stageModels), &r.StageModels); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
var values map[string]any
|
||||
if err := decodeValues(output, &values); err != nil {
|
||||
return err
|
||||
@@ -283,7 +330,7 @@ func (s *Store) GetRun(ctx context.Context, id int64) (*protocol.RunDetail, erro
|
||||
if err := json.Unmarshal([]byte(paths), &r.WritePaths); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx, `SELECT sequence,stage_key,status,outcome,target,input_json,output_json,diagnostic,session_id,agent_active,transcript,
|
||||
rows, err := tx.QueryContext(ctx, `SELECT sequence,stage_key,status,outcome,target,input_json,output_json,diagnostic,session_id,agent_active,transcript,provider,model,
|
||||
COALESCE((SELECT receipts_json FROM step_notifications n WHERE n.run_id=run_steps.run_id AND n.sequence=run_steps.sequence),'[]'),
|
||||
COALESCE((SELECT changes_json FROM step_file_changes f WHERE f.run_id=run_steps.run_id AND f.sequence=run_steps.sequence),'null'),
|
||||
COALESCE((SELECT artifacts_json FROM step_artifacts a WHERE a.run_id=run_steps.run_id AND a.sequence=run_steps.sequence),'null'),
|
||||
@@ -297,7 +344,7 @@ strftime('%Y-%m-%dT%H:%M:%SZ',started_at),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ'
|
||||
var step protocol.RunStep
|
||||
var notifications string
|
||||
var fileChanges, artifacts, storedVerdict string
|
||||
if err := rows.Scan(&step.Sequence, &step.Stage, &step.Status, &step.Outcome, &step.Target, &input, &output, &step.Diagnostic, &step.SessionID, &step.AgentActive, &step.Transcript, ¬ifications, &fileChanges, &artifacts, &storedVerdict, &step.StartedAt, &step.FinishedAt); err != nil {
|
||||
if err := rows.Scan(&step.Sequence, &step.Stage, &step.Status, &step.Outcome, &step.Target, &input, &output, &step.Diagnostic, &step.SessionID, &step.AgentActive, &step.Transcript, &step.Provider, &step.Model, ¬ifications, &fileChanges, &artifacts, &storedVerdict, &step.StartedAt, &step.FinishedAt); err != nil {
|
||||
rows.Close()
|
||||
return nil, err
|
||||
}
|
||||
@@ -537,8 +584,8 @@ func (s *Store) FinishStep(ctx context.Context, id int64, step protocol.RunStep)
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
if err := affected(tx.ExecContext(ctx, `UPDATE run_steps SET status=?,outcome=?,target=?,output_json=?,diagnostic=?,transcript=?,finished_at=CURRENT_TIMESTAMP
|
||||
WHERE run_id=? AND sequence=? AND status='running'`, step.Status, step.Outcome, step.Target, string(output), step.Diagnostic, step.Transcript, id, step.Sequence)); err != nil {
|
||||
if err := affected(tx.ExecContext(ctx, `UPDATE run_steps SET status=?,outcome=?,target=?,output_json=?,diagnostic=?,transcript=?,provider=?,model=?,finished_at=CURRENT_TIMESTAMP
|
||||
WHERE run_id=? AND sequence=? AND status='running'`, step.Status, step.Outcome, step.Target, string(output), step.Diagnostic, step.Transcript, step.Provider, step.Model, id, step.Sequence)); err != nil {
|
||||
return err
|
||||
}
|
||||
if step.FileChanges != nil {
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/summarise"
|
||||
)
|
||||
|
||||
@@ -58,7 +59,7 @@ func TestRunTitleStorageMigrationAndFallback(t *testing.T) {
|
||||
definition, _ := store.Get(ctx, items[0].ID, 1)
|
||||
|
||||
explicit, _, err := store.CreateRunTitled(ctx, definition, "explicit", "workspace", false, nil,
|
||||
map[string]any{"prompt": "a prompt that must not become the title"},
|
||||
map[string]any{"prompt": "a prompt that must not become the title"}, RunOverrides{},
|
||||
summarise.Title{Text: "Fix the flaky login test", Source: summarise.SourceModel, Model: "openai/gpt"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -160,6 +161,116 @@ func TestSetRunTitleRenamesWithoutTouchingInput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunModelOverrideRoundTripAndRequestKey proves the launch-time model
|
||||
// overrides round-trip through the run record and the per-step provider/model,
|
||||
// and that a request-key replay with different overrides is refused as a
|
||||
// different run.
|
||||
func TestRunModelOverrideRoundTripAndRequestKey(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store, err := Open(filepath.Join(t.TempDir(), "override.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
if _, err := store.Submit(ctx, "override", Template([]string{"one"}, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := store.List(ctx)
|
||||
definition, _ := store.Get(ctx, items[0].ID, 1)
|
||||
overrides := RunOverrides{Model: "acme/whole", StageModels: map[string]string{"one": "acme/strong"}}
|
||||
id, created, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
|
||||
map[string]any{"prompt": "task"}, overrides, summarise.Title{})
|
||||
if err != nil || !created {
|
||||
t.Fatalf("create override run = %d %v %v", id, created, err)
|
||||
}
|
||||
if err := store.BeginStep(ctx, id, 1, "one", map[string]any{"prompt": "task"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := store.FinishStep(ctx, id, protocol.RunStep{Sequence: 1, Stage: "one", Status: "completed", Outcome: "success", Output: map[string]any{}, Provider: "acme", Model: "strong"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := store.GetRun(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Model != "acme/whole" || got.StageModels["one"] != "acme/strong" {
|
||||
t.Fatalf("run record did not echo overrides: %+v", got)
|
||||
}
|
||||
if len(got.Steps) != 1 || got.Steps[0].Provider != "acme" || got.Steps[0].Model != "strong" {
|
||||
t.Fatalf("step did not record the effective model: %+v", got.Steps)
|
||||
}
|
||||
list, err := store.ListRuns(ctx)
|
||||
if err != nil || len(list) != 1 || list[0].Model != "acme/whole" || list[0].StageModels["one"] != "acme/strong" {
|
||||
t.Fatalf("run list omitted the overrides: %+v %v", list, err)
|
||||
}
|
||||
// An idempotent replay with the same overrides returns the same run.
|
||||
again, created, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
|
||||
map[string]any{"prompt": "task"}, overrides, summarise.Title{})
|
||||
if err != nil || created || again != id {
|
||||
t.Fatalf("same-override replay = %d %v %v, want the stored run", again, created, err)
|
||||
}
|
||||
// A different override under the same key is a different run.
|
||||
if _, _, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
|
||||
map[string]any{"prompt": "task"}, RunOverrides{StageModels: map[string]string{"one": "acme/other"}}, summarise.Title{}); err == nil || !strings.Contains(err.Error(), "request key already used") {
|
||||
t.Fatalf("differing-override replay was not refused: %v", err)
|
||||
}
|
||||
if _, _, err := store.CreateRunTitled(ctx, definition, "override-key", "workspace", false, nil,
|
||||
map[string]any{"prompt": "task"}, RunOverrides{}, summarise.Title{}); err == nil || !strings.Contains(err.Error(), "request key already used") {
|
||||
t.Fatalf("no-override replay was not refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunModelOverrideMigrationKeepsHistory proves a database created before the
|
||||
// override columns existed gains them idempotently on Open and still lists and
|
||||
// reads its historical rows with empty overrides.
|
||||
func TestRunModelOverrideMigrationKeepsHistory(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
path := filepath.Join(t.TempDir(), "override-migration.db")
|
||||
store, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := store.Submit(ctx, "legacy", Template([]string{"one"}, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := store.List(ctx)
|
||||
definition, _ := store.Get(ctx, items[0].ID, 1)
|
||||
id, _, err := store.CreateRun(ctx, definition, "legacy-key", "workspace", false, nil, map[string]any{"prompt": "task"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Simulate a pre-override database by dropping the columns, then reopen.
|
||||
for _, column := range []string{"model", "stage_models"} {
|
||||
if _, err := store.db.ExecContext(ctx, "ALTER TABLE pipeline_runs DROP COLUMN "+column); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, column := range []string{"provider", "model"} {
|
||||
if _, err := store.db.ExecContext(ctx, "ALTER TABLE run_steps DROP COLUMN "+column); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := store.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store, err = Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
runs, err := store.ListRuns(ctx)
|
||||
if err != nil || len(runs) != 1 {
|
||||
t.Fatalf("legacy run did not survive the override migration: %+v %v", runs, err)
|
||||
}
|
||||
got, err := store.GetRun(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Model != "" || len(got.StageModels) != 0 {
|
||||
t.Fatalf("migrated row should keep empty overrides: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRecoveryAndVersionPinning(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
path := filepath.Join(t.TempDir(), "audit.db")
|
||||
|
||||
@@ -147,7 +147,13 @@ type Message struct {
|
||||
WritePaths []string `json:"write_paths,omitempty"`
|
||||
RequestKey string `json:"request_key,omitempty"`
|
||||
Workspace string `json:"workspace,omitempty"`
|
||||
Limit int `json:"limit,omitempty"`
|
||||
// Model and StageModels are the launch-time model overrides: Model applies to
|
||||
// every model stage of the run and StageModels overrides individual stages by
|
||||
// key (per-stage wins over whole-run). Both are optional so older peers ignore
|
||||
// them; the same fields are echoed on the run record.
|
||||
Model string `json:"model,omitempty"`
|
||||
StageModels map[string]string `json:"stage_models,omitempty"`
|
||||
Limit int `json:"limit,omitempty"`
|
||||
// Chat is the brokered OpenCode chat traffic (route B). See chat.go for the
|
||||
// per-type payload structs and the wire shapes they mirror.
|
||||
ChatSessions []ChatSession `json:"chat_sessions,omitempty"`
|
||||
|
||||
@@ -118,6 +118,12 @@ type RunSummary struct {
|
||||
Diagnostic string `json:"diagnostic"`
|
||||
ResultPreview string `json:"result_preview,omitempty"`
|
||||
CompletedSteps int `json:"completed_steps"`
|
||||
// Model and StageModels are the launch-time overrides this run records so it
|
||||
// stays reproducible: Model covers every model stage and StageModels overrides
|
||||
// individual stages by key. They are empty for a run that used the pinned
|
||||
// models and for historical rows.
|
||||
Model string `json:"model,omitempty"`
|
||||
StageModels map[string]string `json:"stage_models,omitempty"`
|
||||
}
|
||||
|
||||
// Label is the operator-facing name of a run: the generated Title when the
|
||||
@@ -147,6 +153,11 @@ type RunStep struct {
|
||||
SessionID string `json:"session_id"`
|
||||
AgentActive bool `json:"agent_active"`
|
||||
Transcript string `json:"transcript"`
|
||||
// Provider and Model are the effective provider/model this visit actually ran
|
||||
// with, recorded for every model stage (including failures). They are empty
|
||||
// for non-model stages and historical steps.
|
||||
Provider string `json:"provider,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
// ArtifactsSkipped and ArtifactsTruncated count files excluded from the
|
||||
// bounded per-step scan (over the file/size cap) and by the total budget.
|
||||
ArtifactsSkipped int `json:"artifacts_skipped,omitempty"`
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/pipeline"
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
"git.a-wan.net/alan/pipelines/internal/stage"
|
||||
)
|
||||
|
||||
// TestRunnerAppliesModelOverrides proves a run with exactly one overridden stage
|
||||
// records that step's provider/model as the override and the other model stage as
|
||||
// the pinned pair, that the same effective values reach the resolved ModelConfig
|
||||
// (the single source the repair/formatter path reads), and that the run record
|
||||
// and returned graph echo the overrides while the pinned source is unchanged.
|
||||
func TestRunnerAppliesModelOverrides(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "overrides.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
if _, err := store.Submit(ctx, "overrides", pipeline.Template([]string{"one", "two"}, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := store.List(ctx)
|
||||
definition, err := store.Get(ctx, items[0].ID, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pinned := map[string]stage.ModelConfig{}
|
||||
for _, s := range definition.Graph.Stages {
|
||||
pinned[s.Key] = stage.ModelConfig{Provider: s.Configs["provider"], Model: s.Configs["model"]}
|
||||
}
|
||||
r, err := newRunner(ctx, store)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved := map[string]stage.ModelConfig{}
|
||||
r.execute = func(_ context.Context, def stage.Definition, _ stage.Values) (stage.Result, error) {
|
||||
spec, err := (stage.ModelStage{}).Spec(def)
|
||||
if err != nil {
|
||||
return stage.Result{}, err
|
||||
}
|
||||
resolved[def.Key] = spec.Config
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
run, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "one-override", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}, StageModels: map[string]string{"one": "acme/strong"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.wg.Wait()
|
||||
if got := resolved["one"]; got.Provider != "acme" || got.Model != "strong" {
|
||||
t.Fatalf("stage one resolved %+v, want the override", got)
|
||||
}
|
||||
if got := resolved["two"]; got.Provider != pinned["two"].Provider || got.Model != pinned["two"].Model {
|
||||
t.Fatalf("stage two resolved %+v, want the pinned %+v", got, pinned["two"])
|
||||
}
|
||||
detail, err := store.GetRun(ctx, run.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if detail.Model != "" || detail.StageModels["one"] != "acme/strong" {
|
||||
t.Fatalf("run record did not echo the overrides: %+v", detail)
|
||||
}
|
||||
if len(detail.Steps) != 2 {
|
||||
t.Fatalf("steps = %+v", detail.Steps)
|
||||
}
|
||||
if detail.Steps[0].Provider != "acme" || detail.Steps[0].Model != "strong" {
|
||||
t.Fatalf("overridden step evidence = %+v", detail.Steps[0])
|
||||
}
|
||||
if detail.Steps[1].Provider != pinned["two"].Provider || detail.Steps[1].Model != pinned["two"].Model {
|
||||
t.Fatalf("pinned step evidence = %+v, want %+v", detail.Steps[1], pinned["two"])
|
||||
}
|
||||
// The returned graph shows the effective pair for the overridden stage and
|
||||
// leaves the other at its pinned configs; `show --version N` (the immutable
|
||||
// source) is untouched.
|
||||
overlaid, err := r.getRun(run.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
configs := map[string]map[string]string{}
|
||||
for _, s := range overlaid.Graph.Stages {
|
||||
configs[s.Key] = s.Configs
|
||||
}
|
||||
if configs["one"]["provider"] != "acme" || configs["one"]["model"] != "strong" {
|
||||
t.Fatalf("overlaid graph stage one = %+v, want the override", configs["one"])
|
||||
}
|
||||
if configs["two"]["provider"] != pinned["two"].Provider || configs["two"]["model"] != pinned["two"].Model {
|
||||
t.Fatalf("overlaid graph stage two = %+v, want the pinned pair", configs["two"])
|
||||
}
|
||||
loaded, err := store.Get(ctx, definition.ID, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if loaded.Source != definition.Source {
|
||||
t.Fatal("the pinned immutable source was rewritten by an override")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunnerRefusesInvalidModelOverridesWithoutRunRow proves an unknown stage key// and a malformed provider/model both fail before admission and create no run row.
|
||||
func TestRunnerRefusesInvalidModelOverridesWithoutRunRow(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "reject.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
if _, err := store.Submit(ctx, "reject", pipeline.Template([]string{"one", "two"}, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := store.List(ctx)
|
||||
r, err := newRunner(ctx, store)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
|
||||
return stage.Result{Outcome: "success", Output: stage.Values{}}, nil
|
||||
}
|
||||
before, _ := store.ListRuns(ctx)
|
||||
cases := []struct {
|
||||
name string
|
||||
key string
|
||||
model string
|
||||
stage map[string]string
|
||||
want string
|
||||
}{
|
||||
{"unknown stage key", "unknown-key", "", map[string]string{"nope": "acme/strong"}, "unknown stage key"},
|
||||
{"invalid whole-run pair", "invalid-model", "not-a-pair", nil, "provider/model"},
|
||||
{"invalid stage pair", "invalid-stage", "", map[string]string{"one": "acme/"}, "provider/model"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
_, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: tc.key, Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}, Model: tc.model, StageModels: tc.stage})
|
||||
if err == nil || !strings.Contains(err.Error(), tc.want) {
|
||||
t.Fatalf("error = %v, want %q", err, tc.want)
|
||||
}
|
||||
if id, err := store.RunByKey(ctx, tc.key); err != nil || id != 0 {
|
||||
t.Fatalf("rejected request created run %d: %v", id, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
after, _ := store.ListRuns(ctx)
|
||||
if len(after) != len(before) {
|
||||
t.Fatalf("rejected requests created run rows: before %d after %d", len(before), len(after))
|
||||
}
|
||||
}
|
||||
|
||||
// TestRetryReusesModelOverrides proves a retry of a failed run reuses the
|
||||
// original run's recorded overrides, so the new attempt is reproducible.
|
||||
func TestRetryReusesModelOverrides(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
store, err := pipeline.Open(filepath.Join(t.TempDir(), "retry-override.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
if _, err := store.Submit(ctx, "retry-override", pipeline.Template([]string{"one"}, true)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, _ := store.List(ctx)
|
||||
r, err := newRunner(ctx, store)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.execute = func(context.Context, stage.Definition, stage.Values) (stage.Result, error) {
|
||||
return stage.Result{}, errors.New("provider failed")
|
||||
}
|
||||
first, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "first", Workspace: t.TempDir(), Input: map[string]any{"prompt": "task"}, StageModels: map[string]string{"one": "acme/strong"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.wg.Wait()
|
||||
if detail, _ := store.GetRun(ctx, first.ID); detail.Status != "failed" {
|
||||
t.Fatalf("original run did not fail: %+v", detail)
|
||||
}
|
||||
retried, err := r.retry(protocol.Message{RunID: first.ID, RequestKey: "retry"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.wg.Wait()
|
||||
if retried.StageModels["one"] != "acme/strong" || retried.Model != "" {
|
||||
t.Fatalf("retry did not reuse the original overrides: %+v", retried)
|
||||
}
|
||||
}
|
||||
@@ -141,6 +141,13 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
|
||||
if _, err := pipeline.Parse(pipeline.ExecutionSource(d.Source)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Validate the launch-time model overrides against the pinned graph before
|
||||
// admission: an unknown stage key or a malformed provider/model must never
|
||||
// insert a run row. The resolution is O(#overrides + #stages) with no model or
|
||||
// network call, and the same helper is the authoritative precedence rule.
|
||||
if _, err := pipeline.EffectiveModels(d.Graph, request.Model, request.StageModels); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, s := range d.Graph.Stages {
|
||||
if stage.EffectiveType(s.Type) != stage.ModelType {
|
||||
continue
|
||||
@@ -229,7 +236,7 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
id, created, err := r.store.CreateRunTitled(r.ctx, d, request.RequestKey, workspace, request.AllowTools, request.WritePaths, request.Input, runTitle, retryOf...)
|
||||
id, created, err := r.store.CreateRunTitled(r.ctx, d, request.RequestKey, workspace, request.AllowTools, request.WritePaths, request.Input, pipeline.RunOverrides{Model: request.Model, StageModels: request.StageModels}, runTitle, retryOf...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -241,6 +248,7 @@ func (r *runner) start(request protocol.Message, retryOf ...int64) (*protocol.Ru
|
||||
detail, err := r.store.GetRun(r.ctx, id)
|
||||
if detail != nil {
|
||||
_, detail.Executing = r.active[id]
|
||||
r.overlayEffectiveModels(detail)
|
||||
}
|
||||
return detail, err
|
||||
}
|
||||
@@ -285,10 +293,41 @@ func (r *runner) getRun(id int64) (*protocol.RunDetail, error) {
|
||||
d.Executing = true
|
||||
}
|
||||
}
|
||||
r.overlayEffectiveModels(d)
|
||||
}
|
||||
return d, err
|
||||
}
|
||||
|
||||
// overlayEffectiveModels replaces each model stage's provider/model in a returned
|
||||
// run's graph with the effective pair the run actually uses, so the app and TUI
|
||||
// stage-definition surfaces show the override with no client change. It is
|
||||
// presentation only: the persisted graph_json and the pinned source returned by
|
||||
// `show` are never rewritten. A run with no overrides is left untouched.
|
||||
func (r *runner) overlayEffectiveModels(d *protocol.RunDetail) {
|
||||
if d == nil || (d.Model == "" && len(d.StageModels) == 0) {
|
||||
return
|
||||
}
|
||||
effective, err := pipeline.EffectiveModels(d.Graph, d.Model, d.StageModels)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for i := range d.Graph.Stages {
|
||||
s := &d.Graph.Stages[i]
|
||||
if stage.EffectiveType(s.Type) != stage.ModelType {
|
||||
continue
|
||||
}
|
||||
override, ok := effective[s.Key]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if s.Configs == nil {
|
||||
s.Configs = map[string]string{}
|
||||
}
|
||||
s.Configs["provider"] = override.Provider
|
||||
s.Configs["model"] = override.Model
|
||||
}
|
||||
}
|
||||
|
||||
func (r *runner) listRuns() ([]protocol.RunSummary, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
@@ -324,7 +363,7 @@ func (r *runner) retry(request protocol.Message) (*protocol.RunDetail, error) {
|
||||
if original.Status != "failed" || original.FinishedAt == "" {
|
||||
return nil, fmt.Errorf("only failed runs can be retried (run %d is %s)", original.ID, original.Status)
|
||||
}
|
||||
return r.start(protocol.Message{PipelineID: original.PipelineID, Version: original.Version, RequestKey: request.RequestKey, Workspace: original.Workspace, Input: original.Input, AllowTools: original.AllowTools, WritePaths: original.WritePaths}, original.ID)
|
||||
return r.start(protocol.Message{PipelineID: original.PipelineID, Version: original.Version, RequestKey: request.RequestKey, Workspace: original.Workspace, Input: original.Input, AllowTools: original.AllowTools, WritePaths: original.WritePaths, Model: original.Model, StageModels: original.StageModels}, original.ID)
|
||||
}
|
||||
|
||||
func (r *runner) cancel(id int64) error {
|
||||
@@ -427,7 +466,7 @@ func (r *runner) resume(ctx context.Context, id int64, d *protocol.RunDetail, wo
|
||||
return
|
||||
}
|
||||
state := rebuildRunState(d, step)
|
||||
r.runLoop(ctx, id, d.Graph, d.Name, workspace, protocol.Message{WritePaths: d.WritePaths, AllowTools: d.AllowTools}, state)
|
||||
r.runLoop(ctx, id, d.Graph, d.Name, workspace, protocol.Message{WritePaths: d.WritePaths, AllowTools: d.AllowTools, Model: d.Model, StageModels: d.StageModels}, state)
|
||||
}
|
||||
|
||||
// runState is the loop's whole mutable state, extracted so a fresh run and a
|
||||
@@ -486,6 +525,20 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
finish("failed", "error", "stage returned an undeclared outcome")
|
||||
return
|
||||
}
|
||||
// Resolve the effective provider/model for every model stage once, before the
|
||||
// loop, so applying an override is an in-memory O(1) map lookup per visit and
|
||||
// never re-parses the graph or re-resolves the whole definition. A run with no
|
||||
// overrides skips the resolution entirely. Launch validation already ran this
|
||||
// helper, so a failure here only guards a resumed run whose overrides drifted.
|
||||
var effectiveModels map[string]pipeline.ModelOverride
|
||||
if request.Model != "" || len(request.StageModels) > 0 {
|
||||
resolved, err := pipeline.EffectiveModels(graph, request.Model, request.StageModels)
|
||||
if err != nil {
|
||||
finish("failed", "error", err.Error())
|
||||
return
|
||||
}
|
||||
effectiveModels = resolved
|
||||
}
|
||||
for sequence := state.nextSequence; sequence <= 30; sequence++ {
|
||||
state.nextSequence = sequence + 1
|
||||
if ctx.Err() != nil {
|
||||
@@ -508,6 +561,16 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
finish("failed", "error", "missing stage")
|
||||
return
|
||||
}
|
||||
// StageDefinition clones Configs, so setting the effective pair on the
|
||||
// clone is safe and the pinned graph is never mutated. Because
|
||||
// ModelStage.Spec reads def.Configs and passes spec.Config into the repair
|
||||
// path, the same effective model governs the stage and its repair.
|
||||
if stage.EffectiveType(def.Type) == stage.ModelType {
|
||||
if override, ok := effectiveModels[key]; ok {
|
||||
def.Configs["provider"] = override.Provider
|
||||
def.Configs["model"] = override.Model
|
||||
}
|
||||
}
|
||||
isHuman := stage.EffectiveType(def.Type) == stage.HumanType
|
||||
state.visits[key]++
|
||||
maxVisits := 5
|
||||
@@ -672,6 +735,13 @@ func (r *runner) runLoop(ctx context.Context, id int64, graph protocol.Graph, na
|
||||
}
|
||||
artifacts, artifactsSkipped, artifactsTruncated := r.scanArtifacts(id, sequence, artifactsDir)
|
||||
step := protocol.RunStep{Sequence: sequence, Stage: key, Status: "completed", Outcome: result.Outcome, Output: result.Output, Transcript: operations.Transcript(client.Transcript), SessionID: stageSession, FileChanges: changes, Artifacts: artifacts, ArtifactsSkipped: artifactsSkipped, ArtifactsTruncated: artifactsTruncated}
|
||||
// Stamp the effective provider/model this visit actually ran with for every
|
||||
// model stage, including failures, so the app/TUI evidence reflects any
|
||||
// override. Non-model stages keep empty values.
|
||||
if stage.EffectiveType(def.Type) == stage.ModelType {
|
||||
step.Provider = modelConfig.Provider
|
||||
step.Model = modelConfig.Model
|
||||
}
|
||||
if budget := budgets[key]; budget != nil {
|
||||
step.Transcript = stage.BudgetTranscript(step.Transcript, budget, modelConfig.Timeout, modelConfig.Retries)
|
||||
}
|
||||
|
||||
@@ -13,6 +13,10 @@ type RunOptions struct {
|
||||
Pipeline, Prompt, Title, Workspace string
|
||||
Yes, AllowTools bool
|
||||
WritePaths []string
|
||||
// Model and StageModels are the CLI's launch-time model overrides, passed
|
||||
// through to the run request without adding a TUI picker (that is a follow-up).
|
||||
Model string
|
||||
StageModels map[string]string
|
||||
}
|
||||
|
||||
func NewRun(ctx context.Context, url string, options RunOptions) Model {
|
||||
|
||||
@@ -359,6 +359,11 @@ func evidenceLines(step protocol.RunStep) []string {
|
||||
if step.SessionID == "" {
|
||||
lines = []string{"Recorded result evidence"}
|
||||
}
|
||||
// The effective provider/model this visit ran with, for a model stage; it
|
||||
// reflects any launch-time override rather than the pinned model.
|
||||
if step.Provider != "" || step.Model != "" {
|
||||
lines = append(lines, "Ran with model: "+step.Provider+"/"+step.Model)
|
||||
}
|
||||
if step.Transcript == "" {
|
||||
return append(lines, "No captured tool evidence yet.")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
package tui
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
// TestEvidenceLinesShowEffectiveModel proves the Evidence pane renders the
|
||||
// provider/model a step actually ran with, so a launch-time override is visible
|
||||
// rather than the pinned model.
|
||||
func TestEvidenceLinesShowEffectiveModel(t *testing.T) {
|
||||
step := protocol.RunStep{Sequence: 1, Stage: "one", Status: "completed", Outcome: "success", Provider: "acme", Model: "strong"}
|
||||
text := strings.Join(evidenceLines(step), "\n")
|
||||
if !strings.Contains(text, "acme/strong") {
|
||||
t.Fatalf("evidence pane omitted the effective model:\n%s", text)
|
||||
}
|
||||
// A non-model step records no provider/model and shows no model line.
|
||||
plain := strings.Join(evidenceLines(protocol.RunStep{Sequence: 2, Stage: "note", Status: "completed", Outcome: "success"}), "\n")
|
||||
if strings.Contains(plain, "Ran with model:") {
|
||||
t.Fatalf("non-model step claimed a model:\n%s", plain)
|
||||
}
|
||||
}
|
||||
@@ -147,7 +147,7 @@ func (m *Model) startSelectedRun() tea.Cmd {
|
||||
m.runCollapsed = nil
|
||||
m.runDetail = nil
|
||||
m.runTop = 0
|
||||
request := protocol.Message{Type: protocol.RunPipeline, PipelineID: m.detail.ID, Version: m.detail.Version, RequestKey: fmt.Sprintf("tui-%d", time.Now().UnixNano()), Title: m.cliOptions.Title, Input: initialRunInput(m.runTask), AllowTools: m.runHasMutationTools(), WritePaths: m.runPaths, Workspace: m.runWorkspace}
|
||||
request := protocol.Message{Type: protocol.RunPipeline, PipelineID: m.detail.ID, Version: m.detail.Version, RequestKey: fmt.Sprintf("tui-%d", time.Now().UnixNano()), Title: m.cliOptions.Title, Input: initialRunInput(m.runTask), AllowTools: m.runHasMutationTools(), WritePaths: m.runPaths, Workspace: m.runWorkspace, Model: m.cliOptions.Model, StageModels: m.cliOptions.StageModels}
|
||||
updated, cmd := m.runRequest(request)
|
||||
*m = updated.(Model)
|
||||
return cmd
|
||||
|
||||
Reference in New Issue
Block a user