diff --git a/app/notes/chat-thinking-level.md b/app/notes/chat-thinking-level.md new file mode 100644 index 00000000..4d7cc14b --- /dev/null +++ b/app/notes/chat-thinking-level.md @@ -0,0 +1,98 @@ +# Chat thinking level + +A per-chat thinking (reasoning effort) control in the conversation header, +beside the model and mode controls: "Thinking: High". Material opens a menu, +Cupertino an action sheet; both list only the levels the chat's current (or +saved next-turn) model accepts and check the current one. With no levels the +control is not shown. It uses the chat's own gateway, so a chat on another +machine is set on that machine's server. + +## Protocol + +- Session rows (`chat_sessions`, `chat_create`, session reads) carry + `effort` (current level, `""` = harness default), `efforts` + (`[{id, name, description?}]`) and `default_effort`, only when the server + knows the row model's levels without starting a harness. +- `chat_models` entries carry `efforts`/`default_effort` per model, and the + reply carries `chat_effort` with the chat's current state. +- `chat_effort {session_id, effort}` sets a level. The server validates it + against the current model's levels; the reply echoes + `{session_id, effort, efforts, default_effort}`. Errors: unsupported harness, + model without levels, a level the model does not list. +- Levels apply from the next turn. A running turn keeps its level; the menu + says so and the wide label reads "Thinking: Low ยท next turn" until the turn + ends. + +## Per harness + +- **Codex** (app-server 0.164): levels are model/list + `supportedReasoningEfforts` for the chat's model (catalog cached for one + minute so a list does not read it per row), the default is + `defaultReasoningEffort`, and the current level is the thread's own + `reasoningEffort` from thread/read unless Pipelines stored one. The stored + level is sent as turn/start `effort` (which Codex keeps for later turns). + Pipelines stores it in `operator_sessions.chat_effort`, re-applies it + before every turn and after a restart, and `/new` copies it. +- **Claude Code** (2.1.319): levels are the initialize models' + `supportedEffortLevels` where `supportsEffort` is true (Haiku 4.5 has none, + so no control). The level is sent with the `apply_flag_settings` control + request (`{"effortLevel": level}`, `null` restores the default) before the + next turn, only when the process was told another level. It is session-only + and never written to a settings file. Stored and re-applied like Codex. + Claude Code's initialize reports no default level, so with none chosen the + label reads "Thinking: Default". The levels come from a CLI's initialize: after + a server restart, before any Claude chat has started a CLI, rows carry no + levels and the control stays hidden until a CLI initializes (opening the + model chooser or sending a turn does that). +- **OpenCode** (2.0.x, v2 API): levels are the `/api/model` entry's + `variants` for the session's model. The chosen variant is the session + model's `variant`, set with `POST /api/session/{id}/model + {model:{id, providerID, variant}}`, which OpenCode itself persists, so + Pipelines stores nothing. A session without one reports `variant: + "default"`, which is not a listed level and shows as the default. The v1 + serve backend has no variants: no control. +- A model switch (Codex/Claude) keeps a stored level the new model accepts; + otherwise it becomes the new model's default (or the harness default). The + open control re-reads the levels from `chat_models` after a model switch. + +## Limits + +- The create-chat dialog offers no model choice (new chats start on the + server default model), so it shows no level either; set it in the header. +- If the opened row's model had no levels, the control is absent, so a switch + to a model with levels shows the control after the list re-reads (reopen). +- Phone widths show the control as an icon (its semantics label and menu + name the level), like the mode control. + +## Verification (2026-11-09) + +- Go: `go test ./internal/protocol ./internal/pipeline ./internal/codex + ./internal/server` ok. `./internal/claude` and `./internal/opencode`: all new + tests pass; one pre-existing failure each (`TestServiceTokenIsPassedOnlyFromAPrivateFile`, + `TestStageDiscoveryRequiresPrivateRegistrationBeforeSessionWork`), which fail + identically on base 8ca5a1fd. +- Flutter: `tool/memwatch/full.sh test/features/chat`: `PASS: 1123 passed, 0 + failed`. `tool/analyze.sh`: clean. +- Live Claude Code (opt-in `PIPELINES_LIVE_CLAUDE_EFFORT=1 go test + ./internal/claude -run TestClaudeEffortLive -v`, scratch directory, two Haiku + turns through the adapter, then `get_settings`): + + ``` + chose low: CLI applied model=claude-haiku-5-5 effort=low + chose max: CLI applied model=claude-haiku-5-5 effort=max + haiku levels from initialize: [low medium high xhigh max] + --- PASS: TestClaudeEffortLive (5.09s) + ``` + +- Live Codex, read-only (`PIPELINES_LIVE_CODEX_EFFORT=1 go test + ./internal/codex -run TestCodexEffortCatalogLive -v`): e.g. + `gpt-6.1-sol default="low" levels=[low medium high xhigh max ultra]`, + `gpt-6-luna default="medium" levels=[low medium high xhigh max]`. No Codex + turn was run, so turn/start `effort` was verified by the adapter test only. +- Live OpenCode, no inference, isolated HOME with `opencode api --standalone`: + a new session reported `"variant":"default"`; after posting + `variant:"high"` the session read back + `{"id": "inkling-2-small-free", "providerID": "opencode", "variant": "high"}`. + No OpenCode turn was run. +- Not run: a full Pipelines server with the Flutter app against a live + harness. Browser captures used the fixture app (`?chats=2&header=1`). diff --git a/internal/claude/efforts_live_test.go b/internal/claude/efforts_live_test.go new file mode 100644 index 00000000..eaa26aa7 --- /dev/null +++ b/internal/claude/efforts_live_test.go @@ -0,0 +1,85 @@ +package claude + +import ( + "context" + "encoding/json" + "os" + "testing" + "time" + + "git.a-wan.net/alan/pipelines/internal/protocol" +) + +// TestClaudeEffortLive drives the installed Claude Code through this adapter +// in a scratch directory: two short Haiku turns, each after a level change, +// and reads back the level the CLI applied (get_settings applied.effort). It +// runs inference on the signed-in account, so it is opt-in. +func TestClaudeEffortLive(t *testing.T) { + if os.Getenv("PIPELINES_LIVE_CLAUDE_EFFORT") == "" { + t.Skip("set PIPELINES_LIVE_CLAUDE_EFFORT=1 to run two short real Claude Code turns") + } + ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute) + defer cancel() + chat := New(ctx) + defer chat.Close() + session, err := chat.Create(ctx, "Pipelines effort live check", t.TempDir()) + if err != nil { + t.Fatal(err) + } + if err := chat.SwitchModel(ctx, session.ID, Backend, "haiku"); err != nil { + t.Fatal(err) + } + frames := make(chan protocol.ChatEventFrame, 4096) + go func() { + _ = chat.Stream(ctx, session.ID, nil, func(frame protocol.ChatEventFrame) error { + select { + case frames <- frame: + default: + } + return nil + }) + }() + time.Sleep(200 * time.Millisecond) + for _, level := range []string{"low", "max"} { + if err := chat.SetEffort(ctx, session.ID, level); err != nil { + t.Fatal(err) + } + if err := chat.Prompt(ctx, session.ID, "Reply with the single word OK."); err != nil { + t.Fatal(err) + } + for ended := false; !ended; { + select { + case frame := <-frames: + ended = frame.Kind == protocol.ChatEventStep && frame.Step != nil && frame.Step.Status == protocol.ChatStepEnded + case <-ctx.Done(): + t.Fatal("turn did not end") + } + } + s := chat.session(session.ID) + s.mu.Lock() + proc := s.proc + s.mu.Unlock() + raw, err := proc.request(ctx, "get_settings", nil) + if err != nil { + t.Fatal(err) + } + var settings struct { + Applied struct { + Model string `json:"model"` + Effort string `json:"effort"` + } `json:"applied"` + } + if err := json.Unmarshal(raw, &settings); err != nil { + t.Fatal(err) + } + t.Logf("chose %s: CLI applied model=%s effort=%s", level, settings.Applied.Model, settings.Applied.Effort) + if settings.Applied.Effort != level { + t.Fatalf("CLI applied effort %q, want %q", settings.Applied.Effort, level) + } + } + efforts, _, err := chat.Efforts(ctx, protocol.ChatSession{ID: session.ID, Model: "haiku"}) + if err != nil || len(efforts) == 0 { + t.Fatalf("haiku levels = %+v, %v", efforts, err) + } + t.Logf("haiku levels from initialize: %+v", efforts) +} diff --git a/internal/codex/efforts_live_test.go b/internal/codex/efforts_live_test.go new file mode 100644 index 00000000..1e451957 --- /dev/null +++ b/internal/codex/efforts_live_test.go @@ -0,0 +1,42 @@ +package codex + +import ( + "context" + "os" + "testing" + "time" +) + +// TestCodexEffortCatalogLive reads the installed App Server's model/list +// through this adapter. It is read-only (no thread, no turn) and opt-in, +// because it attaches to the user's Codex connection host. +func TestCodexEffortCatalogLive(t *testing.T) { + if os.Getenv("PIPELINES_LIVE_CODEX_EFFORT") == "" { + t.Skip("set PIPELINES_LIVE_CODEX_EFFORT=1 to read the real Codex model catalog") + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + chat, err := Open(ctx) + if err != nil { + t.Fatal(err) + } + defer chat.Close() + catalog, err := chat.catalog(ctx) + if err != nil { + t.Fatal(err) + } + withLevels := 0 + for _, model := range catalog { + ids := []string{} + for _, effort := range model.Efforts { + ids = append(ids, effort.ID) + } + t.Logf("%s default=%q levels=%v", model.ID, model.DefaultEffort, ids) + if len(model.Efforts) > 0 { + withLevels++ + } + } + if withLevels == 0 { + t.Fatal("no Codex model reported reasoning efforts") + } +}